2024-03-25T16:24:05.979 INFO:root:teuthology version: 0.0.1.dev262+ge691533 2024-03-25T16:24:05.979 DEBUG:teuthology.run:Teuthology command: teuthology --verbose --name amathuri-2024-03-25_13:28:33-rados:verify-main-distro-default-smithi --archive /home/teuthworker/archive/amathuri-2024-03-25_13:28:33-rados:verify-main-distro-default-smithi/7621279 --description rados:verify/{centos_latest ceph clusters/{fixed-2 openstack} d-thrash/default/{default thrashosds-health} mon_election/connectivity msgr-failures/few msgr/async-v1only objectstore/bluestore-stupid rados tasks/rados_api_tests validater/valgrind} --owner scheduled_amathuri@teuthology -- /home/teuthworker/archive/amathuri-2024-03-25_13:28:33-rados:verify-main-distro-default-smithi/7621279/orig.config.yaml 2024-03-25T16:24:06.023 DEBUG:teuthology.report:Pushing job info to https://paddles.front.sepia.ceph.com/ 2024-03-25T16:24:06.117 INFO:teuthology.run:Config: archive_path: /home/teuthworker/archive/amathuri-2024-03-25_13:28:33-rados:verify-main-distro-default-smithi/7621279 branch: main description: rados:verify/{centos_latest ceph clusters/{fixed-2 openstack} d-thrash/default/{default thrashosds-health} mon_election/connectivity msgr-failures/few msgr/async-v1only objectstore/bluestore-stupid rados tasks/rados_api_tests validater/valgrind} email: null first_in_suite: false job_id: '7621279' kernel: kdb: true sha1: distro last_in_suite: false machine_type: smithi name: amathuri-2024-03-25_13:28:33-rados:verify-main-distro-default-smithi no_nested_subset: false nuke-on-error: true openstack: - volumes: count: 4 size: 10 - volumes: count: 4 size: 10 os_type: centos os_version: 9.stream overrides: admin_socket: branch: main ceph: conf: client: debug monc: 20 debug ms: 1 debug objecter: 20 debug rados: 20 global: mon client directed command retry: 5 mon election default strategy: 3 ms bind msgr2: false ms inject socket failures: 5000 ms type: async osd heartbeat grace: 80 mgr: debug mgr: 20 debug ms: 1 mon: debug mon: 20 debug ms: 1 debug paxos: 20 mon min osdmap epochs: 50 mon osd crush smoke test: false mon osdmap full prune interval: 2 mon osdmap full prune min: 15 mon osdmap full prune txsize: 2 mon scrub interval: 300 mon warn on pool no app: false paxos service trim min: 10 osd: bdev async discard: true bdev enable discard: true bluestore allocator: stupid bluestore block size: 96636764160 bluestore fsck on mount: true bluestore zero block detection: true debug bluefs: 1 debug bluestore: 1 debug monc: 20 debug ms: 1 debug osd: 20 debug rocksdb: 10 mon osd backfillfull_ratio: 0.85 mon osd full ratio: 0.9 mon osd nearfull ratio: 0.8 osd blocked scrub grace period: 3600 osd class default list: '*' osd class load list: '*' osd client watch timeout: 120 osd debug verify cached snaps: true osd debug verify missing on start: true osd failsafe full ratio: 0.95 osd fast shutdown: false osd max markdown count: 1000 osd mclock override recovery settings: true osd mclock profile: high_recovery_ops osd objectstore: bluestore osd op queue: debug_random osd op queue cut off: debug_random osd shutdown pgref assert: true flavor: default fs: xfs log-ignorelist: - \(MDS_ALL_DOWN\) - \(MDS_UP_LESS_THAN_MAX\) - but it is still running - objects unfound and apparently lost - overall HEALTH_ - \(OSDMAP_FLAGS\) - \(OSD_ - \(PG_ - \(POOL_ - \(CACHE_POOL_ - \(SMALLER_PGP_NUM\) - \(OBJECT_ - \(SLOW_OPS\) - \(REQUEST_SLOW\) - \(TOO_FEW_PGS\) - slow request - timeout on replica - late reservation from - MON_DOWN - OSDMAP_FLAGS - OSD_DOWN - PG_DEGRADED - PG_AVAILABILITY - POOL_APP_NOT_ENABLED - \(OSD_SLOW_PING_TIME - reached quota - overall HEALTH_ - \(CACHE_POOL_NO_HIT_SET\) - \(POOL_FULL\) - \(SMALLER_PGP_NUM\) - \(SLOW_OPS\) - \(CACHE_POOL_NEAR_FULL\) - \(POOL_APP_NOT_ENABLED\) - \(PG_AVAILABILITY\) - \(OBJECT_MISPLACED\) - slow request - overall HEALTH_ - \(PG_ - \(MON_DOWN\) - \(SLOW_OPS\) - slow request mon_bind_msgr2: false sha1: 5dcb190da270440ef141d088eb27a9fdb8afc5f2 valgrind: mds: - --tool=memcheck mon: - --tool=memcheck - --leak-check=full - --show-reachable=yes osd: - --tool=memcheck ceph-deploy: bluestore: true conf: client: log file: /var/log/ceph/ceph-$name.$pid.log mon: {} osd: bdev async discard: true bdev enable discard: true bluestore block size: 96636764160 bluestore fsck on mount: true debug bluefs: 20 debug bluestore: 20 debug rocksdb: 10 mon osd backfillfull_ratio: 0.85 mon osd full ratio: 0.9 mon osd nearfull ratio: 0.8 osd failsafe full ratio: 0.95 osd objectstore: bluestore fs: xfs install: ceph: debuginfo: true flavor: default sha1: 5dcb190da270440ef141d088eb27a9fdb8afc5f2 thrashosds: bdev_inject_crash: 2 bdev_inject_crash_probability: 0.5 workunit: branch: main sha1: 1fb5d10608dccee73edd6ef8a2c7b804a097c324 owner: scheduled_amathuri@teuthology priority: 150 repo: https://git.ceph.com/ceph-ci.git roles: - - mon.a - mon.c - mgr.y - osd.0 - osd.1 - osd.2 - osd.3 - client.0 - node-exporter.a - - mon.b - mgr.x - osd.4 - osd.5 - osd.6 - osd.7 - client.1 - prometheus.a - node-exporter.b seed: 6785 sha1: 5dcb190da270440ef141d088eb27a9fdb8afc5f2 sleep_before_teardown: 0 subset: 99/1000 suite: rados:verify suite_branch: main suite_path: /home/teuthworker/src/git.ceph.com_ceph-c_1fb5d10608dccee73edd6ef8a2c7b804a097c324/qa suite_relpath: qa suite_repo: https://git.ceph.com/ceph-ci.git suite_sha1: 1fb5d10608dccee73edd6ef8a2c7b804a097c324 targets: smithi001.front.sepia.ceph.com: ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBJLX2RU8lPyaGpFOX5F2XcKa3PqVDhgC0eHcD7GRZ/nAnWQ6WgV2gQWu4nRS1XboDX2zvaMZYEkHmNn70BXWi7o= smithi115.front.sepia.ceph.com: ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBNV2pGoe90+assupoD4niScAzAzu4+FXUpDaFYAxOMgNegi3l1KLiZsa3tGXUVJrb6Id8T/FVoGK7g9F8xwHv9c= tasks: - install: null - ceph: null - thrashosds: chance_pgnum_grow: 1 chance_pgnum_shrink: 1 chance_pgpnum_fix: 1 timeout: 1200 - workunit: clients: client.0: - rados/test.sh env: ALLOW_TIMEOUTS: '1' timeout: 6h teuthology: fragments_dropped: [] meta: {} postmerge: [] teuthology_branch: main teuthology_sha1: e691533f9cbb33d85b2187bba20d7102f098636d timestamp: 2024-03-25_13:28:33 tube: smithi user: amathuri verbose: false worker_log: /home/teuthworker/archive/worker_logs/dispatcher.smithi.2226885 2024-03-25T16:24:06.117 INFO:teuthology.run:suite_path is set to /home/teuthworker/src/git.ceph.com_ceph-c_1fb5d10608dccee73edd6ef8a2c7b804a097c324/qa; will attempt to use it 2024-03-25T16:24:06.118 INFO:teuthology.run:Found tasks at /home/teuthworker/src/git.ceph.com_ceph-c_1fb5d10608dccee73edd6ef8a2c7b804a097c324/qa/tasks 2024-03-25T16:24:06.118 INFO:teuthology.run_tasks:Running task internal.check_packages... 2024-03-25T16:24:06.120 INFO:teuthology.task.internal:Checking packages... 2024-03-25T16:24:06.144 INFO:teuthology.task.internal:Checking packages for os_type 'centos', flavor 'default' and ceph hash '5dcb190da270440ef141d088eb27a9fdb8afc5f2' 2024-03-25T16:24:06.144 WARNING:teuthology.packaging:More than one of ref, tag, branch, or sha1 supplied; using branch 2024-03-25T16:24:06.144 INFO:teuthology.packaging:ref: None 2024-03-25T16:24:06.145 INFO:teuthology.packaging:tag: None 2024-03-25T16:24:06.145 INFO:teuthology.packaging:branch: main 2024-03-25T16:24:06.145 INFO:teuthology.packaging:sha1: 5dcb190da270440ef141d088eb27a9fdb8afc5f2 2024-03-25T16:24:06.145 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F9%2Fx86_64&ref=main 2024-03-25T16:24:06.479 INFO:teuthology.task.internal:Found packages for ceph version 19.0.0-2381.g44a9b520 2024-03-25T16:24:06.481 INFO:teuthology.run_tasks:Running task internal.buildpackages_prep... 2024-03-25T16:24:06.488 INFO:teuthology.task.internal:no buildpackages task found 2024-03-25T16:24:06.488 INFO:teuthology.run_tasks:Running task internal.save_config... 2024-03-25T16:24:06.549 INFO:teuthology.task.internal:Saving configuration 2024-03-25T16:24:06.567 INFO:teuthology.run_tasks:Running task internal.check_lock... 2024-03-25T16:24:06.578 INFO:teuthology.task.internal.check_lock:Checking locks... 2024-03-25T16:24:06.604 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi001.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/amathuri-2024-03-25_13:28:33-rados:verify-main-distro-default-smithi/7621279', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'centos', 'os_version': '9', 'arch': 'x86_64', 'locked': True, 'locked_since': '2024-03-25 16:18:57.127681', 'locked_by': 'scheduled_amathuri@teuthology', 'mac_address': None, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBJLX2RU8lPyaGpFOX5F2XcKa3PqVDhgC0eHcD7GRZ/nAnWQ6WgV2gQWu4nRS1XboDX2zvaMZYEkHmNn70BXWi7o='} 2024-03-25T16:24:06.629 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi115.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/amathuri-2024-03-25_13:28:33-rados:verify-main-distro-default-smithi/7621279', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'centos', 'os_version': '9', 'arch': 'x86_64', 'locked': True, 'locked_since': '2024-03-25 16:18:57.128889', 'locked_by': 'scheduled_amathuri@teuthology', 'mac_address': None, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBNV2pGoe90+assupoD4niScAzAzu4+FXUpDaFYAxOMgNegi3l1KLiZsa3tGXUVJrb6Id8T/FVoGK7g9F8xwHv9c='} 2024-03-25T16:24:06.629 INFO:teuthology.run_tasks:Running task internal.add_remotes... 2024-03-25T16:24:06.637 INFO:teuthology.task.internal:roles: ubuntu@smithi001.front.sepia.ceph.com - ['mon.a', 'mon.c', 'mgr.y', 'osd.0', 'osd.1', 'osd.2', 'osd.3', 'client.0', 'node-exporter.a'] 2024-03-25T16:24:06.637 INFO:teuthology.task.internal:roles: ubuntu@smithi115.front.sepia.ceph.com - ['mon.b', 'mgr.x', 'osd.4', 'osd.5', 'osd.6', 'osd.7', 'client.1', 'prometheus.a', 'node-exporter.b'] 2024-03-25T16:24:06.638 INFO:teuthology.run_tasks:Running task console_log... 2024-03-25T16:24:06.713 DEBUG:teuthology.exit:Installing handler: Handler(exiter=, func=.kill_console_loggers at 0x7f7dcc16c0d0>, signals=[15]) 2024-03-25T16:24:06.713 INFO:teuthology.run_tasks:Running task internal.connect... 2024-03-25T16:24:06.720 INFO:teuthology.task.internal:Opening connections... 2024-03-25T16:24:06.720 DEBUG:teuthology.task.internal:connecting to ubuntu@smithi001.front.sepia.ceph.com 2024-03-25T16:24:06.722 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi001.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-03-25T16:24:06.836 DEBUG:teuthology.task.internal:connecting to ubuntu@smithi115.front.sepia.ceph.com 2024-03-25T16:24:06.837 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi115.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-03-25T16:24:06.925 INFO:teuthology.run_tasks:Running task internal.push_inventory... 2024-03-25T16:24:06.934 DEBUG:teuthology.orchestra.run.smithi001:> uname -m 2024-03-25T16:24:06.952 INFO:teuthology.orchestra.run.smithi001.stdout:x86_64 2024-03-25T16:24:06.952 DEBUG:teuthology.orchestra.run.smithi001:> cat /etc/os-release 2024-03-25T16:24:07.009 INFO:teuthology.orchestra.run.smithi001.stdout:NAME="CentOS Stream" 2024-03-25T16:24:07.010 INFO:teuthology.orchestra.run.smithi001.stdout:VERSION="9" 2024-03-25T16:24:07.010 INFO:teuthology.orchestra.run.smithi001.stdout:ID="centos" 2024-03-25T16:24:07.010 INFO:teuthology.orchestra.run.smithi001.stdout:ID_LIKE="rhel fedora" 2024-03-25T16:24:07.010 INFO:teuthology.orchestra.run.smithi001.stdout:VERSION_ID="9" 2024-03-25T16:24:07.010 INFO:teuthology.orchestra.run.smithi001.stdout:PLATFORM_ID="platform:el9" 2024-03-25T16:24:07.010 INFO:teuthology.orchestra.run.smithi001.stdout:PRETTY_NAME="CentOS Stream 9" 2024-03-25T16:24:07.010 INFO:teuthology.orchestra.run.smithi001.stdout:ANSI_COLOR="0;31" 2024-03-25T16:24:07.010 INFO:teuthology.orchestra.run.smithi001.stdout:LOGO="fedora-logo-icon" 2024-03-25T16:24:07.010 INFO:teuthology.orchestra.run.smithi001.stdout:CPE_NAME="cpe:/o:centos:centos:9" 2024-03-25T16:24:07.010 INFO:teuthology.orchestra.run.smithi001.stdout:HOME_URL="https://centos.org/" 2024-03-25T16:24:07.010 INFO:teuthology.orchestra.run.smithi001.stdout:BUG_REPORT_URL="https://bugzilla.redhat.com/" 2024-03-25T16:24:07.010 INFO:teuthology.orchestra.run.smithi001.stdout:REDHAT_SUPPORT_PRODUCT="Red Hat Enterprise Linux 9" 2024-03-25T16:24:07.011 INFO:teuthology.orchestra.run.smithi001.stdout:REDHAT_SUPPORT_PRODUCT_VERSION="CentOS Stream" 2024-03-25T16:24:07.011 INFO:teuthology.lock.ops:Updating smithi001.front.sepia.ceph.com on lock server 2024-03-25T16:24:07.040 DEBUG:teuthology.orchestra.run.smithi115:> uname -m 2024-03-25T16:24:07.057 INFO:teuthology.orchestra.run.smithi115.stdout:x86_64 2024-03-25T16:24:07.057 DEBUG:teuthology.orchestra.run.smithi115:> cat /etc/os-release 2024-03-25T16:24:07.111 INFO:teuthology.orchestra.run.smithi115.stdout:NAME="CentOS Stream" 2024-03-25T16:24:07.112 INFO:teuthology.orchestra.run.smithi115.stdout:VERSION="9" 2024-03-25T16:24:07.112 INFO:teuthology.orchestra.run.smithi115.stdout:ID="centos" 2024-03-25T16:24:07.112 INFO:teuthology.orchestra.run.smithi115.stdout:ID_LIKE="rhel fedora" 2024-03-25T16:24:07.112 INFO:teuthology.orchestra.run.smithi115.stdout:VERSION_ID="9" 2024-03-25T16:24:07.112 INFO:teuthology.orchestra.run.smithi115.stdout:PLATFORM_ID="platform:el9" 2024-03-25T16:24:07.112 INFO:teuthology.orchestra.run.smithi115.stdout:PRETTY_NAME="CentOS Stream 9" 2024-03-25T16:24:07.112 INFO:teuthology.orchestra.run.smithi115.stdout:ANSI_COLOR="0;31" 2024-03-25T16:24:07.112 INFO:teuthology.orchestra.run.smithi115.stdout:LOGO="fedora-logo-icon" 2024-03-25T16:24:07.112 INFO:teuthology.orchestra.run.smithi115.stdout:CPE_NAME="cpe:/o:centos:centos:9" 2024-03-25T16:24:07.112 INFO:teuthology.orchestra.run.smithi115.stdout:HOME_URL="https://centos.org/" 2024-03-25T16:24:07.112 INFO:teuthology.orchestra.run.smithi115.stdout:BUG_REPORT_URL="https://bugzilla.redhat.com/" 2024-03-25T16:24:07.112 INFO:teuthology.orchestra.run.smithi115.stdout:REDHAT_SUPPORT_PRODUCT="Red Hat Enterprise Linux 9" 2024-03-25T16:24:07.113 INFO:teuthology.orchestra.run.smithi115.stdout:REDHAT_SUPPORT_PRODUCT_VERSION="CentOS Stream" 2024-03-25T16:24:07.113 INFO:teuthology.lock.ops:Updating smithi115.front.sepia.ceph.com on lock server 2024-03-25T16:24:07.139 INFO:teuthology.run_tasks:Running task internal.serialize_remote_roles... 2024-03-25T16:24:07.150 INFO:teuthology.run_tasks:Running task internal.check_conflict... 2024-03-25T16:24:07.187 INFO:teuthology.task.internal:Checking for old test directory... 2024-03-25T16:24:07.187 DEBUG:teuthology.orchestra.run.smithi001:> test '!' -e /home/ubuntu/cephtest 2024-03-25T16:24:07.190 DEBUG:teuthology.orchestra.run.smithi115:> test '!' -e /home/ubuntu/cephtest 2024-03-25T16:24:07.207 INFO:teuthology.run_tasks:Running task internal.check_ceph_data... 2024-03-25T16:24:07.216 INFO:teuthology.task.internal:Checking for non-empty /var/lib/ceph... 2024-03-25T16:24:07.217 DEBUG:teuthology.orchestra.run.smithi001:> test -z $(ls -A /var/lib/ceph) 2024-03-25T16:24:07.248 DEBUG:teuthology.orchestra.run.smithi115:> test -z $(ls -A /var/lib/ceph) 2024-03-25T16:24:07.274 INFO:teuthology.run_tasks:Running task internal.vm_setup... 2024-03-25T16:24:07.322 INFO:teuthology.run_tasks:Running task kernel... 2024-03-25T16:24:07.338 INFO:teuthology.task.kernel:normalize config orig: {'kdb': True, 'sha1': 'distro'} 2024-03-25T16:24:07.338 INFO:teuthology.task.kernel:config {'mon.a': {'kdb': True, 'sha1': 'distro'}, 'mon.b': {'kdb': True, 'sha1': 'distro'}}, timeout 300 2024-03-25T16:24:07.339 DEBUG:teuthology.orchestra.run.smithi001:> test -f /run/.containerenv -o -f /.dockerenv 2024-03-25T16:24:07.339 DEBUG:teuthology.orchestra.run.smithi115:> test -f /run/.containerenv -o -f /.dockerenv 2024-03-25T16:24:07.356 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-25T16:24:07.356 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-25T16:24:07.356 DEBUG:teuthology.orchestra.run.smithi001:> uname -r 2024-03-25T16:24:07.357 DEBUG:teuthology.orchestra.run.smithi115:> uname -r 2024-03-25T16:24:07.414 INFO:teuthology.orchestra.run.smithi115.stdout:5.14.0-361.el9.x86_64 2024-03-25T16:24:07.414 INFO:teuthology.task.kernel:Running kernel on smithi115: 5.14.0-361.el9.x86_64 2024-03-25T16:24:07.414 DEBUG:teuthology.orchestra.run.smithi115:> sudo yum install -y kernel 2024-03-25T16:24:07.415 INFO:teuthology.orchestra.run.smithi001.stdout:5.14.0-361.el9.x86_64 2024-03-25T16:24:07.415 INFO:teuthology.task.kernel:Running kernel on smithi001: 5.14.0-361.el9.x86_64 2024-03-25T16:24:07.415 DEBUG:teuthology.orchestra.run.smithi001:> sudo yum install -y kernel 2024-03-25T16:24:09.701 INFO:teuthology.orchestra.run.smithi001.stdout:CentOS-9 - AppStream 10 MB/s | 6.7 MB 00:00 2024-03-25T16:24:09.706 INFO:teuthology.orchestra.run.smithi115.stdout:CentOS-9 - AppStream 11 MB/s | 6.7 MB 00:00 2024-03-25T16:24:11.273 INFO:teuthology.orchestra.run.smithi115.stdout:CentOS-9 - BaseOS 4.5 MB/s | 2.0 MB 00:00 2024-03-25T16:24:11.276 INFO:teuthology.orchestra.run.smithi001.stdout:CentOS-9 - BaseOS 5.0 MB/s | 2.0 MB 00:00 2024-03-25T16:24:12.122 INFO:teuthology.orchestra.run.smithi115.stdout:CentOS Stream 9 - BaseOS 15 MB/s | 8.0 MB 00:00 2024-03-25T16:24:12.658 INFO:teuthology.orchestra.run.smithi001.stdout:CentOS Stream 9 - BaseOS 7.7 MB/s | 8.0 MB 00:01 2024-03-25T16:24:14.385 INFO:teuthology.orchestra.run.smithi001.stdout:CentOS Stream 9 - AppStream 30 MB/s | 19 MB 00:00 2024-03-25T16:24:16.481 INFO:teuthology.orchestra.run.smithi115.stdout:CentOS Stream 9 - AppStream 5.7 MB/s | 19 MB 00:03 2024-03-25T16:24:18.394 INFO:teuthology.orchestra.run.smithi001.stdout:CentOS Stream 9 - CRB 16 MB/s | 6.1 MB 00:00 2024-03-25T16:24:19.853 INFO:teuthology.orchestra.run.smithi001.stdout:CentOS Stream 9 - Extras packages 60 kB/s | 16 kB 00:00 2024-03-25T16:24:21.190 INFO:teuthology.orchestra.run.smithi115.stdout:CentOS Stream 9 - CRB 4.8 MB/s | 6.1 MB 00:01 2024-03-25T16:24:22.632 INFO:teuthology.orchestra.run.smithi115.stdout:CentOS Stream 9 - Extras packages 52 kB/s | 16 kB 00:00 2024-03-25T16:24:23.433 INFO:teuthology.orchestra.run.smithi115.stdout:Extra Packages for Enterprise Linux 30 MB/s | 21 MB 00:00 2024-03-25T16:24:28.416 INFO:teuthology.orchestra.run.smithi115.stdout:lab-extras 13 kB/s | 1.7 kB 00:00 2024-03-25T16:24:30.489 INFO:teuthology.orchestra.run.smithi115.stdout:Package kernel-5.14.0-239.el9.x86_64 is already installed. 2024-03-25T16:24:30.490 INFO:teuthology.orchestra.run.smithi115.stdout:Package kernel-5.14.0-361.el9.x86_64 is already installed. 2024-03-25T16:24:30.590 INFO:teuthology.orchestra.run.smithi115.stdout:Dependencies resolved. 2024-03-25T16:24:30.606 INFO:teuthology.orchestra.run.smithi115.stdout:================================================================================ 2024-03-25T16:24:30.607 INFO:teuthology.orchestra.run.smithi115.stdout: Package Architecture Version Repository Size 2024-03-25T16:24:30.607 INFO:teuthology.orchestra.run.smithi115.stdout:================================================================================ 2024-03-25T16:24:30.607 INFO:teuthology.orchestra.run.smithi115.stdout:Installing: 2024-03-25T16:24:30.607 INFO:teuthology.orchestra.run.smithi115.stdout: kernel x86_64 5.14.0-430.el9 baseos 44 k 2024-03-25T16:24:30.607 INFO:teuthology.orchestra.run.smithi115.stdout:Installing dependencies: 2024-03-25T16:24:30.607 INFO:teuthology.orchestra.run.smithi115.stdout: kernel-core x86_64 5.14.0-430.el9 baseos 15 M 2024-03-25T16:24:30.607 INFO:teuthology.orchestra.run.smithi115.stdout: kernel-modules x86_64 5.14.0-430.el9 baseos 33 M 2024-03-25T16:24:30.607 INFO:teuthology.orchestra.run.smithi115.stdout: kernel-modules-core x86_64 5.14.0-430.el9 baseos 27 M 2024-03-25T16:24:30.607 INFO:teuthology.orchestra.run.smithi115.stdout: 2024-03-25T16:24:30.607 INFO:teuthology.orchestra.run.smithi115.stdout:Transaction Summary 2024-03-25T16:24:30.607 INFO:teuthology.orchestra.run.smithi115.stdout:================================================================================ 2024-03-25T16:24:30.608 INFO:teuthology.orchestra.run.smithi115.stdout:Install 4 Packages 2024-03-25T16:24:30.608 INFO:teuthology.orchestra.run.smithi115.stdout: 2024-03-25T16:24:30.608 INFO:teuthology.orchestra.run.smithi115.stdout:Total download size: 76 M 2024-03-25T16:24:30.608 INFO:teuthology.orchestra.run.smithi115.stdout:Installed size: 123 M 2024-03-25T16:24:30.608 INFO:teuthology.orchestra.run.smithi115.stdout:Downloading Packages: 2024-03-25T16:24:30.742 INFO:teuthology.orchestra.run.smithi115.stdout:(1/4): kernel-5.14.0-430.el9.x86_64.rpm 470 kB/s | 44 kB 00:00 2024-03-25T16:24:31.351 INFO:teuthology.orchestra.run.smithi115.stdout:(2/4): kernel-core-5.14.0-430.el9.x86_64.rpm 21 MB/s | 15 MB 00:00 2024-03-25T16:24:31.893 INFO:teuthology.orchestra.run.smithi115.stdout:(3/4): kernel-modules-5.14.0-430.el9.x86_64.rpm 27 MB/s | 33 MB 00:01 2024-03-25T16:24:32.260 INFO:teuthology.orchestra.run.smithi115.stdout:(4/4): kernel-modules-core-5.14.0-430.el9.x86_6 18 MB/s | 27 MB 00:01 2024-03-25T16:24:32.260 INFO:teuthology.orchestra.run.smithi115.stdout:-------------------------------------------------------------------------------- 2024-03-25T16:24:32.260 INFO:teuthology.orchestra.run.smithi115.stdout:Total 46 MB/s | 76 MB 00:01 2024-03-25T16:24:32.926 INFO:teuthology.orchestra.run.smithi115.stdout:Running transaction check 2024-03-25T16:24:33.285 INFO:teuthology.orchestra.run.smithi115.stdout:Transaction check succeeded. 2024-03-25T16:24:33.286 INFO:teuthology.orchestra.run.smithi115.stdout:Running transaction test 2024-03-25T16:24:34.293 INFO:teuthology.orchestra.run.smithi115.stdout:Transaction test succeeded. 2024-03-25T16:24:34.293 INFO:teuthology.orchestra.run.smithi115.stdout:Running transaction 2024-03-25T16:24:36.840 INFO:teuthology.orchestra.run.smithi115.stdout: Preparing : 1/1 2024-03-25T16:24:37.438 INFO:teuthology.orchestra.run.smithi115.stdout: Installing : kernel-modules-core-5.14.0-430.el9.x86_64 1/4 2024-03-25T16:24:37.526 INFO:teuthology.orchestra.run.smithi115.stdout: Installing : kernel-core-5.14.0-430.el9.x86_64 2/4 2024-03-25T16:24:40.239 INFO:teuthology.orchestra.run.smithi001.stdout:Extra Packages for Enterprise Linux 1.0 MB/s | 21 MB 00:20 2024-03-25T16:24:40.264 INFO:teuthology.orchestra.run.smithi115.stdout: Running scriptlet: kernel-core-5.14.0-430.el9.x86_64 2/4 2024-03-25T16:24:40.394 INFO:teuthology.orchestra.run.smithi115.stdout: Installing : kernel-modules-5.14.0-430.el9.x86_64 3/4 2024-03-25T16:24:45.127 INFO:teuthology.orchestra.run.smithi115.stdout: Running scriptlet: kernel-modules-5.14.0-430.el9.x86_64 3/4 2024-03-25T16:24:45.229 INFO:teuthology.orchestra.run.smithi115.stdout: Installing : kernel-5.14.0-430.el9.x86_64 4/4 2024-03-25T16:24:45.444 INFO:teuthology.orchestra.run.smithi001.stdout:lab-extras 46 kB/s | 1.7 kB 00:00 2024-03-25T16:24:47.576 INFO:teuthology.orchestra.run.smithi001.stdout:Package kernel-5.14.0-239.el9.x86_64 is already installed. 2024-03-25T16:24:47.576 INFO:teuthology.orchestra.run.smithi001.stdout:Package kernel-5.14.0-361.el9.x86_64 is already installed. 2024-03-25T16:24:47.663 INFO:teuthology.orchestra.run.smithi001.stdout:Dependencies resolved. 2024-03-25T16:24:47.667 INFO:teuthology.orchestra.run.smithi001.stdout:================================================================================ 2024-03-25T16:24:47.667 INFO:teuthology.orchestra.run.smithi001.stdout: Package Architecture Version Repository Size 2024-03-25T16:24:47.668 INFO:teuthology.orchestra.run.smithi001.stdout:================================================================================ 2024-03-25T16:24:47.668 INFO:teuthology.orchestra.run.smithi001.stdout:Installing: 2024-03-25T16:24:47.668 INFO:teuthology.orchestra.run.smithi001.stdout: kernel x86_64 5.14.0-430.el9 baseos 44 k 2024-03-25T16:24:47.668 INFO:teuthology.orchestra.run.smithi001.stdout:Installing dependencies: 2024-03-25T16:24:47.668 INFO:teuthology.orchestra.run.smithi001.stdout: kernel-core x86_64 5.14.0-430.el9 baseos 15 M 2024-03-25T16:24:47.668 INFO:teuthology.orchestra.run.smithi001.stdout: kernel-modules x86_64 5.14.0-430.el9 baseos 33 M 2024-03-25T16:24:47.668 INFO:teuthology.orchestra.run.smithi001.stdout: kernel-modules-core x86_64 5.14.0-430.el9 baseos 27 M 2024-03-25T16:24:47.668 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-25T16:24:47.668 INFO:teuthology.orchestra.run.smithi001.stdout:Transaction Summary 2024-03-25T16:24:47.668 INFO:teuthology.orchestra.run.smithi001.stdout:================================================================================ 2024-03-25T16:24:47.668 INFO:teuthology.orchestra.run.smithi001.stdout:Install 4 Packages 2024-03-25T16:24:47.668 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-25T16:24:47.669 INFO:teuthology.orchestra.run.smithi001.stdout:Total download size: 76 M 2024-03-25T16:24:47.669 INFO:teuthology.orchestra.run.smithi001.stdout:Installed size: 123 M 2024-03-25T16:24:47.669 INFO:teuthology.orchestra.run.smithi001.stdout:Downloading Packages: 2024-03-25T16:24:48.154 INFO:teuthology.orchestra.run.smithi001.stdout:(1/4): kernel-5.14.0-430.el9.x86_64.rpm 210 kB/s | 44 kB 00:00 2024-03-25T16:24:48.679 INFO:teuthology.orchestra.run.smithi001.stdout:(2/4): kernel-core-5.14.0-430.el9.x86_64.rpm 20 MB/s | 15 MB 00:00 2024-03-25T16:24:49.263 INFO:teuthology.orchestra.run.smithi001.stdout:(3/4): kernel-modules-5.14.0-430.el9.x86_64.rpm 25 MB/s | 33 MB 00:01 2024-03-25T16:24:49.705 INFO:teuthology.orchestra.run.smithi001.stdout:(4/4): kernel-modules-core-5.14.0-430.el9.x86_6 18 MB/s | 27 MB 00:01 2024-03-25T16:24:49.707 INFO:teuthology.orchestra.run.smithi001.stdout:-------------------------------------------------------------------------------- 2024-03-25T16:24:49.707 INFO:teuthology.orchestra.run.smithi001.stdout:Total 37 MB/s | 76 MB 00:02 2024-03-25T16:24:49.928 INFO:teuthology.orchestra.run.smithi115.stdout: Running scriptlet: kernel-modules-core-5.14.0-430.el9.x86_64 4/4 2024-03-25T16:24:50.401 INFO:teuthology.orchestra.run.smithi001.stdout:Running transaction check 2024-03-25T16:24:50.848 INFO:teuthology.orchestra.run.smithi001.stdout:Transaction check succeeded. 2024-03-25T16:24:50.848 INFO:teuthology.orchestra.run.smithi001.stdout:Running transaction test 2024-03-25T16:24:51.908 INFO:teuthology.orchestra.run.smithi001.stdout:Transaction test succeeded. 2024-03-25T16:24:51.908 INFO:teuthology.orchestra.run.smithi001.stdout:Running transaction 2024-03-25T16:24:54.500 INFO:teuthology.orchestra.run.smithi001.stdout: Preparing : 1/1 2024-03-25T16:24:55.125 INFO:teuthology.orchestra.run.smithi001.stdout: Installing : kernel-modules-core-5.14.0-430.el9.x86_64 1/4 2024-03-25T16:24:55.221 INFO:teuthology.orchestra.run.smithi001.stdout: Installing : kernel-core-5.14.0-430.el9.x86_64 2/4 2024-03-25T16:24:58.121 INFO:teuthology.orchestra.run.smithi001.stdout: Running scriptlet: kernel-core-5.14.0-430.el9.x86_64 2/4 2024-03-25T16:24:58.256 INFO:teuthology.orchestra.run.smithi001.stdout: Installing : kernel-modules-5.14.0-430.el9.x86_64 3/4 2024-03-25T16:25:03.343 INFO:teuthology.orchestra.run.smithi001.stdout: Running scriptlet: kernel-modules-5.14.0-430.el9.x86_64 3/4 2024-03-25T16:25:03.383 INFO:teuthology.orchestra.run.smithi001.stdout: Installing : kernel-5.14.0-430.el9.x86_64 4/4 2024-03-25T16:25:08.503 INFO:teuthology.orchestra.run.smithi001.stdout: Running scriptlet: kernel-modules-core-5.14.0-430.el9.x86_64 4/4 2024-03-25T16:25:19.470 INFO:teuthology.orchestra.run.smithi115.stdout: Running scriptlet: kernel-core-5.14.0-430.el9.x86_64 4/4 2024-03-25T16:25:19.470 INFO:teuthology.orchestra.run.smithi115.stdout:dracut: Disabling early microcode, because kernel does not support it. CONFIG_MICROCODE_[AMD|INTEL]!=y 2024-03-25T16:25:19.470 INFO:teuthology.orchestra.run.smithi115.stdout: 2024-03-25T16:25:19.515 INFO:teuthology.orchestra.run.smithi115.stdout: Running scriptlet: kernel-modules-5.14.0-430.el9.x86_64 4/4 2024-03-25T16:25:19.867 INFO:teuthology.orchestra.run.smithi115.stdout: Running scriptlet: kernel-5.14.0-430.el9.x86_64 4/4 2024-03-25T16:25:19.868 INFO:teuthology.orchestra.run.smithi115.stdout: Verifying : kernel-5.14.0-430.el9.x86_64 1/4 2024-03-25T16:25:19.868 INFO:teuthology.orchestra.run.smithi115.stdout: Verifying : kernel-core-5.14.0-430.el9.x86_64 2/4 2024-03-25T16:25:19.868 INFO:teuthology.orchestra.run.smithi115.stdout: Verifying : kernel-modules-5.14.0-430.el9.x86_64 3/4 2024-03-25T16:25:20.286 INFO:teuthology.orchestra.run.smithi115.stdout: Verifying : kernel-modules-core-5.14.0-430.el9.x86_64 4/4 2024-03-25T16:25:20.286 INFO:teuthology.orchestra.run.smithi115.stdout: 2024-03-25T16:25:20.287 INFO:teuthology.orchestra.run.smithi115.stdout:Installed: 2024-03-25T16:25:20.287 INFO:teuthology.orchestra.run.smithi115.stdout: kernel-5.14.0-430.el9.x86_64 2024-03-25T16:25:20.287 INFO:teuthology.orchestra.run.smithi115.stdout: kernel-core-5.14.0-430.el9.x86_64 2024-03-25T16:25:20.287 INFO:teuthology.orchestra.run.smithi115.stdout: kernel-modules-5.14.0-430.el9.x86_64 2024-03-25T16:25:20.287 INFO:teuthology.orchestra.run.smithi115.stdout: kernel-modules-core-5.14.0-430.el9.x86_64 2024-03-25T16:25:20.287 INFO:teuthology.orchestra.run.smithi115.stdout: 2024-03-25T16:25:20.287 INFO:teuthology.orchestra.run.smithi115.stdout:Complete! 2024-03-25T16:25:20.616 DEBUG:teuthology.orchestra.run.smithi115:> rpm -q kernel | sort -rV | head -n 1 2024-03-25T16:25:20.732 INFO:teuthology.orchestra.run.smithi115.stdout:kernel-5.14.0-430.el9.x86_64 2024-03-25T16:25:20.732 DEBUG:teuthology.task.kernel:get_latest_image_version_rpm: 5.14.0-430.el9.x86_64 2024-03-25T16:25:20.732 INFO:teuthology.task.kernel:Not newest distro kernel. Current: 5.14.0-361.el9.x86_64 Expected: 5.14.0-430.el9.x86_64 2024-03-25T16:25:20.732 INFO:teuthology.task.kernel:Skipping firmware on distro kernel 2024-03-25T16:25:20.732 DEBUG:teuthology.task.kernel:src is distro, skipping download 2024-03-25T16:25:20.733 INFO:teuthology.task.kernel:Installing distro kernel on mon.b... 2024-03-25T16:25:20.733 DEBUG:teuthology.task.kernel:install_kernel(remote=ubuntu@smithi115.front.sepia.ceph.com, path=None, version=distro) 2024-03-25T16:25:20.733 DEBUG:teuthology.orchestra.run.smithi115:> rpm -q kernel | sort -rV | head -n 1 2024-03-25T16:25:20.811 INFO:teuthology.orchestra.run.smithi115.stdout:kernel-5.14.0-430.el9.x86_64 2024-03-25T16:25:20.812 DEBUG:teuthology.task.kernel:get_latest_image_version_rpm: 5.14.0-430.el9.x86_64 2024-03-25T16:25:20.812 DEBUG:teuthology.orchestra.run.smithi115:> sudo rpm -qi grub2-tools 2024-03-25T16:25:20.846 INFO:teuthology.orchestra.run.smithi115.stdout:Name : grub2-tools 2024-03-25T16:25:20.846 INFO:teuthology.orchestra.run.smithi115.stdout:Epoch : 1 2024-03-25T16:25:20.846 INFO:teuthology.orchestra.run.smithi115.stdout:Version : 2.06 2024-03-25T16:25:20.846 INFO:teuthology.orchestra.run.smithi115.stdout:Release : 68.el9 2024-03-25T16:25:20.846 INFO:teuthology.orchestra.run.smithi115.stdout:Architecture: x86_64 2024-03-25T16:25:20.846 INFO:teuthology.orchestra.run.smithi115.stdout:Install Date: Fri 01 Sep 2023 09:14:30 PM UTC 2024-03-25T16:25:20.846 INFO:teuthology.orchestra.run.smithi115.stdout:Group : Unspecified 2024-03-25T16:25:20.846 INFO:teuthology.orchestra.run.smithi115.stdout:Size : 8264254 2024-03-25T16:25:20.846 INFO:teuthology.orchestra.run.smithi115.stdout:License : GPLv3+ 2024-03-25T16:25:20.847 INFO:teuthology.orchestra.run.smithi115.stdout:Signature : RSA/SHA256, Mon 07 Aug 2023 01:57:16 PM UTC, Key ID 05b555b38483c65d 2024-03-25T16:25:20.847 INFO:teuthology.orchestra.run.smithi115.stdout:Source RPM : grub2-2.06-68.el9.src.rpm 2024-03-25T16:25:20.847 INFO:teuthology.orchestra.run.smithi115.stdout:Build Date : Wed 02 Aug 2023 03:15:48 PM UTC 2024-03-25T16:25:20.847 INFO:teuthology.orchestra.run.smithi115.stdout:Build Host : x86-05.stream.rdu2.redhat.com 2024-03-25T16:25:20.847 INFO:teuthology.orchestra.run.smithi115.stdout:Packager : builder@centos.org 2024-03-25T16:25:20.847 INFO:teuthology.orchestra.run.smithi115.stdout:Vendor : CentOS 2024-03-25T16:25:20.847 INFO:teuthology.orchestra.run.smithi115.stdout:URL : http://www.gnu.org/software/grub/ 2024-03-25T16:25:20.847 INFO:teuthology.orchestra.run.smithi115.stdout:Summary : Support tools for GRUB. 2024-03-25T16:25:20.847 INFO:teuthology.orchestra.run.smithi115.stdout:Description : 2024-03-25T16:25:20.847 INFO:teuthology.orchestra.run.smithi115.stdout: 2024-03-25T16:25:20.847 INFO:teuthology.orchestra.run.smithi115.stdout:The GRand Unified Bootloader (GRUB) is a highly configurable and 2024-03-25T16:25:20.847 INFO:teuthology.orchestra.run.smithi115.stdout:customizable bootloader with modular architecture. It supports a rich 2024-03-25T16:25:20.847 INFO:teuthology.orchestra.run.smithi115.stdout:variety of kernel formats, file systems, computer architectures and 2024-03-25T16:25:20.847 INFO:teuthology.orchestra.run.smithi115.stdout:hardware devices. 2024-03-25T16:25:20.847 INFO:teuthology.orchestra.run.smithi115.stdout: 2024-03-25T16:25:20.848 INFO:teuthology.orchestra.run.smithi115.stdout:This subpackage provides tools for support of all platforms. 2024-03-25T16:25:20.849 INFO:teuthology.task.kernel:Updating Grub Version: grub2 2024-03-25T16:25:20.849 INFO:teuthology.task.kernel:Updating grub on smithi115 to boot 5.14.0-430.el9.x86_64 2024-03-25T16:25:20.849 DEBUG:teuthology.orchestra.run.smithi115:> sudo grub2-mkconfig -o /boot/grub2/grub.cfg 2024-03-25T16:25:21.505 INFO:teuthology.orchestra.run.smithi115.stderr:Generating grub configuration file ... 2024-03-25T16:25:22.873 INFO:teuthology.orchestra.run.smithi115.stderr:Adding boot menu entry for UEFI Firmware Settings ... 2024-03-25T16:25:22.910 INFO:teuthology.orchestra.run.smithi115.stderr:done 2024-03-25T16:25:22.913 DEBUG:teuthology.orchestra.run.smithi115:> mktemp 2024-03-25T16:25:22.928 INFO:teuthology.orchestra.run.smithi115.stdout:/tmp/tmp.wwyqNcIRIv 2024-03-25T16:25:22.928 DEBUG:teuthology.orchestra.run.smithi115:> sudo cp /boot/grub2/grub.cfg /tmp/tmp.wwyqNcIRIv 2024-03-25T16:25:22.996 DEBUG:teuthology.orchestra.run.smithi115:> sudo chmod 0666 /tmp/tmp.wwyqNcIRIv 2024-03-25T16:25:23.139 DEBUG:teuthology.orchestra.remote:smithi115:/tmp/tmp.wwyqNcIRIv is 6KB 2024-03-25T16:25:23.189 DEBUG:teuthology.orchestra.run.smithi115:> rm -fr /tmp/tmp.wwyqNcIRIv 2024-03-25T16:25:23.203 DEBUG:teuthology.orchestra.run.smithi115:> sudo /bin/ls /boot/loader/entries || true 2024-03-25T16:25:23.267 INFO:teuthology.orchestra.run.smithi115.stdout:15949e3c88704300882482eddfd7baae-0-rescue.conf 2024-03-25T16:25:23.268 INFO:teuthology.orchestra.run.smithi115.stdout:15949e3c88704300882482eddfd7baae-5.14.0-239.el9.x86_64.conf 2024-03-25T16:25:23.268 INFO:teuthology.orchestra.run.smithi115.stdout:15949e3c88704300882482eddfd7baae-5.14.0-361.el9.x86_64.conf 2024-03-25T16:25:23.268 INFO:teuthology.orchestra.run.smithi115.stdout:15949e3c88704300882482eddfd7baae-5.14.0-430.el9.x86_64.conf 2024-03-25T16:25:23.269 DEBUG:teuthology.orchestra.run.smithi115:> sudo grub2-set-default 15949e3c88704300882482eddfd7baae-5.14.0-430.el9.x86_64 2024-03-25T16:25:23.425 DEBUG:teuthology.orchestra.run.smithi115:> sudo shutdown -r now 2024-03-25T16:25:39.337 INFO:teuthology.orchestra.run.smithi001.stdout: Running scriptlet: kernel-core-5.14.0-430.el9.x86_64 4/4 2024-03-25T16:25:39.337 INFO:teuthology.orchestra.run.smithi001.stdout:dracut: Disabling early microcode, because kernel does not support it. CONFIG_MICROCODE_[AMD|INTEL]!=y 2024-03-25T16:25:39.337 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-25T16:25:39.383 INFO:teuthology.orchestra.run.smithi001.stdout: Running scriptlet: kernel-modules-5.14.0-430.el9.x86_64 4/4 2024-03-25T16:25:39.775 INFO:teuthology.orchestra.run.smithi001.stdout: Running scriptlet: kernel-5.14.0-430.el9.x86_64 4/4 2024-03-25T16:25:39.775 INFO:teuthology.orchestra.run.smithi001.stdout: Verifying : kernel-5.14.0-430.el9.x86_64 1/4 2024-03-25T16:25:39.775 INFO:teuthology.orchestra.run.smithi001.stdout: Verifying : kernel-core-5.14.0-430.el9.x86_64 2/4 2024-03-25T16:25:39.775 INFO:teuthology.orchestra.run.smithi001.stdout: Verifying : kernel-modules-5.14.0-430.el9.x86_64 3/4 2024-03-25T16:25:40.229 INFO:teuthology.orchestra.run.smithi001.stdout: Verifying : kernel-modules-core-5.14.0-430.el9.x86_64 4/4 2024-03-25T16:25:40.229 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-25T16:25:40.229 INFO:teuthology.orchestra.run.smithi001.stdout:Installed: 2024-03-25T16:25:40.229 INFO:teuthology.orchestra.run.smithi001.stdout: kernel-5.14.0-430.el9.x86_64 2024-03-25T16:25:40.229 INFO:teuthology.orchestra.run.smithi001.stdout: kernel-core-5.14.0-430.el9.x86_64 2024-03-25T16:25:40.229 INFO:teuthology.orchestra.run.smithi001.stdout: kernel-modules-5.14.0-430.el9.x86_64 2024-03-25T16:25:40.229 INFO:teuthology.orchestra.run.smithi001.stdout: kernel-modules-core-5.14.0-430.el9.x86_64 2024-03-25T16:25:40.229 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-25T16:25:40.229 INFO:teuthology.orchestra.run.smithi001.stdout:Complete! 2024-03-25T16:25:40.556 DEBUG:teuthology.orchestra.run.smithi001:> rpm -q kernel | sort -rV | head -n 1 2024-03-25T16:25:40.670 INFO:teuthology.orchestra.run.smithi001.stdout:kernel-5.14.0-430.el9.x86_64 2024-03-25T16:25:40.670 DEBUG:teuthology.task.kernel:get_latest_image_version_rpm: 5.14.0-430.el9.x86_64 2024-03-25T16:25:40.670 INFO:teuthology.task.kernel:Not newest distro kernel. Current: 5.14.0-361.el9.x86_64 Expected: 5.14.0-430.el9.x86_64 2024-03-25T16:25:40.670 INFO:teuthology.task.kernel:Skipping firmware on distro kernel 2024-03-25T16:25:40.670 DEBUG:teuthology.task.kernel:src is distro, skipping download 2024-03-25T16:25:40.671 INFO:teuthology.task.kernel:Installing distro kernel on mon.a... 2024-03-25T16:25:40.671 DEBUG:teuthology.task.kernel:install_kernel(remote=ubuntu@smithi001.front.sepia.ceph.com, path=None, version=distro) 2024-03-25T16:25:40.671 DEBUG:teuthology.orchestra.run.smithi001:> rpm -q kernel | sort -rV | head -n 1 2024-03-25T16:25:40.754 INFO:teuthology.orchestra.run.smithi001.stdout:kernel-5.14.0-430.el9.x86_64 2024-03-25T16:25:40.754 DEBUG:teuthology.task.kernel:get_latest_image_version_rpm: 5.14.0-430.el9.x86_64 2024-03-25T16:25:40.755 DEBUG:teuthology.orchestra.run.smithi001:> sudo rpm -qi grub2-tools 2024-03-25T16:25:40.792 INFO:teuthology.orchestra.run.smithi001.stdout:Name : grub2-tools 2024-03-25T16:25:40.792 INFO:teuthology.orchestra.run.smithi001.stdout:Epoch : 1 2024-03-25T16:25:40.792 INFO:teuthology.orchestra.run.smithi001.stdout:Version : 2.06 2024-03-25T16:25:40.793 INFO:teuthology.orchestra.run.smithi001.stdout:Release : 68.el9 2024-03-25T16:25:40.793 INFO:teuthology.orchestra.run.smithi001.stdout:Architecture: x86_64 2024-03-25T16:25:40.793 INFO:teuthology.orchestra.run.smithi001.stdout:Install Date: Fri 01 Sep 2023 09:14:30 PM UTC 2024-03-25T16:25:40.793 INFO:teuthology.orchestra.run.smithi001.stdout:Group : Unspecified 2024-03-25T16:25:40.793 INFO:teuthology.orchestra.run.smithi001.stdout:Size : 8264254 2024-03-25T16:25:40.793 INFO:teuthology.orchestra.run.smithi001.stdout:License : GPLv3+ 2024-03-25T16:25:40.793 INFO:teuthology.orchestra.run.smithi001.stdout:Signature : RSA/SHA256, Mon 07 Aug 2023 01:57:16 PM UTC, Key ID 05b555b38483c65d 2024-03-25T16:25:40.793 INFO:teuthology.orchestra.run.smithi001.stdout:Source RPM : grub2-2.06-68.el9.src.rpm 2024-03-25T16:25:40.793 INFO:teuthology.orchestra.run.smithi001.stdout:Build Date : Wed 02 Aug 2023 03:15:48 PM UTC 2024-03-25T16:25:40.793 INFO:teuthology.orchestra.run.smithi001.stdout:Build Host : x86-05.stream.rdu2.redhat.com 2024-03-25T16:25:40.793 INFO:teuthology.orchestra.run.smithi001.stdout:Packager : builder@centos.org 2024-03-25T16:25:40.793 INFO:teuthology.orchestra.run.smithi001.stdout:Vendor : CentOS 2024-03-25T16:25:40.793 INFO:teuthology.orchestra.run.smithi001.stdout:URL : http://www.gnu.org/software/grub/ 2024-03-25T16:25:40.794 INFO:teuthology.orchestra.run.smithi001.stdout:Summary : Support tools for GRUB. 2024-03-25T16:25:40.794 INFO:teuthology.orchestra.run.smithi001.stdout:Description : 2024-03-25T16:25:40.794 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-25T16:25:40.794 INFO:teuthology.orchestra.run.smithi001.stdout:The GRand Unified Bootloader (GRUB) is a highly configurable and 2024-03-25T16:25:40.794 INFO:teuthology.orchestra.run.smithi001.stdout:customizable bootloader with modular architecture. It supports a rich 2024-03-25T16:25:40.794 INFO:teuthology.orchestra.run.smithi001.stdout:variety of kernel formats, file systems, computer architectures and 2024-03-25T16:25:40.794 INFO:teuthology.orchestra.run.smithi001.stdout:hardware devices. 2024-03-25T16:25:40.794 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-25T16:25:40.794 INFO:teuthology.orchestra.run.smithi001.stdout:This subpackage provides tools for support of all platforms. 2024-03-25T16:25:40.795 INFO:teuthology.task.kernel:Updating Grub Version: grub2 2024-03-25T16:25:40.795 INFO:teuthology.task.kernel:Updating grub on smithi001 to boot 5.14.0-430.el9.x86_64 2024-03-25T16:25:40.795 DEBUG:teuthology.orchestra.run.smithi001:> sudo grub2-mkconfig -o /boot/grub2/grub.cfg 2024-03-25T16:25:41.383 INFO:teuthology.orchestra.run.smithi001.stderr:Generating grub configuration file ... 2024-03-25T16:25:42.809 INFO:teuthology.orchestra.run.smithi001.stderr:Adding boot menu entry for UEFI Firmware Settings ... 2024-03-25T16:25:42.850 INFO:teuthology.orchestra.run.smithi001.stderr:done 2024-03-25T16:25:42.852 DEBUG:teuthology.orchestra.run.smithi001:> mktemp 2024-03-25T16:25:42.867 INFO:teuthology.orchestra.run.smithi001.stdout:/tmp/tmp.7FJglb8aqK 2024-03-25T16:25:42.867 DEBUG:teuthology.orchestra.run.smithi001:> sudo cp /boot/grub2/grub.cfg /tmp/tmp.7FJglb8aqK 2024-03-25T16:25:42.940 DEBUG:teuthology.orchestra.run.smithi001:> sudo chmod 0666 /tmp/tmp.7FJglb8aqK 2024-03-25T16:25:43.083 DEBUG:teuthology.orchestra.remote:smithi001:/tmp/tmp.7FJglb8aqK is 6KB 2024-03-25T16:25:43.131 DEBUG:teuthology.orchestra.run.smithi001:> rm -fr /tmp/tmp.7FJglb8aqK 2024-03-25T16:25:43.146 DEBUG:teuthology.orchestra.run.smithi001:> sudo /bin/ls /boot/loader/entries || true 2024-03-25T16:25:43.214 INFO:teuthology.orchestra.run.smithi001.stdout:15949e3c88704300882482eddfd7baae-0-rescue.conf 2024-03-25T16:25:43.215 INFO:teuthology.orchestra.run.smithi001.stdout:15949e3c88704300882482eddfd7baae-5.14.0-239.el9.x86_64.conf 2024-03-25T16:25:43.215 INFO:teuthology.orchestra.run.smithi001.stdout:15949e3c88704300882482eddfd7baae-5.14.0-361.el9.x86_64.conf 2024-03-25T16:25:43.215 INFO:teuthology.orchestra.run.smithi001.stdout:15949e3c88704300882482eddfd7baae-5.14.0-430.el9.x86_64.conf 2024-03-25T16:25:43.216 DEBUG:teuthology.orchestra.run.smithi001:> sudo grub2-set-default 15949e3c88704300882482eddfd7baae-5.14.0-430.el9.x86_64 2024-03-25T16:25:43.373 DEBUG:teuthology.orchestra.run.smithi001:> sudo shutdown -r now 2024-03-25T16:25:53.436 INFO:teuthology.task.kernel:Checking client mon.b for new kernel version... 2024-03-25T16:25:53.436 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi115.front.sepia.ceph.com' 2024-03-25T16:25:53.437 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi115.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-03-25T16:26:11.871 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.115 2024-03-25T16:26:13.377 INFO:teuthology.task.kernel:Checking client mon.a for new kernel version... 2024-03-25T16:26:13.377 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi001.front.sepia.ceph.com' 2024-03-25T16:26:13.378 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi001.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-03-25T16:26:20.878 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi115.front.sepia.ceph.com' 2024-03-25T16:26:20.879 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi115.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-03-25T16:26:23.938 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.115 2024-03-25T16:26:31.838 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.1 2024-03-25T16:26:35.942 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi115.front.sepia.ceph.com' 2024-03-25T16:26:35.943 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi115.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-03-25T16:26:40.842 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi001.front.sepia.ceph.com' 2024-03-25T16:26:40.843 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi001.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-03-25T16:26:54.370 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.115 2024-03-25T16:27:09.373 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi115.front.sepia.ceph.com' 2024-03-25T16:27:09.374 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi115.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-03-25T16:27:11.907 DEBUG:teuthology.orchestra.run.smithi115:> true 2024-03-25T16:27:12.373 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi115.front.sepia.ceph.com' 2024-03-25T16:27:12.373 INFO:teuthology.task.kernel:Checking kernel version of mon.b, want "5.14.0-430.el9.x86_64"... 2024-03-25T16:27:12.373 DEBUG:teuthology.orchestra.run.smithi115:> uname -r 2024-03-25T16:27:12.387 INFO:teuthology.orchestra.run.smithi115.stdout:5.14.0-430.el9.x86_64 2024-03-25T16:27:12.387 DEBUG:teuthology.task.kernel:current kernel version is 5.14.0-430.el9.x86_64 vs 5.14.0-430.el9.x86_64 2024-03-25T16:27:12.387 DEBUG:teuthology.task.kernel:utsrelease strings match, do not need to install 2024-03-25T16:27:12.387 DEBUG:teuthology.task.kernel:Distro of this test job: centos 2024-03-25T16:27:13.388 INFO:teuthology.task.kernel:Enabling kdb on mon.b... 2024-03-25T16:27:13.388 DEBUG:teuthology.orchestra.run.smithi115:> echo ttyS1 | sudo tee /sys/module/kgdboc/parameters/kgdboc 2024-03-25T16:27:13.484 INFO:teuthology.orchestra.run.smithi115.stdout:ttyS1 2024-03-25T16:27:13.517 DEBUG:teuthology.parallel:result is None 2024-03-25T16:27:40.845 DEBUG:teuthology.orchestra.remote:timed out 2024-03-25T16:27:52.847 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi001.front.sepia.ceph.com' 2024-03-25T16:27:52.848 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi001.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-03-25T16:27:53.153 DEBUG:teuthology.orchestra.run.smithi001:> true 2024-03-25T16:27:53.557 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi001.front.sepia.ceph.com' 2024-03-25T16:27:53.557 INFO:teuthology.task.kernel:Checking kernel version of mon.a, want "5.14.0-430.el9.x86_64"... 2024-03-25T16:27:53.558 DEBUG:teuthology.orchestra.run.smithi001:> uname -r 2024-03-25T16:27:53.573 INFO:teuthology.orchestra.run.smithi001.stdout:5.14.0-430.el9.x86_64 2024-03-25T16:27:53.573 DEBUG:teuthology.task.kernel:current kernel version is 5.14.0-430.el9.x86_64 vs 5.14.0-430.el9.x86_64 2024-03-25T16:27:53.573 DEBUG:teuthology.task.kernel:utsrelease strings match, do not need to install 2024-03-25T16:27:53.573 DEBUG:teuthology.task.kernel:Distro of this test job: centos 2024-03-25T16:27:54.574 INFO:teuthology.task.kernel:Enabling kdb on mon.a... 2024-03-25T16:27:54.575 DEBUG:teuthology.orchestra.run.smithi001:> echo ttyS1 | sudo tee /sys/module/kgdboc/parameters/kgdboc 2024-03-25T16:27:54.643 INFO:teuthology.orchestra.run.smithi001.stdout:ttyS1 2024-03-25T16:27:54.679 DEBUG:teuthology.parallel:result is None 2024-03-25T16:27:54.680 INFO:teuthology.run_tasks:Running task internal.base... 2024-03-25T16:27:54.687 INFO:teuthology.task.internal:Creating test directory... 2024-03-25T16:27:54.687 DEBUG:teuthology.orchestra.run.smithi001:> mkdir -p -m0755 -- /home/ubuntu/cephtest 2024-03-25T16:27:54.689 DEBUG:teuthology.orchestra.run.smithi115:> mkdir -p -m0755 -- /home/ubuntu/cephtest 2024-03-25T16:27:54.708 INFO:teuthology.run_tasks:Running task internal.archive_upload... 2024-03-25T16:27:54.714 INFO:teuthology.run_tasks:Running task internal.archive... 2024-03-25T16:27:54.721 INFO:teuthology.task.internal:Creating archive directory... 2024-03-25T16:27:54.721 DEBUG:teuthology.orchestra.run.smithi001:> install -d -m0755 -- /home/ubuntu/cephtest/archive 2024-03-25T16:27:54.746 DEBUG:teuthology.orchestra.run.smithi115:> install -d -m0755 -- /home/ubuntu/cephtest/archive 2024-03-25T16:27:54.813 INFO:teuthology.run_tasks:Running task internal.coredump... 2024-03-25T16:27:54.819 INFO:teuthology.task.internal:Enabling coredump saving... 2024-03-25T16:27:54.820 DEBUG:teuthology.orchestra.run.smithi001:> install -d -m0755 -- /home/ubuntu/cephtest/archive/coredump && sudo sysctl -w kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core && echo kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core | sudo tee -a /etc/sysctl.conf 2024-03-25T16:27:54.822 DEBUG:teuthology.orchestra.run.smithi115:> install -d -m0755 -- /home/ubuntu/cephtest/archive/coredump && sudo sysctl -w kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core && echo kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core | sudo tee -a /etc/sysctl.conf 2024-03-25T16:27:54.859 INFO:teuthology.orchestra.run.smithi115.stdout:kernel.core_pattern = /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-03-25T16:27:54.860 INFO:teuthology.orchestra.run.smithi001.stdout:kernel.core_pattern = /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-03-25T16:27:54.874 INFO:teuthology.orchestra.run.smithi115.stdout:kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-03-25T16:27:54.874 INFO:teuthology.orchestra.run.smithi001.stdout:kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-03-25T16:27:54.876 INFO:teuthology.run_tasks:Running task internal.sudo... 2024-03-25T16:27:54.882 INFO:teuthology.task.internal:Configuring sudo... 2024-03-25T16:27:54.883 DEBUG:teuthology.orchestra.run.smithi001:> sudo sed -i.orig.teuthology -e 's/^\([^#]*\) \(requiretty\)/\1 !\2/g' -e 's/^\([^#]*\) !\(visiblepw\)/\1 \2/g' /etc/sudoers 2024-03-25T16:27:54.919 DEBUG:teuthology.orchestra.run.smithi115:> sudo sed -i.orig.teuthology -e 's/^\([^#]*\) \(requiretty\)/\1 !\2/g' -e 's/^\([^#]*\) !\(visiblepw\)/\1 \2/g' /etc/sudoers 2024-03-25T16:27:54.948 INFO:teuthology.run_tasks:Running task internal.syslog... 2024-03-25T16:27:54.957 INFO:teuthology.task.internal.syslog:Starting syslog monitoring... 2024-03-25T16:27:54.957 DEBUG:teuthology.orchestra.run.smithi001:> mkdir -p -m0755 -- /home/ubuntu/cephtest/archive/syslog 2024-03-25T16:27:54.988 DEBUG:teuthology.orchestra.run.smithi115:> mkdir -p -m0755 -- /home/ubuntu/cephtest/archive/syslog 2024-03-25T16:27:55.004 DEBUG:teuthology.orchestra.run.smithi001:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/kern.log 2024-03-25T16:27:55.082 DEBUG:teuthology.orchestra.run.smithi001:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/kern.log 2024-03-25T16:27:55.129 DEBUG:teuthology.orchestra.run.smithi001:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/misc.log 2024-03-25T16:27:55.210 DEBUG:teuthology.orchestra.run.smithi001:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/misc.log 2024-03-25T16:27:55.253 DEBUG:teuthology.orchestra.run.smithi001:> set -ex 2024-03-25T16:27:55.253 DEBUG:teuthology.orchestra.run.smithi001:> sudo dd of=/etc/rsyslog.d/80-cephtest.conf 2024-03-25T16:27:55.320 DEBUG:teuthology.orchestra.run.smithi115:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/kern.log 2024-03-25T16:27:55.366 DEBUG:teuthology.orchestra.run.smithi115:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/kern.log 2024-03-25T16:27:55.417 DEBUG:teuthology.orchestra.run.smithi115:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/misc.log 2024-03-25T16:27:55.500 DEBUG:teuthology.orchestra.run.smithi115:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/misc.log 2024-03-25T16:27:55.548 DEBUG:teuthology.orchestra.run.smithi115:> set -ex 2024-03-25T16:27:55.548 DEBUG:teuthology.orchestra.run.smithi115:> sudo dd of=/etc/rsyslog.d/80-cephtest.conf 2024-03-25T16:27:55.618 DEBUG:teuthology.orchestra.run.smithi001:> sudo service rsyslog restart 2024-03-25T16:27:55.620 DEBUG:teuthology.orchestra.run.smithi115:> sudo service rsyslog restart 2024-03-25T16:27:55.663 INFO:teuthology.orchestra.run.smithi001.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2024-03-25T16:27:55.721 INFO:teuthology.orchestra.run.smithi115.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2024-03-25T16:27:56.035 INFO:teuthology.run_tasks:Running task internal.timer... 2024-03-25T16:27:56.043 INFO:teuthology.task.internal:Starting timer... 2024-03-25T16:27:56.043 INFO:teuthology.run_tasks:Running task pcp... 2024-03-25T16:27:56.054 INFO:teuthology.run_tasks:Running task selinux... 2024-03-25T16:27:56.063 DEBUG:teuthology.orchestra.run.smithi001:> sudo service auditd rotate 2024-03-25T16:27:56.125 INFO:teuthology.orchestra.run.smithi001.stdout:Rotating logs: 2024-03-25T16:27:56.127 DEBUG:teuthology.orchestra.run.smithi115:> sudo service auditd rotate 2024-03-25T16:27:56.174 INFO:teuthology.orchestra.run.smithi115.stdout:Rotating logs: 2024-03-25T16:27:56.176 DEBUG:teuthology.task.selinux:Getting current SELinux state 2024-03-25T16:27:56.176 DEBUG:teuthology.orchestra.run.smithi001:> /usr/sbin/getenforce 2024-03-25T16:27:56.194 INFO:teuthology.orchestra.run.smithi001.stdout:Permissive 2024-03-25T16:27:56.194 DEBUG:teuthology.orchestra.run.smithi115:> /usr/sbin/getenforce 2024-03-25T16:27:56.258 INFO:teuthology.orchestra.run.smithi115.stdout:Permissive 2024-03-25T16:27:56.259 DEBUG:teuthology.task.selinux:Existing SELinux modes: {'ubuntu@smithi001.front.sepia.ceph.com': 'permissive', 'ubuntu@smithi115.front.sepia.ceph.com': 'permissive'} 2024-03-25T16:27:56.259 DEBUG:teuthology.orchestra.run.smithi001:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 2024-03-25T16:27:56.286 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-25T16:27:56.287 DEBUG:teuthology.orchestra.run.smithi115:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 2024-03-25T16:27:56.316 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-25T16:27:56.317 INFO:teuthology.task.selinux:Putting SELinux into permissive mode 2024-03-25T16:27:56.317 DEBUG:teuthology.orchestra.run.smithi001:> sudo /usr/sbin/setenforce permissive 2024-03-25T16:27:56.363 DEBUG:teuthology.orchestra.run.smithi115:> sudo /usr/sbin/setenforce permissive 2024-03-25T16:27:56.393 INFO:teuthology.run_tasks:Running task ansible.cephlab... 2024-03-25T16:27:56.403 DEBUG:teuthology.repo_utils:Setting repo remote to https://git.ceph.com/ceph-cm-ansible.git 2024-03-25T16:27:56.410 INFO:teuthology.repo_utils:Fetching git.ceph.com_ceph-cm-ansible_main from origin 2024-03-25T16:27:56.526 DEBUG:teuthology.repo_utils:Resetting repo at /home/teuthworker/src/git.ceph.com_ceph-cm-ansible_main to origin/main 2024-03-25T16:27:56.543 INFO:teuthology.task.ansible:Playbook: [{'import_playbook': 'ansible_managed.yml'}, {'import_playbook': 'teuthology.yml'}, {'hosts': 'testnodes', 'tasks': [{'set_fact': {'ran_from_cephlab_playbook': True}}]}, {'import_playbook': 'testnodes.yml'}, {'import_playbook': 'container-host.yml'}, {'import_playbook': 'cobbler.yml'}, {'import_playbook': 'paddles.yml'}, {'import_playbook': 'pulpito.yml'}, {'hosts': 'testnodes', 'become': True, 'tasks': [{'name': 'Touch /ceph-qa-ready', 'file': {'path': '/ceph-qa-ready', 'state': 'touch'}, 'when': 'ran_from_cephlab_playbook|bool'}]}] 2024-03-25T16:27:56.544 DEBUG:teuthology.task.ansible:Running ansible-playbook -v --extra-vars '{"ansible_ssh_user": "ubuntu"}' -i /etc/ansible/hosts --limit smithi001.front.sepia.ceph.com,smithi115.front.sepia.ceph.com /home/teuthworker/src/git.ceph.com_ceph-cm-ansible_main/cephlab.yml 2024-03-25T16:32:11.560 DEBUG:teuthology.task.ansible:Reconnecting to [Remote(name='ubuntu@smithi001.front.sepia.ceph.com'), Remote(name='ubuntu@smithi115.front.sepia.ceph.com')] 2024-03-25T16:32:11.561 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi001.front.sepia.ceph.com' 2024-03-25T16:32:11.562 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi001.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-03-25T16:32:11.657 DEBUG:teuthology.orchestra.run.smithi001:> true 2024-03-25T16:32:11.739 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi001.front.sepia.ceph.com' 2024-03-25T16:32:11.740 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi115.front.sepia.ceph.com' 2024-03-25T16:32:11.741 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi115.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-03-25T16:32:11.830 DEBUG:teuthology.orchestra.run.smithi115:> true 2024-03-25T16:32:11.906 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi115.front.sepia.ceph.com' 2024-03-25T16:32:11.906 INFO:teuthology.run_tasks:Running task clock... 2024-03-25T16:32:11.923 INFO:teuthology.task.clock:Syncing clocks and checking initial clock skew... 2024-03-25T16:32:11.923 INFO:teuthology.orchestra.run:Running command with timeout 360 2024-03-25T16:32:11.924 DEBUG:teuthology.orchestra.run.smithi001:> sudo systemctl stop ntp.service || sudo systemctl stop ntpd.service || sudo systemctl stop chronyd.service ; sudo ntpd -gq || sudo chronyc makestep ; sudo systemctl start ntp.service || sudo systemctl start ntpd.service || sudo systemctl start chronyd.service ; PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2024-03-25T16:32:11.927 INFO:teuthology.orchestra.run:Running command with timeout 360 2024-03-25T16:32:11.928 DEBUG:teuthology.orchestra.run.smithi115:> sudo systemctl stop ntp.service || sudo systemctl stop ntpd.service || sudo systemctl stop chronyd.service ; sudo ntpd -gq || sudo chronyc makestep ; sudo systemctl start ntp.service || sudo systemctl start ntpd.service || sudo systemctl start chronyd.service ; PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2024-03-25T16:32:11.959 INFO:teuthology.orchestra.run.smithi001.stderr:Failed to stop ntp.service: Unit ntp.service not loaded. 2024-03-25T16:32:11.976 INFO:teuthology.orchestra.run.smithi001.stderr:Failed to stop ntpd.service: Unit ntpd.service not loaded. 2024-03-25T16:32:11.978 INFO:teuthology.orchestra.run.smithi115.stderr:Failed to stop ntp.service: Unit ntp.service not loaded. 2024-03-25T16:32:11.995 INFO:teuthology.orchestra.run.smithi115.stderr:Failed to stop ntpd.service: Unit ntpd.service not loaded. 2024-03-25T16:32:12.006 INFO:teuthology.orchestra.run.smithi001.stderr:sudo: ntpd: command not found 2024-03-25T16:32:12.021 INFO:teuthology.orchestra.run.smithi001.stdout:506 Cannot talk to daemon 2024-03-25T16:32:12.025 INFO:teuthology.orchestra.run.smithi115.stderr:sudo: ntpd: command not found 2024-03-25T16:32:12.037 INFO:teuthology.orchestra.run.smithi001.stderr:Failed to start ntp.service: Unit ntp.service not found. 2024-03-25T16:32:12.039 INFO:teuthology.orchestra.run.smithi115.stdout:506 Cannot talk to daemon 2024-03-25T16:32:12.054 INFO:teuthology.orchestra.run.smithi001.stderr:Failed to start ntpd.service: Unit ntpd.service not found. 2024-03-25T16:32:12.055 INFO:teuthology.orchestra.run.smithi115.stderr:Failed to start ntp.service: Unit ntp.service not found. 2024-03-25T16:32:12.072 INFO:teuthology.orchestra.run.smithi115.stderr:Failed to start ntpd.service: Unit ntpd.service not found. 2024-03-25T16:32:12.105 INFO:teuthology.orchestra.run.smithi001.stderr:bash: line 1: ntpq: command not found 2024-03-25T16:32:12.109 INFO:teuthology.orchestra.run.smithi001.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2024-03-25T16:32:12.109 INFO:teuthology.orchestra.run.smithi001.stdout:=============================================================================== 2024-03-25T16:32:12.109 INFO:teuthology.orchestra.run.smithi001.stdout:^? hv01.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-03-25T16:32:12.109 INFO:teuthology.orchestra.run.smithi001.stdout:^? hv02.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-03-25T16:32:12.109 INFO:teuthology.orchestra.run.smithi001.stdout:^? hv03.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-03-25T16:32:12.109 INFO:teuthology.orchestra.run.smithi001.stdout:^? hv04.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-03-25T16:32:12.124 INFO:teuthology.orchestra.run.smithi115.stderr:bash: line 1: ntpq: command not found 2024-03-25T16:32:12.127 INFO:teuthology.orchestra.run.smithi115.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2024-03-25T16:32:12.127 INFO:teuthology.orchestra.run.smithi115.stdout:=============================================================================== 2024-03-25T16:32:12.127 INFO:teuthology.orchestra.run.smithi115.stdout:^? hv01.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-03-25T16:32:12.127 INFO:teuthology.orchestra.run.smithi115.stdout:^? hv02.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-03-25T16:32:12.127 INFO:teuthology.orchestra.run.smithi115.stdout:^? hv03.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-03-25T16:32:12.127 INFO:teuthology.orchestra.run.smithi115.stdout:^? hv04.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-03-25T16:32:12.128 INFO:teuthology.run_tasks:Running task install... 2024-03-25T16:32:12.136 DEBUG:teuthology.task.install:project ceph 2024-03-25T16:32:12.137 DEBUG:teuthology.task.install:INSTALL overrides: {'ceph': {'debuginfo': True, 'flavor': 'default', 'sha1': '5dcb190da270440ef141d088eb27a9fdb8afc5f2'}} 2024-03-25T16:32:12.137 DEBUG:teuthology.task.install:config {'debuginfo': True, 'flavor': 'default', 'sha1': '5dcb190da270440ef141d088eb27a9fdb8afc5f2'} 2024-03-25T16:32:12.137 INFO:teuthology.task.install:Using flavor: default 2024-03-25T16:32:12.142 DEBUG:teuthology.task.install:Package list is: {'deb': ['ceph', 'cephadm', 'ceph-mds', 'ceph-mgr', 'ceph-common', 'ceph-fuse', 'ceph-test', 'ceph-volume', 'radosgw', 'python3-rados', 'python3-rgw', 'python3-cephfs', 'python3-rbd', 'libcephfs2', 'libcephfs-dev', 'librados2', 'librbd1', 'rbd-fuse', 'ceph-common-dbg', 'ceph-fuse-dbg', 'ceph-mds-dbg', 'ceph-mgr-dbg', 'ceph-mon-dbg', 'ceph-osd-dbg', 'ceph-test-dbg', 'libcephfs2-dbg', 'librados2-dbg', 'libradosstriper1-dbg', 'librbd1-dbg', 'librgw2-dbg', 'radosgw-dbg', 'rbd-fuse-dbg', 'rbd-mirror-dbg', 'rbd-nbd-dbg'], 'rpm': ['ceph-radosgw', 'ceph-test', 'ceph', 'ceph-base', 'cephadm', 'ceph-immutable-object-cache', 'ceph-mgr', 'ceph-mgr-dashboard', 'ceph-mgr-diskprediction-local', 'ceph-mgr-rook', 'ceph-mgr-cephadm', 'ceph-fuse', 'ceph-volume', 'librados-devel', 'libcephfs2', 'libcephfs-devel', 'librados2', 'librbd1', 'python3-rados', 'python3-rgw', 'python3-cephfs', 'python3-rbd', 'rbd-fuse', 'rbd-mirror', 'rbd-nbd', 'ceph-base-debuginfo', 'ceph-common-debuginfo', 'ceph-immutable-object-cache-debuginfo', 'ceph-radosgw-debuginfo', 'ceph-test-debuginfo', 'ceph-base-debuginfo', 'ceph-mgr-debuginfo', 'ceph-mds-debuginfo', 'ceph-mon-debuginfo', 'ceph-osd-debuginfo', 'ceph-fuse-debuginfo', 'librados-devel-debuginfo', 'libcephfs2-debuginfo', 'librados2-debuginfo', 'librbd1-debuginfo', 'python3-cephfs-debuginfo', 'python3-rados-debuginfo', 'python3-rbd-debuginfo', 'python3-rgw-debuginfo', 'rbd-fuse-debuginfo', 'rbd-mirror-debuginfo', 'rbd-nbd-debuginfo']} 2024-03-25T16:32:12.143 INFO:teuthology.task.install:extra packages: [] 2024-03-25T16:32:12.143 DEBUG:teuthology.task.install.rpm:_update_package_list_and_install: config is {'branch': None, 'cleanup': None, 'debuginfo': True, 'downgrade_packages': [], 'exclude_packages': [], 'extra_packages': [], 'extra_system_packages': [], 'extras': None, 'enable_coprs': [], 'flavor': 'default', 'install_ceph_packages': True, 'packages': {}, 'project': 'ceph', 'repos_only': False, 'sha1': '5dcb190da270440ef141d088eb27a9fdb8afc5f2', 'tag': None, 'wait_for_package': False} 2024-03-25T16:32:12.143 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F9%2Fx86_64&sha1=5dcb190da270440ef141d088eb27a9fdb8afc5f2 2024-03-25T16:32:12.145 DEBUG:teuthology.task.install.rpm:_update_package_list_and_install: config is {'branch': None, 'cleanup': None, 'debuginfo': True, 'downgrade_packages': [], 'exclude_packages': [], 'extra_packages': [], 'extra_system_packages': [], 'extras': None, 'enable_coprs': [], 'flavor': 'default', 'install_ceph_packages': True, 'packages': {}, 'project': 'ceph', 'repos_only': False, 'sha1': '5dcb190da270440ef141d088eb27a9fdb8afc5f2', 'tag': None, 'wait_for_package': False} 2024-03-25T16:32:12.146 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F9%2Fx86_64&sha1=5dcb190da270440ef141d088eb27a9fdb8afc5f2 2024-03-25T16:32:12.293 INFO:teuthology.task.install.rpm:Pulling from https://2.chacra.ceph.com/r/ceph/wip-amat-fix-encode-payload/5dcb190da270440ef141d088eb27a9fdb8afc5f2/centos/9/flavors/default/ 2024-03-25T16:32:12.293 INFO:teuthology.task.install.rpm:Package version is 19.0.0-1439.g5dcb190d 2024-03-25T16:32:12.362 INFO:teuthology.task.install.rpm:Pulling from https://2.chacra.ceph.com/r/ceph/wip-amat-fix-encode-payload/5dcb190da270440ef141d088eb27a9fdb8afc5f2/centos/9/flavors/default/ 2024-03-25T16:32:12.363 INFO:teuthology.task.install.rpm:Package version is 19.0.0-1439.g5dcb190d 2024-03-25T16:32:12.699 INFO:teuthology.packaging:Writing yum repo: [ceph] name=ceph packages for $basearch baseurl=https://2.chacra.ceph.com/r/ceph/wip-amat-fix-encode-payload/5dcb190da270440ef141d088eb27a9fdb8afc5f2/centos/9/flavors/default/$basearch enabled=1 gpgcheck=0 type=rpm-md [ceph-noarch] name=ceph noarch packages baseurl=https://2.chacra.ceph.com/r/ceph/wip-amat-fix-encode-payload/5dcb190da270440ef141d088eb27a9fdb8afc5f2/centos/9/flavors/default/noarch enabled=1 gpgcheck=0 type=rpm-md [ceph-source] name=ceph source packages baseurl=https://2.chacra.ceph.com/r/ceph/wip-amat-fix-encode-payload/5dcb190da270440ef141d088eb27a9fdb8afc5f2/centos/9/flavors/default/SRPMS enabled=1 gpgcheck=0 type=rpm-md 2024-03-25T16:32:12.699 DEBUG:teuthology.orchestra.run.smithi001:> set -ex 2024-03-25T16:32:12.699 DEBUG:teuthology.orchestra.run.smithi001:> sudo dd of=/etc/yum.repos.d/ceph.repo 2024-03-25T16:32:12.701 INFO:teuthology.packaging:Writing yum repo: [ceph] name=ceph packages for $basearch baseurl=https://2.chacra.ceph.com/r/ceph/wip-amat-fix-encode-payload/5dcb190da270440ef141d088eb27a9fdb8afc5f2/centos/9/flavors/default/$basearch enabled=1 gpgcheck=0 type=rpm-md [ceph-noarch] name=ceph noarch packages baseurl=https://2.chacra.ceph.com/r/ceph/wip-amat-fix-encode-payload/5dcb190da270440ef141d088eb27a9fdb8afc5f2/centos/9/flavors/default/noarch enabled=1 gpgcheck=0 type=rpm-md [ceph-source] name=ceph source packages baseurl=https://2.chacra.ceph.com/r/ceph/wip-amat-fix-encode-payload/5dcb190da270440ef141d088eb27a9fdb8afc5f2/centos/9/flavors/default/SRPMS enabled=1 gpgcheck=0 type=rpm-md 2024-03-25T16:32:12.701 DEBUG:teuthology.orchestra.run.smithi115:> set -ex 2024-03-25T16:32:12.701 DEBUG:teuthology.orchestra.run.smithi115:> sudo dd of=/etc/yum.repos.d/ceph.repo 2024-03-25T16:32:12.730 INFO:teuthology.task.install.rpm:Installing packages: ceph-radosgw, ceph-test, ceph, ceph-base, cephadm, ceph-immutable-object-cache, ceph-mgr, ceph-mgr-dashboard, ceph-mgr-diskprediction-local, ceph-mgr-rook, ceph-mgr-cephadm, ceph-fuse, ceph-volume, librados-devel, libcephfs2, libcephfs-devel, librados2, librbd1, python3-rados, python3-rgw, python3-cephfs, python3-rbd, rbd-fuse, rbd-mirror, rbd-nbd, ceph-base-debuginfo, ceph-common-debuginfo, ceph-immutable-object-cache-debuginfo, ceph-radosgw-debuginfo, ceph-test-debuginfo, ceph-base-debuginfo, ceph-mgr-debuginfo, ceph-mds-debuginfo, ceph-mon-debuginfo, ceph-osd-debuginfo, ceph-fuse-debuginfo, librados-devel-debuginfo, libcephfs2-debuginfo, librados2-debuginfo, librbd1-debuginfo, python3-cephfs-debuginfo, python3-rados-debuginfo, python3-rbd-debuginfo, python3-rgw-debuginfo, rbd-fuse-debuginfo, rbd-mirror-debuginfo, rbd-nbd-debuginfo on remote rpm x86_64 2024-03-25T16:32:12.730 DEBUG:teuthology.orchestra.run.smithi115:> if test -f /etc/yum.repos.d/ceph.repo ; then sudo sed -i -e ':a;N;$!ba;s/enabled=1\ngpg/enabled=1\npriority=1\ngpg/g' -e 's;ref/[a-zA-Z0-9_-]*/;sha1/5dcb190da270440ef141d088eb27a9fdb8afc5f2/;g' /etc/yum.repos.d/ceph.repo ; fi 2024-03-25T16:32:12.731 INFO:teuthology.task.install.rpm:Installing packages: ceph-radosgw, ceph-test, ceph, ceph-base, cephadm, ceph-immutable-object-cache, ceph-mgr, ceph-mgr-dashboard, ceph-mgr-diskprediction-local, ceph-mgr-rook, ceph-mgr-cephadm, ceph-fuse, ceph-volume, librados-devel, libcephfs2, libcephfs-devel, librados2, librbd1, python3-rados, python3-rgw, python3-cephfs, python3-rbd, rbd-fuse, rbd-mirror, rbd-nbd, ceph-base-debuginfo, ceph-common-debuginfo, ceph-immutable-object-cache-debuginfo, ceph-radosgw-debuginfo, ceph-test-debuginfo, ceph-base-debuginfo, ceph-mgr-debuginfo, ceph-mds-debuginfo, ceph-mon-debuginfo, ceph-osd-debuginfo, ceph-fuse-debuginfo, librados-devel-debuginfo, libcephfs2-debuginfo, librados2-debuginfo, librbd1-debuginfo, python3-cephfs-debuginfo, python3-rados-debuginfo, python3-rbd-debuginfo, python3-rgw-debuginfo, rbd-fuse-debuginfo, rbd-mirror-debuginfo, rbd-nbd-debuginfo on remote rpm x86_64 2024-03-25T16:32:12.732 DEBUG:teuthology.orchestra.run.smithi001:> if test -f /etc/yum.repos.d/ceph.repo ; then sudo sed -i -e ':a;N;$!ba;s/enabled=1\ngpg/enabled=1\npriority=1\ngpg/g' -e 's;ref/[a-zA-Z0-9_-]*/;sha1/5dcb190da270440ef141d088eb27a9fdb8afc5f2/;g' /etc/yum.repos.d/ceph.repo ; fi 2024-03-25T16:32:12.799 DEBUG:teuthology.orchestra.run.smithi115:> sudo touch -a /etc/yum/pluginconf.d/priorities.conf ; test -e /etc/yum/pluginconf.d/priorities.conf.orig || sudo cp -af /etc/yum/pluginconf.d/priorities.conf /etc/yum/pluginconf.d/priorities.conf.orig 2024-03-25T16:32:12.800 DEBUG:teuthology.orchestra.run.smithi001:> sudo touch -a /etc/yum/pluginconf.d/priorities.conf ; test -e /etc/yum/pluginconf.d/priorities.conf.orig || sudo cp -af /etc/yum/pluginconf.d/priorities.conf /etc/yum/pluginconf.d/priorities.conf.orig 2024-03-25T16:32:12.880 DEBUG:teuthology.orchestra.run.smithi115:> grep check_obsoletes /etc/yum/pluginconf.d/priorities.conf && sudo sed -i 's/check_obsoletes.*0/check_obsoletes = 1/g' /etc/yum/pluginconf.d/priorities.conf || echo 'check_obsoletes = 1' | sudo tee -a /etc/yum/pluginconf.d/priorities.conf 2024-03-25T16:32:12.881 DEBUG:teuthology.orchestra.run.smithi001:> grep check_obsoletes /etc/yum/pluginconf.d/priorities.conf && sudo sed -i 's/check_obsoletes.*0/check_obsoletes = 1/g' /etc/yum/pluginconf.d/priorities.conf || echo 'check_obsoletes = 1' | sudo tee -a /etc/yum/pluginconf.d/priorities.conf 2024-03-25T16:32:12.947 INFO:teuthology.orchestra.run.smithi115.stdout:check_obsoletes = 1 2024-03-25T16:32:12.948 DEBUG:teuthology.orchestra.run.smithi115:> sudo yum clean all 2024-03-25T16:32:12.949 INFO:teuthology.orchestra.run.smithi001.stdout:check_obsoletes = 1 2024-03-25T16:32:12.950 DEBUG:teuthology.orchestra.run.smithi001:> sudo yum clean all 2024-03-25T16:32:13.183 INFO:teuthology.orchestra.run.smithi115.stdout:74 files removed 2024-03-25T16:32:13.205 INFO:teuthology.orchestra.run.smithi001.stdout:74 files removed 2024-03-25T16:32:13.210 DEBUG:teuthology.orchestra.run.smithi115:> sudo yum -y install ceph-radosgw 2024-03-25T16:32:13.233 DEBUG:teuthology.orchestra.run.smithi001:> sudo yum -y install ceph-radosgw 2024-03-25T16:32:14.462 INFO:teuthology.orchestra.run.smithi115.stdout:ceph packages for x86_64 86 kB/s | 82 kB 00:00 2024-03-25T16:32:14.560 INFO:teuthology.orchestra.run.smithi001.stdout:ceph packages for x86_64 80 kB/s | 82 kB 00:01 2024-03-25T16:32:15.354 INFO:teuthology.orchestra.run.smithi115.stdout:ceph noarch packages 12 kB/s | 10 kB 00:00 2024-03-25T16:32:15.452 INFO:teuthology.orchestra.run.smithi001.stdout:ceph noarch packages 12 kB/s | 10 kB 00:00 2024-03-25T16:32:16.229 INFO:teuthology.orchestra.run.smithi115.stdout:ceph source packages 2.3 kB/s | 1.9 kB 00:00 2024-03-25T16:32:16.319 INFO:teuthology.orchestra.run.smithi001.stdout:ceph source packages 2.3 kB/s | 1.9 kB 00:00 2024-03-25T16:32:16.657 INFO:teuthology.orchestra.run.smithi115.stdout:CentOS-9 - AppStream 17 MB/s | 6.7 MB 00:00 2024-03-25T16:32:16.813 INFO:teuthology.orchestra.run.smithi001.stdout:CentOS-9 - AppStream 15 MB/s | 6.7 MB 00:00 2024-03-25T16:32:18.165 INFO:teuthology.orchestra.run.smithi115.stdout:CentOS-9 - BaseOS 5.2 MB/s | 2.0 MB 00:00 2024-03-25T16:32:18.264 INFO:teuthology.orchestra.run.smithi001.stdout:CentOS-9 - BaseOS 6.9 MB/s | 2.0 MB 00:00 2024-03-25T16:32:18.916 INFO:teuthology.orchestra.run.smithi115.stdout:CentOS Stream 9 - BaseOS 19 MB/s | 8.0 MB 00:00 2024-03-25T16:32:19.313 INFO:teuthology.orchestra.run.smithi001.stdout:CentOS Stream 9 - BaseOS 11 MB/s | 8.0 MB 00:00 2024-03-25T16:32:34.556 INFO:teuthology.orchestra.run.smithi115.stdout:CentOS Stream 9 - AppStream 1.3 MB/s | 19 MB 00:14 2024-03-25T16:32:34.813 INFO:teuthology.orchestra.run.smithi001.stdout:CentOS Stream 9 - AppStream 1.3 MB/s | 19 MB 00:14 2024-03-25T16:32:38.665 INFO:teuthology.orchestra.run.smithi115.stdout:CentOS Stream 9 - CRB 8.8 MB/s | 6.1 MB 00:00 2024-03-25T16:32:40.400 INFO:teuthology.orchestra.run.smithi115.stdout:CentOS Stream 9 - Extras packages 39 kB/s | 16 kB 00:00 2024-03-25T16:32:40.472 INFO:teuthology.orchestra.run.smithi001.stdout:CentOS Stream 9 - CRB 3.1 MB/s | 6.1 MB 00:02 2024-03-25T16:32:41.175 INFO:teuthology.orchestra.run.smithi115.stdout:Extra Packages for Enterprise Linux 31 MB/s | 21 MB 00:00 2024-03-25T16:32:42.222 INFO:teuthology.orchestra.run.smithi001.stdout:CentOS Stream 9 - Extras packages 30 kB/s | 16 kB 00:00 2024-03-25T16:32:43.974 INFO:teuthology.orchestra.run.smithi001.stdout:Extra Packages for Enterprise Linux 13 MB/s | 21 MB 00:01 2024-03-25T16:32:46.233 INFO:teuthology.orchestra.run.smithi115.stdout:lab-extras 6.8 kB/s | 1.7 kB 00:00 2024-03-25T16:32:48.653 INFO:teuthology.orchestra.run.smithi115.stdout:Dependencies resolved. 2024-03-25T16:32:48.661 INFO:teuthology.orchestra.run.smithi115.stdout:================================================================================ 2024-03-25T16:32:48.662 INFO:teuthology.orchestra.run.smithi115.stdout: Package Arch Version Repository Size 2024-03-25T16:32:48.662 INFO:teuthology.orchestra.run.smithi115.stdout:================================================================================ 2024-03-25T16:32:48.662 INFO:teuthology.orchestra.run.smithi115.stdout:Installing: 2024-03-25T16:32:48.662 INFO:teuthology.orchestra.run.smithi115.stdout: ceph-radosgw x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 9.7 M 2024-03-25T16:32:48.662 INFO:teuthology.orchestra.run.smithi115.stdout:Upgrading: 2024-03-25T16:32:48.662 INFO:teuthology.orchestra.run.smithi115.stdout: librados2 x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 3.4 M 2024-03-25T16:32:48.662 INFO:teuthology.orchestra.run.smithi115.stdout: librbd1 x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 3.1 M 2024-03-25T16:32:48.662 INFO:teuthology.orchestra.run.smithi115.stdout:Installing dependencies: 2024-03-25T16:32:48.662 INFO:teuthology.orchestra.run.smithi115.stdout: boost-program-options 2024-03-25T16:32:48.662 INFO:teuthology.orchestra.run.smithi115.stdout: x86_64 1.75.0-8.el9 CentOS-AppStream 106 k 2024-03-25T16:32:48.662 INFO:teuthology.orchestra.run.smithi115.stdout: ceph-base x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 5.3 M 2024-03-25T16:32:48.662 INFO:teuthology.orchestra.run.smithi115.stdout: ceph-common x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 20 M 2024-03-25T16:32:48.663 INFO:teuthology.orchestra.run.smithi115.stdout: ceph-selinux x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 25 k 2024-03-25T16:32:48.663 INFO:teuthology.orchestra.run.smithi115.stdout: gperftools-libs x86_64 2.9.1-2.el9 epel 309 k 2024-03-25T16:32:48.663 INFO:teuthology.orchestra.run.smithi115.stdout: libarrow x86_64 9.0.0-10.el9 epel 4.4 M 2024-03-25T16:32:48.663 INFO:teuthology.orchestra.run.smithi115.stdout: libarrow-doc noarch 9.0.0-10.el9 epel 26 k 2024-03-25T16:32:48.663 INFO:teuthology.orchestra.run.smithi115.stdout: libcephfs2 x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 742 k 2024-03-25T16:32:48.663 INFO:teuthology.orchestra.run.smithi115.stdout: liboath x86_64 2.6.7-2.el9 epel 49 k 2024-03-25T16:32:48.663 INFO:teuthology.orchestra.run.smithi115.stdout: libpmemobj x86_64 1.12.1-1.el9 CentOS-AppStream 159 k 2024-03-25T16:32:48.663 INFO:teuthology.orchestra.run.smithi115.stdout: librabbitmq x86_64 0.11.0-7.el9 CentOS-AppStream 44 k 2024-03-25T16:32:48.663 INFO:teuthology.orchestra.run.smithi115.stdout: libradosstriper1 x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 498 k 2024-03-25T16:32:48.663 INFO:teuthology.orchestra.run.smithi115.stdout: librdkafka x86_64 1.6.1-102.el9 CentOS-AppStream 661 k 2024-03-25T16:32:48.663 INFO:teuthology.orchestra.run.smithi115.stdout: librgw2 x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 5.1 M 2024-03-25T16:32:48.663 INFO:teuthology.orchestra.run.smithi115.stdout: libunwind x86_64 1.6.2-1.el9 epel 67 k 2024-03-25T16:32:48.663 INFO:teuthology.orchestra.run.smithi115.stdout: lttng-ust x86_64 2.12.0-6.el9 CentOS-AppStream 282 k 2024-03-25T16:32:48.663 INFO:teuthology.orchestra.run.smithi115.stdout: parquet-libs x86_64 9.0.0-10.el9 epel 838 k 2024-03-25T16:32:48.664 INFO:teuthology.orchestra.run.smithi115.stdout: python3-ceph-argparse 2024-03-25T16:32:48.664 INFO:teuthology.orchestra.run.smithi115.stdout: x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 45 k 2024-03-25T16:32:48.664 INFO:teuthology.orchestra.run.smithi115.stdout: python3-ceph-common x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 131 k 2024-03-25T16:32:48.664 INFO:teuthology.orchestra.run.smithi115.stdout: python3-cephfs x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 162 k 2024-03-25T16:32:48.664 INFO:teuthology.orchestra.run.smithi115.stdout: python3-prettytable noarch 0.7.2-27.el9 CentOS-AppStream 41 k 2024-03-25T16:32:48.664 INFO:teuthology.orchestra.run.smithi115.stdout: python3-rados x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 320 k 2024-03-25T16:32:48.664 INFO:teuthology.orchestra.run.smithi115.stdout: python3-rbd x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 298 k 2024-03-25T16:32:48.664 INFO:teuthology.orchestra.run.smithi115.stdout: python3-rgw x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 99 k 2024-03-25T16:32:48.664 INFO:teuthology.orchestra.run.smithi115.stdout: qatlib x86_64 23.11.0-1.el9 appstream 216 k 2024-03-25T16:32:48.664 INFO:teuthology.orchestra.run.smithi115.stdout: qatzip-libs x86_64 1.1.2-2.el9 appstream 44 k 2024-03-25T16:32:48.664 INFO:teuthology.orchestra.run.smithi115.stdout: re2 x86_64 1:20211101-3.el9 epel 192 k 2024-03-25T16:32:48.664 INFO:teuthology.orchestra.run.smithi115.stdout: thrift x86_64 0.15.0-2.el9 epel 1.6 M 2024-03-25T16:32:48.664 INFO:teuthology.orchestra.run.smithi115.stdout:Installing weak dependencies: 2024-03-25T16:32:48.665 INFO:teuthology.orchestra.run.smithi115.stdout: qatlib-service x86_64 23.11.0-1.el9 appstream 35 k 2024-03-25T16:32:48.665 INFO:teuthology.orchestra.run.smithi115.stdout: 2024-03-25T16:32:48.665 INFO:teuthology.orchestra.run.smithi115.stdout:Transaction Summary 2024-03-25T16:32:48.665 INFO:teuthology.orchestra.run.smithi115.stdout:================================================================================ 2024-03-25T16:32:48.665 INFO:teuthology.orchestra.run.smithi115.stdout:Install 30 Packages 2024-03-25T16:32:48.665 INFO:teuthology.orchestra.run.smithi115.stdout:Upgrade 2 Packages 2024-03-25T16:32:48.665 INFO:teuthology.orchestra.run.smithi115.stdout: 2024-03-25T16:32:48.666 INFO:teuthology.orchestra.run.smithi115.stdout:Total download size: 58 M 2024-03-25T16:32:48.666 INFO:teuthology.orchestra.run.smithi115.stdout:Downloading Packages: 2024-03-25T16:32:49.341 INFO:teuthology.orchestra.run.smithi001.stdout:lab-extras 26 kB/s | 1.7 kB 00:00 2024-03-25T16:32:51.973 INFO:teuthology.orchestra.run.smithi001.stdout:Dependencies resolved. 2024-03-25T16:32:51.980 INFO:teuthology.orchestra.run.smithi001.stdout:================================================================================ 2024-03-25T16:32:51.980 INFO:teuthology.orchestra.run.smithi001.stdout: Package Arch Version Repository Size 2024-03-25T16:32:51.981 INFO:teuthology.orchestra.run.smithi001.stdout:================================================================================ 2024-03-25T16:32:51.981 INFO:teuthology.orchestra.run.smithi001.stdout:Installing: 2024-03-25T16:32:51.981 INFO:teuthology.orchestra.run.smithi001.stdout: ceph-radosgw x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 9.7 M 2024-03-25T16:32:51.981 INFO:teuthology.orchestra.run.smithi001.stdout:Upgrading: 2024-03-25T16:32:51.981 INFO:teuthology.orchestra.run.smithi001.stdout: librados2 x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 3.4 M 2024-03-25T16:32:51.981 INFO:teuthology.orchestra.run.smithi001.stdout: librbd1 x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 3.1 M 2024-03-25T16:32:51.981 INFO:teuthology.orchestra.run.smithi001.stdout:Installing dependencies: 2024-03-25T16:32:51.981 INFO:teuthology.orchestra.run.smithi001.stdout: boost-program-options 2024-03-25T16:32:51.981 INFO:teuthology.orchestra.run.smithi001.stdout: x86_64 1.75.0-8.el9 CentOS-AppStream 106 k 2024-03-25T16:32:51.981 INFO:teuthology.orchestra.run.smithi001.stdout: ceph-base x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 5.3 M 2024-03-25T16:32:51.981 INFO:teuthology.orchestra.run.smithi001.stdout: ceph-common x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 20 M 2024-03-25T16:32:51.981 INFO:teuthology.orchestra.run.smithi001.stdout: ceph-selinux x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 25 k 2024-03-25T16:32:51.982 INFO:teuthology.orchestra.run.smithi001.stdout: gperftools-libs x86_64 2.9.1-2.el9 epel 309 k 2024-03-25T16:32:51.982 INFO:teuthology.orchestra.run.smithi001.stdout: libarrow x86_64 9.0.0-10.el9 epel 4.4 M 2024-03-25T16:32:51.982 INFO:teuthology.orchestra.run.smithi001.stdout: libarrow-doc noarch 9.0.0-10.el9 epel 26 k 2024-03-25T16:32:51.982 INFO:teuthology.orchestra.run.smithi001.stdout: libcephfs2 x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 742 k 2024-03-25T16:32:51.982 INFO:teuthology.orchestra.run.smithi001.stdout: liboath x86_64 2.6.7-2.el9 epel 49 k 2024-03-25T16:32:51.982 INFO:teuthology.orchestra.run.smithi001.stdout: libpmemobj x86_64 1.12.1-1.el9 CentOS-AppStream 159 k 2024-03-25T16:32:51.982 INFO:teuthology.orchestra.run.smithi001.stdout: librabbitmq x86_64 0.11.0-7.el9 CentOS-AppStream 44 k 2024-03-25T16:32:51.982 INFO:teuthology.orchestra.run.smithi001.stdout: libradosstriper1 x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 498 k 2024-03-25T16:32:51.982 INFO:teuthology.orchestra.run.smithi001.stdout: librdkafka x86_64 1.6.1-102.el9 CentOS-AppStream 661 k 2024-03-25T16:32:51.982 INFO:teuthology.orchestra.run.smithi001.stdout: librgw2 x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 5.1 M 2024-03-25T16:32:51.983 INFO:teuthology.orchestra.run.smithi001.stdout: libunwind x86_64 1.6.2-1.el9 epel 67 k 2024-03-25T16:32:51.983 INFO:teuthology.orchestra.run.smithi001.stdout: lttng-ust x86_64 2.12.0-6.el9 CentOS-AppStream 282 k 2024-03-25T16:32:51.983 INFO:teuthology.orchestra.run.smithi001.stdout: parquet-libs x86_64 9.0.0-10.el9 epel 838 k 2024-03-25T16:32:51.983 INFO:teuthology.orchestra.run.smithi001.stdout: python3-ceph-argparse 2024-03-25T16:32:51.983 INFO:teuthology.orchestra.run.smithi001.stdout: x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 45 k 2024-03-25T16:32:51.983 INFO:teuthology.orchestra.run.smithi001.stdout: python3-ceph-common x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 131 k 2024-03-25T16:32:51.983 INFO:teuthology.orchestra.run.smithi001.stdout: python3-cephfs x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 162 k 2024-03-25T16:32:51.983 INFO:teuthology.orchestra.run.smithi001.stdout: python3-prettytable noarch 0.7.2-27.el9 CentOS-AppStream 41 k 2024-03-25T16:32:51.983 INFO:teuthology.orchestra.run.smithi001.stdout: python3-rados x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 320 k 2024-03-25T16:32:51.983 INFO:teuthology.orchestra.run.smithi001.stdout: python3-rbd x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 298 k 2024-03-25T16:32:51.984 INFO:teuthology.orchestra.run.smithi001.stdout: python3-rgw x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 99 k 2024-03-25T16:32:51.984 INFO:teuthology.orchestra.run.smithi001.stdout: qatlib x86_64 23.11.0-1.el9 appstream 216 k 2024-03-25T16:32:51.984 INFO:teuthology.orchestra.run.smithi001.stdout: qatzip-libs x86_64 1.1.2-2.el9 appstream 44 k 2024-03-25T16:32:51.984 INFO:teuthology.orchestra.run.smithi001.stdout: re2 x86_64 1:20211101-3.el9 epel 192 k 2024-03-25T16:32:51.984 INFO:teuthology.orchestra.run.smithi001.stdout: thrift x86_64 0.15.0-2.el9 epel 1.6 M 2024-03-25T16:32:51.984 INFO:teuthology.orchestra.run.smithi001.stdout:Installing weak dependencies: 2024-03-25T16:32:51.984 INFO:teuthology.orchestra.run.smithi001.stdout: qatlib-service x86_64 23.11.0-1.el9 appstream 35 k 2024-03-25T16:32:51.984 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-25T16:32:51.984 INFO:teuthology.orchestra.run.smithi001.stdout:Transaction Summary 2024-03-25T16:32:51.984 INFO:teuthology.orchestra.run.smithi001.stdout:================================================================================ 2024-03-25T16:32:51.984 INFO:teuthology.orchestra.run.smithi001.stdout:Install 30 Packages 2024-03-25T16:32:51.985 INFO:teuthology.orchestra.run.smithi001.stdout:Upgrade 2 Packages 2024-03-25T16:32:51.985 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-25T16:32:51.985 INFO:teuthology.orchestra.run.smithi001.stdout:Total download size: 58 M 2024-03-25T16:32:51.985 INFO:teuthology.orchestra.run.smithi001.stdout:Downloading Packages: 2024-03-25T16:33:06.082 INFO:teuthology.orchestra.run.smithi115.stdout:(1/32): ceph-base-19.0.0-1439.g5dcb190d.el9.x86 318 kB/s | 5.3 MB 00:17 2024-03-25T16:33:06.256 INFO:teuthology.orchestra.run.smithi115.stdout:(2/32): ceph-selinux-19.0.0-1439.g5dcb190d.el9. 144 kB/s | 25 kB 00:00 2024-03-25T16:33:08.298 INFO:teuthology.orchestra.run.smithi115.stdout:(3/32): libcephfs2-19.0.0-1439.g5dcb190d.el9.x8 363 kB/s | 742 kB 00:02 2024-03-25T16:33:09.498 INFO:teuthology.orchestra.run.smithi115.stdout:(4/32): libradosstriper1-19.0.0-1439.g5dcb190d. 415 kB/s | 498 kB 00:01 2024-03-25T16:33:10.240 INFO:teuthology.orchestra.run.smithi115.stdout:(5/32): ceph-radosgw-19.0.0-1439.g5dcb190d.el9. 467 kB/s | 9.7 MB 00:21 2024-03-25T16:33:10.765 INFO:teuthology.orchestra.run.smithi115.stdout:(6/32): python3-ceph-argparse-19.0.0-1439.g5dcb 86 kB/s | 45 kB 00:00 2024-03-25T16:33:11.473 INFO:teuthology.orchestra.run.smithi115.stdout:(7/32): python3-ceph-common-19.0.0-1439.g5dcb19 185 kB/s | 131 kB 00:00 2024-03-25T16:33:12.049 INFO:teuthology.orchestra.run.smithi115.stdout:(8/32): python3-cephfs-19.0.0-1439.g5dcb190d.el 282 kB/s | 162 kB 00:00 2024-03-25T16:33:13.165 INFO:teuthology.orchestra.run.smithi115.stdout:(9/32): python3-rados-19.0.0-1439.g5dcb190d.el9 287 kB/s | 320 kB 00:01 2024-03-25T16:33:13.166 INFO:teuthology.orchestra.run.smithi001.stdout:(1/32): ceph-base-19.0.0-1439.g5dcb190d.el9.x86 261 kB/s | 5.3 MB 00:20 2024-03-25T16:33:13.282 INFO:teuthology.orchestra.run.smithi001.stdout:(2/32): ceph-selinux-19.0.0-1439.g5dcb190d.el9. 216 kB/s | 25 kB 00:00 2024-03-25T16:33:13.982 INFO:teuthology.orchestra.run.smithi115.stdout:(10/32): python3-rbd-19.0.0-1439.g5dcb190d.el9. 365 kB/s | 298 kB 00:00 2024-03-25T16:33:14.324 INFO:teuthology.orchestra.run.smithi115.stdout:(11/32): python3-rgw-19.0.0-1439.g5dcb190d.el9. 291 kB/s | 99 kB 00:00 2024-03-25T16:33:14.524 INFO:teuthology.orchestra.run.smithi115.stdout:(12/32): boost-program-options-1.75.0-8.el9.x86 530 kB/s | 106 kB 00:00 2024-03-25T16:33:14.666 INFO:teuthology.orchestra.run.smithi115.stdout:(13/32): libpmemobj-1.12.1-1.el9.x86_64.rpm 1.1 MB/s | 159 kB 00:00 2024-03-25T16:33:14.691 INFO:teuthology.orchestra.run.smithi115.stdout:(14/32): librabbitmq-0.11.0-7.el9.x86_64.rpm 1.7 MB/s | 44 kB 00:00 2024-03-25T16:33:14.758 INFO:teuthology.orchestra.run.smithi115.stdout:(15/32): librdkafka-1.6.1-102.el9.x86_64.rpm 9.7 MB/s | 661 kB 00:00 2024-03-25T16:33:14.800 INFO:teuthology.orchestra.run.smithi115.stdout:(16/32): lttng-ust-2.12.0-6.el9.x86_64.rpm 6.6 MB/s | 282 kB 00:00 2024-03-25T16:33:14.825 INFO:teuthology.orchestra.run.smithi115.stdout:(17/32): python3-prettytable-0.7.2-27.el9.noarc 1.6 MB/s | 41 kB 00:00 2024-03-25T16:33:14.976 INFO:teuthology.orchestra.run.smithi115.stdout:(18/32): qatlib-23.11.0-1.el9.x86_64.rpm 1.4 MB/s | 216 kB 00:00 2024-03-25T16:33:15.018 INFO:teuthology.orchestra.run.smithi115.stdout:(19/32): qatlib-service-23.11.0-1.el9.x86_64.rp 835 kB/s | 35 kB 00:00 2024-03-25T16:33:15.060 INFO:teuthology.orchestra.run.smithi115.stdout:(20/32): qatzip-libs-1.1.2-2.el9.x86_64.rpm 1.0 MB/s | 44 kB 00:00 2024-03-25T16:33:15.293 INFO:teuthology.orchestra.run.smithi115.stdout:(21/32): gperftools-libs-2.9.1-2.el9.x86_64.rpm 1.3 MB/s | 309 kB 00:00 2024-03-25T16:33:15.535 INFO:teuthology.orchestra.run.smithi115.stdout:(22/32): libarrow-9.0.0-10.el9.x86_64.rpm 18 MB/s | 4.4 MB 00:00 2024-03-25T16:33:15.585 INFO:teuthology.orchestra.run.smithi115.stdout:(23/32): libarrow-doc-9.0.0-10.el9.noarch.rpm 521 kB/s | 26 kB 00:00 2024-03-25T16:33:15.636 INFO:teuthology.orchestra.run.smithi115.stdout:(24/32): liboath-2.6.7-2.el9.x86_64.rpm 969 kB/s | 49 kB 00:00 2024-03-25T16:33:15.686 INFO:teuthology.orchestra.run.smithi115.stdout:(25/32): libunwind-1.6.2-1.el9.x86_64.rpm 1.3 MB/s | 67 kB 00:00 2024-03-25T16:33:15.753 INFO:teuthology.orchestra.run.smithi115.stdout:(26/32): parquet-libs-9.0.0-10.el9.x86_64.rpm 12 MB/s | 838 kB 00:00 2024-03-25T16:33:15.837 INFO:teuthology.orchestra.run.smithi115.stdout:(27/32): re2-20211101-3.el9.x86_64.rpm 2.2 MB/s | 192 kB 00:00 2024-03-25T16:33:15.912 INFO:teuthology.orchestra.run.smithi115.stdout:(28/32): thrift-0.15.0-2.el9.x86_64.rpm 21 MB/s | 1.6 MB 00:00 2024-03-25T16:33:16.182 INFO:teuthology.orchestra.run.smithi001.stdout:(3/32): libcephfs2-19.0.0-1439.g5dcb190d.el9.x8 256 kB/s | 742 kB 00:02 2024-03-25T16:33:17.816 INFO:teuthology.orchestra.run.smithi001.stdout:(4/32): libradosstriper1-19.0.0-1439.g5dcb190d. 305 kB/s | 498 kB 00:01 2024-03-25T16:33:24.194 INFO:teuthology.orchestra.run.smithi115.stdout:(29/32): librados2-19.0.0-1439.g5dcb190d.el9.x8 417 kB/s | 3.4 MB 00:08 2024-03-25T16:33:29.090 INFO:teuthology.orchestra.run.smithi001.stdout:(5/32): ceph-radosgw-19.0.0-1439.g5dcb190d.el9. 270 kB/s | 9.7 MB 00:36 2024-03-25T16:33:29.457 INFO:teuthology.orchestra.run.smithi001.stdout:(6/32): python3-ceph-argparse-19.0.0-1439.g5dcb 123 kB/s | 45 kB 00:00 2024-03-25T16:33:30.116 INFO:teuthology.orchestra.run.smithi001.stdout:(7/32): python3-ceph-common-19.0.0-1439.g5dcb19 199 kB/s | 131 kB 00:00 2024-03-25T16:33:30.774 INFO:teuthology.orchestra.run.smithi001.stdout:(8/32): python3-cephfs-19.0.0-1439.g5dcb190d.el 246 kB/s | 162 kB 00:00 2024-03-25T16:33:30.810 INFO:teuthology.orchestra.run.smithi115.stdout:(30/32): librgw2-19.0.0-1439.g5dcb190d.el9.x86_ 245 kB/s | 5.1 MB 00:21 2024-03-25T16:33:31.782 INFO:teuthology.orchestra.run.smithi001.stdout:(9/32): python3-rados-19.0.0-1439.g5dcb190d.el9 318 kB/s | 320 kB 00:01 2024-03-25T16:33:32.266 INFO:teuthology.orchestra.run.smithi001.stdout:(10/32): librgw2-19.0.0-1439.g5dcb190d.el9.x86_ 361 kB/s | 5.1 MB 00:14 2024-03-25T16:33:33.091 INFO:teuthology.orchestra.run.smithi001.stdout:(11/32): python3-rgw-19.0.0-1439.g5dcb190d.el9. 120 kB/s | 99 kB 00:00 2024-03-25T16:33:33.225 INFO:teuthology.orchestra.run.smithi001.stdout:(12/32): boost-program-options-1.75.0-8.el9.x86 795 kB/s | 106 kB 00:00 2024-03-25T16:33:33.267 INFO:teuthology.orchestra.run.smithi001.stdout:(13/32): libpmemobj-1.12.1-1.el9.x86_64.rpm 3.7 MB/s | 159 kB 00:00 2024-03-25T16:33:33.292 INFO:teuthology.orchestra.run.smithi001.stdout:(14/32): librabbitmq-0.11.0-7.el9.x86_64.rpm 1.7 MB/s | 44 kB 00:00 2024-03-25T16:33:33.351 INFO:teuthology.orchestra.run.smithi001.stdout:(15/32): librdkafka-1.6.1-102.el9.x86_64.rpm 11 MB/s | 661 kB 00:00 2024-03-25T16:33:33.393 INFO:teuthology.orchestra.run.smithi001.stdout:(16/32): lttng-ust-2.12.0-6.el9.x86_64.rpm 6.6 MB/s | 282 kB 00:00 2024-03-25T16:33:33.427 INFO:teuthology.orchestra.run.smithi001.stdout:(17/32): python3-prettytable-0.7.2-27.el9.noarc 1.2 MB/s | 41 kB 00:00 2024-03-25T16:33:33.577 INFO:teuthology.orchestra.run.smithi001.stdout:(18/32): qatlib-23.11.0-1.el9.x86_64.rpm 1.4 MB/s | 216 kB 00:00 2024-03-25T16:33:33.619 INFO:teuthology.orchestra.run.smithi001.stdout:(19/32): qatlib-service-23.11.0-1.el9.x86_64.rp 837 kB/s | 35 kB 00:00 2024-03-25T16:33:33.661 INFO:teuthology.orchestra.run.smithi001.stdout:(20/32): qatzip-libs-1.1.2-2.el9.x86_64.rpm 1.0 MB/s | 44 kB 00:00 2024-03-25T16:33:33.844 INFO:teuthology.orchestra.run.smithi001.stdout:(21/32): python3-rbd-19.0.0-1439.g5dcb190d.el9. 144 kB/s | 298 kB 00:02 2024-03-25T16:33:34.084 INFO:teuthology.orchestra.run.smithi115.stdout:(31/32): librbd1-19.0.0-1439.g5dcb190d.el9.x86_ 324 kB/s | 3.1 MB 00:09 2024-03-25T16:33:34.203 INFO:teuthology.orchestra.run.smithi001.stdout:(22/32): gperftools-libs-2.9.1-2.el9.x86_64.rpm 570 kB/s | 309 kB 00:00 2024-03-25T16:33:34.370 INFO:teuthology.orchestra.run.smithi001.stdout:(23/32): libarrow-doc-9.0.0-10.el9.noarch.rpm 156 kB/s | 26 kB 00:00 2024-03-25T16:33:34.428 INFO:teuthology.orchestra.run.smithi001.stdout:(24/32): libarrow-9.0.0-10.el9.x86_64.rpm 7.6 MB/s | 4.4 MB 00:00 2024-03-25T16:33:34.445 INFO:teuthology.orchestra.run.smithi001.stdout:(25/32): liboath-2.6.7-2.el9.x86_64.rpm 650 kB/s | 49 kB 00:00 2024-03-25T16:33:34.496 INFO:teuthology.orchestra.run.smithi001.stdout:(26/32): libunwind-1.6.2-1.el9.x86_64.rpm 1.0 MB/s | 67 kB 00:00 2024-03-25T16:33:34.529 INFO:teuthology.orchestra.run.smithi001.stdout:(27/32): parquet-libs-9.0.0-10.el9.x86_64.rpm 9.8 MB/s | 838 kB 00:00 2024-03-25T16:33:34.571 INFO:teuthology.orchestra.run.smithi001.stdout:(28/32): re2-20211101-3.el9.x86_64.rpm 2.5 MB/s | 192 kB 00:00 2024-03-25T16:33:34.638 INFO:teuthology.orchestra.run.smithi001.stdout:(29/32): thrift-0.15.0-2.el9.x86_64.rpm 15 MB/s | 1.6 MB 00:00 2024-03-25T16:33:38.055 INFO:teuthology.orchestra.run.smithi001.stdout:(30/32): ceph-common-19.0.0-1439.g5dcb190d.el9. 444 kB/s | 20 MB 00:45 2024-03-25T16:33:46.854 INFO:teuthology.orchestra.run.smithi001.stdout:(31/32): librbd1-19.0.0-1439.g5dcb190d.el9.x86_ 262 kB/s | 3.1 MB 00:12 2024-03-25T16:33:48.563 INFO:teuthology.orchestra.run.smithi001.stdout:(32/32): librados2-19.0.0-1439.g5dcb190d.el9.x8 247 kB/s | 3.4 MB 00:13 2024-03-25T16:33:48.567 INFO:teuthology.orchestra.run.smithi001.stdout:-------------------------------------------------------------------------------- 2024-03-25T16:33:48.568 INFO:teuthology.orchestra.run.smithi001.stdout:Total 1.0 MB/s | 58 MB 00:56 2024-03-25T16:33:48.602 INFO:teuthology.orchestra.run.smithi001.stdout:Running transaction check 2024-03-25T16:33:48.645 INFO:teuthology.orchestra.run.smithi001.stdout:Transaction check succeeded. 2024-03-25T16:33:48.645 INFO:teuthology.orchestra.run.smithi001.stdout:Running transaction test 2024-03-25T16:33:49.147 INFO:teuthology.orchestra.run.smithi001.stdout:Transaction test succeeded. 2024-03-25T16:33:49.148 INFO:teuthology.orchestra.run.smithi001.stdout:Running transaction 2024-03-25T16:33:50.765 INFO:teuthology.orchestra.run.smithi001.stdout: Preparing : 1/1 2024-03-25T16:33:50.865 INFO:teuthology.orchestra.run.smithi001.stdout: Installing : lttng-ust-2.12.0-6.el9.x86_64 1/34 2024-03-25T16:33:50.892 INFO:teuthology.orchestra.run.smithi001.stdout: Installing : librdkafka-1.6.1-102.el9.x86_64 2/34 2024-03-25T16:33:50.918 INFO:teuthology.orchestra.run.smithi001.stdout: Installing : librabbitmq-0.11.0-7.el9.x86_64 3/34 2024-03-25T16:33:50.940 INFO:teuthology.orchestra.run.smithi001.stdout: Installing : qatlib-service-23.11.0-1.el9.x86_64 4/34 2024-03-25T16:33:50.957 INFO:teuthology.orchestra.run.smithi001.stdout: Running scriptlet: qatlib-service-23.11.0-1.el9.x86_64 4/34 2024-03-25T16:33:51.115 INFO:teuthology.orchestra.run.smithi001.stdout: Running scriptlet: qatlib-23.11.0-1.el9.x86_64 5/34 2024-03-25T16:33:51.172 INFO:teuthology.orchestra.run.smithi001.stdout: Installing : qatlib-23.11.0-1.el9.x86_64 5/34 2024-03-25T16:33:51.568 INFO:teuthology.orchestra.run.smithi001.stdout: Installing : thrift-0.15.0-2.el9.x86_64 6/34 2024-03-25T16:33:51.602 INFO:teuthology.orchestra.run.smithi001.stdout: Upgrading : librados2-2:19.0.0-1439.g5dcb190d.el9.x86_64 7/34 2024-03-25T16:33:51.681 INFO:teuthology.orchestra.run.smithi001.stdout: Running scriptlet: librados2-2:19.0.0-1439.g5dcb190d.el9.x86_64 7/34 2024-03-25T16:33:51.769 INFO:teuthology.orchestra.run.smithi001.stdout: Installing : python3-rados-2:19.0.0-1439.g5dcb190d.el9.x86_64 8/34 2024-03-25T16:33:51.805 INFO:teuthology.orchestra.run.smithi001.stdout: Installing : libcephfs2-2:19.0.0-1439.g5dcb190d.el9.x86_64 9/34 2024-03-25T16:33:51.869 INFO:teuthology.orchestra.run.smithi001.stdout: Running scriptlet: libcephfs2-2:19.0.0-1439.g5dcb190d.el9.x86_64 9/34 2024-03-25T16:33:51.905 INFO:teuthology.orchestra.run.smithi001.stdout: Installing : liboath-2.6.7-2.el9.x86_64 10/34 2024-03-25T16:33:51.959 INFO:teuthology.orchestra.run.smithi001.stdout: Installing : python3-ceph-argparse-2:19.0.0-1439.g5dcb190d.el9. 11/34 2024-03-25T16:33:52.035 INFO:teuthology.orchestra.run.smithi001.stdout: Installing : python3-cephfs-2:19.0.0-1439.g5dcb190d.el9.x86_64 12/34 2024-03-25T16:33:52.057 INFO:teuthology.orchestra.run.smithi001.stdout: Installing : libradosstriper1-2:19.0.0-1439.g5dcb190d.el9.x86_6 13/34 2024-03-25T16:33:52.103 INFO:teuthology.orchestra.run.smithi001.stdout: Running scriptlet: libradosstriper1-2:19.0.0-1439.g5dcb190d.el9.x86_6 13/34 2024-03-25T16:33:52.131 INFO:teuthology.orchestra.run.smithi001.stdout: Installing : qatzip-libs-1.1.2-2.el9.x86_64 14/34 2024-03-25T16:33:52.242 INFO:teuthology.orchestra.run.smithi001.stdout: Installing : re2-1:20211101-3.el9.x86_64 15/34 2024-03-25T16:33:52.281 INFO:teuthology.orchestra.run.smithi001.stdout: Installing : libarrow-9.0.0-10.el9.x86_64 16/34 2024-03-25T16:33:52.322 INFO:teuthology.orchestra.run.smithi001.stdout: Installing : libunwind-1.6.2-1.el9.x86_64 17/34 2024-03-25T16:33:52.346 INFO:teuthology.orchestra.run.smithi001.stdout: Installing : gperftools-libs-2.9.1-2.el9.x86_64 18/34 2024-03-25T16:33:52.375 INFO:teuthology.orchestra.run.smithi001.stdout: Installing : libarrow-doc-9.0.0-10.el9.noarch 19/34 2024-03-25T16:33:52.408 INFO:teuthology.orchestra.run.smithi001.stdout: Installing : python3-prettytable-0.7.2-27.el9.noarch 20/34 2024-03-25T16:33:52.737 INFO:teuthology.orchestra.run.smithi001.stdout: Installing : libpmemobj-1.12.1-1.el9.x86_64 21/34 2024-03-25T16:33:52.763 INFO:teuthology.orchestra.run.smithi001.stdout: Upgrading : librbd1-2:19.0.0-1439.g5dcb190d.el9.x86_64 22/34 2024-03-25T16:33:52.853 INFO:teuthology.orchestra.run.smithi001.stdout: Running scriptlet: librbd1-2:19.0.0-1439.g5dcb190d.el9.x86_64 22/34 2024-03-25T16:33:52.884 INFO:teuthology.orchestra.run.smithi001.stdout: Installing : python3-rbd-2:19.0.0-1439.g5dcb190d.el9.x86_64 23/34 2024-03-25T16:33:52.925 INFO:teuthology.orchestra.run.smithi001.stdout: Installing : boost-program-options-1.75.0-8.el9.x86_64 24/34 2024-03-25T16:33:53.402 INFO:teuthology.orchestra.run.smithi001.stdout: Installing : parquet-libs-9.0.0-10.el9.x86_64 25/34 2024-03-25T16:33:53.432 INFO:teuthology.orchestra.run.smithi001.stdout: Installing : librgw2-2:19.0.0-1439.g5dcb190d.el9.x86_64 26/34 2024-03-25T16:33:53.487 INFO:teuthology.orchestra.run.smithi001.stdout: Running scriptlet: librgw2-2:19.0.0-1439.g5dcb190d.el9.x86_64 26/34 2024-03-25T16:33:53.574 INFO:teuthology.orchestra.run.smithi001.stdout: Installing : python3-rgw-2:19.0.0-1439.g5dcb190d.el9.x86_64 27/34 2024-03-25T16:33:53.617 INFO:teuthology.orchestra.run.smithi001.stdout: Installing : python3-ceph-common-2:19.0.0-1439.g5dcb190d.el9.x8 28/34 2024-03-25T16:33:53.797 INFO:teuthology.orchestra.run.smithi115.stdout:(32/32): ceph-common-19.0.0-1439.g5dcb190d.el9. 314 kB/s | 20 MB 01:04 2024-03-25T16:33:53.800 INFO:teuthology.orchestra.run.smithi115.stdout:-------------------------------------------------------------------------------- 2024-03-25T16:33:53.801 INFO:teuthology.orchestra.run.smithi115.stdout:Total 909 kB/s | 58 MB 01:05 2024-03-25T16:33:53.831 INFO:teuthology.orchestra.run.smithi115.stdout:Running transaction check 2024-03-25T16:33:53.869 INFO:teuthology.orchestra.run.smithi115.stdout:Transaction check succeeded. 2024-03-25T16:33:53.869 INFO:teuthology.orchestra.run.smithi115.stdout:Running transaction test 2024-03-25T16:33:54.334 INFO:teuthology.orchestra.run.smithi115.stdout:Transaction test succeeded. 2024-03-25T16:33:54.335 INFO:teuthology.orchestra.run.smithi115.stdout:Running transaction 2024-03-25T16:33:55.802 INFO:teuthology.orchestra.run.smithi115.stdout: Preparing : 1/1 2024-03-25T16:33:55.885 INFO:teuthology.orchestra.run.smithi115.stdout: Installing : lttng-ust-2.12.0-6.el9.x86_64 1/34 2024-03-25T16:33:55.918 INFO:teuthology.orchestra.run.smithi115.stdout: Installing : librdkafka-1.6.1-102.el9.x86_64 2/34 2024-03-25T16:33:55.948 INFO:teuthology.orchestra.run.smithi115.stdout: Installing : librabbitmq-0.11.0-7.el9.x86_64 3/34 2024-03-25T16:33:55.974 INFO:teuthology.orchestra.run.smithi115.stdout: Installing : qatlib-service-23.11.0-1.el9.x86_64 4/34 2024-03-25T16:33:55.991 INFO:teuthology.orchestra.run.smithi115.stdout: Running scriptlet: qatlib-service-23.11.0-1.el9.x86_64 4/34 2024-03-25T16:33:56.096 INFO:teuthology.orchestra.run.smithi001.stdout: Running scriptlet: ceph-common-2:19.0.0-1439.g5dcb190d.el9.x86_64 29/34 2024-03-25T16:33:56.139 INFO:teuthology.orchestra.run.smithi001.stdout: Installing : ceph-common-2:19.0.0-1439.g5dcb190d.el9.x86_64 29/34 2024-03-25T16:33:56.177 INFO:teuthology.orchestra.run.smithi115.stdout: Running scriptlet: qatlib-23.11.0-1.el9.x86_64 5/34 2024-03-25T16:33:56.230 INFO:teuthology.orchestra.run.smithi115.stdout: Installing : qatlib-23.11.0-1.el9.x86_64 5/34 2024-03-25T16:33:56.563 INFO:teuthology.orchestra.run.smithi115.stdout: Installing : thrift-0.15.0-2.el9.x86_64 6/34 2024-03-25T16:33:56.603 INFO:teuthology.orchestra.run.smithi115.stdout: Upgrading : librados2-2:19.0.0-1439.g5dcb190d.el9.x86_64 7/34 2024-03-25T16:33:56.696 INFO:teuthology.orchestra.run.smithi115.stdout: Running scriptlet: librados2-2:19.0.0-1439.g5dcb190d.el9.x86_64 7/34 2024-03-25T16:33:56.730 INFO:teuthology.orchestra.run.smithi001.stdout: Running scriptlet: ceph-common-2:19.0.0-1439.g5dcb190d.el9.x86_64 29/34 2024-03-25T16:33:56.776 INFO:teuthology.orchestra.run.smithi001.stdout: Installing : ceph-base-2:19.0.0-1439.g5dcb190d.el9.x86_64 30/34 2024-03-25T16:33:56.793 INFO:teuthology.orchestra.run.smithi115.stdout: Installing : python3-rados-2:19.0.0-1439.g5dcb190d.el9.x86_64 8/34 2024-03-25T16:33:56.822 INFO:teuthology.orchestra.run.smithi115.stdout: Installing : libcephfs2-2:19.0.0-1439.g5dcb190d.el9.x86_64 9/34 2024-03-25T16:33:56.850 INFO:teuthology.orchestra.run.smithi001.stdout: Running scriptlet: ceph-base-2:19.0.0-1439.g5dcb190d.el9.x86_64 30/34 2024-03-25T16:33:56.850 INFO:teuthology.orchestra.run.smithi001.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph.target → /usr/lib/systemd/system/ceph.target. 2024-03-25T16:33:56.850 INFO:teuthology.orchestra.run.smithi001.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-crash.service → /usr/lib/systemd/system/ceph-crash.service. 2024-03-25T16:33:56.850 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-25T16:33:56.867 INFO:teuthology.orchestra.run.smithi115.stdout: Running scriptlet: libcephfs2-2:19.0.0-1439.g5dcb190d.el9.x86_64 9/34 2024-03-25T16:33:56.897 INFO:teuthology.orchestra.run.smithi115.stdout: Installing : liboath-2.6.7-2.el9.x86_64 10/34 2024-03-25T16:33:56.902 INFO:teuthology.orchestra.run.smithi001.stdout: Installing : ceph-selinux-2:19.0.0-1439.g5dcb190d.el9.x86_64 31/34 2024-03-25T16:33:56.941 INFO:teuthology.orchestra.run.smithi115.stdout: Installing : python3-ceph-argparse-2:19.0.0-1439.g5dcb190d.el9. 11/34 2024-03-25T16:33:57.009 INFO:teuthology.orchestra.run.smithi115.stdout: Installing : python3-cephfs-2:19.0.0-1439.g5dcb190d.el9.x86_64 12/34 2024-03-25T16:33:57.032 INFO:teuthology.orchestra.run.smithi115.stdout: Installing : libradosstriper1-2:19.0.0-1439.g5dcb190d.el9.x86_6 13/34 2024-03-25T16:33:57.078 INFO:teuthology.orchestra.run.smithi115.stdout: Running scriptlet: libradosstriper1-2:19.0.0-1439.g5dcb190d.el9.x86_6 13/34 2024-03-25T16:33:57.105 INFO:teuthology.orchestra.run.smithi115.stdout: Installing : qatzip-libs-1.1.2-2.el9.x86_64 14/34 2024-03-25T16:33:57.215 INFO:teuthology.orchestra.run.smithi115.stdout: Installing : re2-1:20211101-3.el9.x86_64 15/34 2024-03-25T16:33:57.257 INFO:teuthology.orchestra.run.smithi115.stdout: Installing : libarrow-9.0.0-10.el9.x86_64 16/34 2024-03-25T16:33:57.303 INFO:teuthology.orchestra.run.smithi115.stdout: Installing : libunwind-1.6.2-1.el9.x86_64 17/34 2024-03-25T16:33:57.330 INFO:teuthology.orchestra.run.smithi115.stdout: Installing : gperftools-libs-2.9.1-2.el9.x86_64 18/34 2024-03-25T16:33:57.358 INFO:teuthology.orchestra.run.smithi115.stdout: Installing : libarrow-doc-9.0.0-10.el9.noarch 19/34 2024-03-25T16:33:57.391 INFO:teuthology.orchestra.run.smithi115.stdout: Installing : python3-prettytable-0.7.2-27.el9.noarch 20/34 2024-03-25T16:33:57.717 INFO:teuthology.orchestra.run.smithi115.stdout: Installing : libpmemobj-1.12.1-1.el9.x86_64 21/34 2024-03-25T16:33:57.754 INFO:teuthology.orchestra.run.smithi115.stdout: Upgrading : librbd1-2:19.0.0-1439.g5dcb190d.el9.x86_64 22/34 2024-03-25T16:33:57.826 INFO:teuthology.orchestra.run.smithi115.stdout: Running scriptlet: librbd1-2:19.0.0-1439.g5dcb190d.el9.x86_64 22/34 2024-03-25T16:33:57.859 INFO:teuthology.orchestra.run.smithi115.stdout: Installing : python3-rbd-2:19.0.0-1439.g5dcb190d.el9.x86_64 23/34 2024-03-25T16:33:57.901 INFO:teuthology.orchestra.run.smithi115.stdout: Installing : boost-program-options-1.75.0-8.el9.x86_64 24/34 2024-03-25T16:33:58.356 INFO:teuthology.orchestra.run.smithi115.stdout: Installing : parquet-libs-9.0.0-10.el9.x86_64 25/34 2024-03-25T16:33:58.382 INFO:teuthology.orchestra.run.smithi115.stdout: Installing : librgw2-2:19.0.0-1439.g5dcb190d.el9.x86_64 26/34 2024-03-25T16:33:58.452 INFO:teuthology.orchestra.run.smithi115.stdout: Running scriptlet: librgw2-2:19.0.0-1439.g5dcb190d.el9.x86_64 26/34 2024-03-25T16:33:58.521 INFO:teuthology.orchestra.run.smithi115.stdout: Installing : python3-rgw-2:19.0.0-1439.g5dcb190d.el9.x86_64 27/34 2024-03-25T16:33:58.551 INFO:teuthology.orchestra.run.smithi115.stdout: Installing : python3-ceph-common-2:19.0.0-1439.g5dcb190d.el9.x8 28/34 2024-03-25T16:34:00.725 INFO:teuthology.orchestra.run.smithi115.stdout: Running scriptlet: ceph-common-2:19.0.0-1439.g5dcb190d.el9.x86_64 29/34 2024-03-25T16:34:00.772 INFO:teuthology.orchestra.run.smithi115.stdout: Installing : ceph-common-2:19.0.0-1439.g5dcb190d.el9.x86_64 29/34 2024-03-25T16:34:01.344 INFO:teuthology.orchestra.run.smithi115.stdout: Running scriptlet: ceph-common-2:19.0.0-1439.g5dcb190d.el9.x86_64 29/34 2024-03-25T16:34:01.375 INFO:teuthology.orchestra.run.smithi115.stdout: Installing : ceph-base-2:19.0.0-1439.g5dcb190d.el9.x86_64 30/34 2024-03-25T16:34:01.468 INFO:teuthology.orchestra.run.smithi115.stdout: Running scriptlet: ceph-base-2:19.0.0-1439.g5dcb190d.el9.x86_64 30/34 2024-03-25T16:34:01.468 INFO:teuthology.orchestra.run.smithi115.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph.target → /usr/lib/systemd/system/ceph.target. 2024-03-25T16:34:01.468 INFO:teuthology.orchestra.run.smithi115.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-crash.service → /usr/lib/systemd/system/ceph-crash.service. 2024-03-25T16:34:01.468 INFO:teuthology.orchestra.run.smithi115.stdout: 2024-03-25T16:34:01.510 INFO:teuthology.orchestra.run.smithi115.stdout: Installing : ceph-selinux-2:19.0.0-1439.g5dcb190d.el9.x86_64 31/34 2024-03-25T16:34:06.674 INFO:teuthology.orchestra.run.smithi001.stdout: Running scriptlet: ceph-selinux-2:19.0.0-1439.g5dcb190d.el9.x86_64 31/34 2024-03-25T16:34:06.674 INFO:teuthology.orchestra.run.smithi001.stdout:skipping the directory /sys 2024-03-25T16:34:06.674 INFO:teuthology.orchestra.run.smithi001.stdout:skipping the directory /proc 2024-03-25T16:34:06.674 INFO:teuthology.orchestra.run.smithi001.stdout:skipping the directory /mnt 2024-03-25T16:34:06.674 INFO:teuthology.orchestra.run.smithi001.stdout:skipping the directory /var/tmp 2024-03-25T16:34:06.675 INFO:teuthology.orchestra.run.smithi001.stdout:skipping the directory /home 2024-03-25T16:34:06.675 INFO:teuthology.orchestra.run.smithi001.stdout:skipping the directory /root 2024-03-25T16:34:06.675 INFO:teuthology.orchestra.run.smithi001.stdout:skipping the directory /tmp 2024-03-25T16:34:06.675 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-25T16:34:07.613 INFO:teuthology.orchestra.run.smithi001.stdout: Installing : ceph-radosgw-2:19.0.0-1439.g5dcb190d.el9.x86_64 32/34 2024-03-25T16:34:07.634 INFO:teuthology.orchestra.run.smithi001.stdout: Running scriptlet: ceph-radosgw-2:19.0.0-1439.g5dcb190d.el9.x86_64 32/34 2024-03-25T16:34:07.634 INFO:teuthology.orchestra.run.smithi001.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-25T16:34:07.634 INFO:teuthology.orchestra.run.smithi001.stdout:Invalid unit name "ceph-radosgw@*.service" escaped as "ceph-radosgw@\x2a.service". 2024-03-25T16:34:07.634 INFO:teuthology.orchestra.run.smithi001.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-radosgw.target → /usr/lib/systemd/system/ceph-radosgw.target. 2024-03-25T16:34:07.634 INFO:teuthology.orchestra.run.smithi001.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-radosgw.target → /usr/lib/systemd/system/ceph-radosgw.target. 2024-03-25T16:34:07.634 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-25T16:34:07.636 INFO:teuthology.orchestra.run.smithi001.stdout: Cleanup : librbd1-2:16.2.4-5.el9.x86_64 33/34 2024-03-25T16:34:07.731 INFO:teuthology.orchestra.run.smithi001.stdout: Running scriptlet: librbd1-2:16.2.4-5.el9.x86_64 33/34 2024-03-25T16:34:07.732 INFO:teuthology.orchestra.run.smithi001.stdout: Cleanup : librados2-2:16.2.4-5.el9.x86_64 34/34 2024-03-25T16:34:08.867 INFO:teuthology.orchestra.run.smithi001.stdout: Running scriptlet: librados2-2:16.2.4-5.el9.x86_64 34/34 2024-03-25T16:34:08.867 INFO:teuthology.orchestra.run.smithi001.stdout: Verifying : ceph-base-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/34 2024-03-25T16:34:08.867 INFO:teuthology.orchestra.run.smithi001.stdout: Verifying : ceph-common-2:19.0.0-1439.g5dcb190d.el9.x86_64 2/34 2024-03-25T16:34:08.867 INFO:teuthology.orchestra.run.smithi001.stdout: Verifying : ceph-radosgw-2:19.0.0-1439.g5dcb190d.el9.x86_64 3/34 2024-03-25T16:34:08.867 INFO:teuthology.orchestra.run.smithi001.stdout: Verifying : ceph-selinux-2:19.0.0-1439.g5dcb190d.el9.x86_64 4/34 2024-03-25T16:34:08.867 INFO:teuthology.orchestra.run.smithi001.stdout: Verifying : libcephfs2-2:19.0.0-1439.g5dcb190d.el9.x86_64 5/34 2024-03-25T16:34:08.867 INFO:teuthology.orchestra.run.smithi001.stdout: Verifying : libradosstriper1-2:19.0.0-1439.g5dcb190d.el9.x86_6 6/34 2024-03-25T16:34:08.867 INFO:teuthology.orchestra.run.smithi001.stdout: Verifying : librgw2-2:19.0.0-1439.g5dcb190d.el9.x86_64 7/34 2024-03-25T16:34:08.868 INFO:teuthology.orchestra.run.smithi001.stdout: Verifying : python3-ceph-argparse-2:19.0.0-1439.g5dcb190d.el9. 8/34 2024-03-25T16:34:08.868 INFO:teuthology.orchestra.run.smithi001.stdout: Verifying : python3-ceph-common-2:19.0.0-1439.g5dcb190d.el9.x8 9/34 2024-03-25T16:34:08.868 INFO:teuthology.orchestra.run.smithi001.stdout: Verifying : python3-cephfs-2:19.0.0-1439.g5dcb190d.el9.x86_64 10/34 2024-03-25T16:34:08.868 INFO:teuthology.orchestra.run.smithi001.stdout: Verifying : python3-rados-2:19.0.0-1439.g5dcb190d.el9.x86_64 11/34 2024-03-25T16:34:08.868 INFO:teuthology.orchestra.run.smithi001.stdout: Verifying : python3-rbd-2:19.0.0-1439.g5dcb190d.el9.x86_64 12/34 2024-03-25T16:34:08.868 INFO:teuthology.orchestra.run.smithi001.stdout: Verifying : python3-rgw-2:19.0.0-1439.g5dcb190d.el9.x86_64 13/34 2024-03-25T16:34:08.868 INFO:teuthology.orchestra.run.smithi001.stdout: Verifying : boost-program-options-1.75.0-8.el9.x86_64 14/34 2024-03-25T16:34:08.868 INFO:teuthology.orchestra.run.smithi001.stdout: Verifying : libpmemobj-1.12.1-1.el9.x86_64 15/34 2024-03-25T16:34:08.868 INFO:teuthology.orchestra.run.smithi001.stdout: Verifying : librabbitmq-0.11.0-7.el9.x86_64 16/34 2024-03-25T16:34:08.868 INFO:teuthology.orchestra.run.smithi001.stdout: Verifying : librdkafka-1.6.1-102.el9.x86_64 17/34 2024-03-25T16:34:08.868 INFO:teuthology.orchestra.run.smithi001.stdout: Verifying : lttng-ust-2.12.0-6.el9.x86_64 18/34 2024-03-25T16:34:08.868 INFO:teuthology.orchestra.run.smithi001.stdout: Verifying : python3-prettytable-0.7.2-27.el9.noarch 19/34 2024-03-25T16:34:08.868 INFO:teuthology.orchestra.run.smithi001.stdout: Verifying : qatlib-23.11.0-1.el9.x86_64 20/34 2024-03-25T16:34:08.868 INFO:teuthology.orchestra.run.smithi001.stdout: Verifying : qatlib-service-23.11.0-1.el9.x86_64 21/34 2024-03-25T16:34:08.869 INFO:teuthology.orchestra.run.smithi001.stdout: Verifying : qatzip-libs-1.1.2-2.el9.x86_64 22/34 2024-03-25T16:34:08.869 INFO:teuthology.orchestra.run.smithi001.stdout: Verifying : gperftools-libs-2.9.1-2.el9.x86_64 23/34 2024-03-25T16:34:08.869 INFO:teuthology.orchestra.run.smithi001.stdout: Verifying : libarrow-9.0.0-10.el9.x86_64 24/34 2024-03-25T16:34:08.869 INFO:teuthology.orchestra.run.smithi001.stdout: Verifying : libarrow-doc-9.0.0-10.el9.noarch 25/34 2024-03-25T16:34:08.869 INFO:teuthology.orchestra.run.smithi001.stdout: Verifying : liboath-2.6.7-2.el9.x86_64 26/34 2024-03-25T16:34:08.869 INFO:teuthology.orchestra.run.smithi001.stdout: Verifying : libunwind-1.6.2-1.el9.x86_64 27/34 2024-03-25T16:34:08.869 INFO:teuthology.orchestra.run.smithi001.stdout: Verifying : parquet-libs-9.0.0-10.el9.x86_64 28/34 2024-03-25T16:34:08.869 INFO:teuthology.orchestra.run.smithi001.stdout: Verifying : re2-1:20211101-3.el9.x86_64 29/34 2024-03-25T16:34:08.869 INFO:teuthology.orchestra.run.smithi001.stdout: Verifying : thrift-0.15.0-2.el9.x86_64 30/34 2024-03-25T16:34:08.869 INFO:teuthology.orchestra.run.smithi001.stdout: Verifying : librados2-2:19.0.0-1439.g5dcb190d.el9.x86_64 31/34 2024-03-25T16:34:08.869 INFO:teuthology.orchestra.run.smithi001.stdout: Verifying : librados2-2:16.2.4-5.el9.x86_64 32/34 2024-03-25T16:34:08.869 INFO:teuthology.orchestra.run.smithi001.stdout: Verifying : librbd1-2:19.0.0-1439.g5dcb190d.el9.x86_64 33/34 2024-03-25T16:34:10.159 INFO:teuthology.orchestra.run.smithi001.stdout: Verifying : librbd1-2:16.2.4-5.el9.x86_64 34/34 2024-03-25T16:34:10.159 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-25T16:34:10.160 INFO:teuthology.orchestra.run.smithi001.stdout:Upgraded: 2024-03-25T16:34:10.160 INFO:teuthology.orchestra.run.smithi001.stdout: librados2-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T16:34:10.160 INFO:teuthology.orchestra.run.smithi001.stdout: librbd1-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T16:34:10.160 INFO:teuthology.orchestra.run.smithi001.stdout:Installed: 2024-03-25T16:34:10.160 INFO:teuthology.orchestra.run.smithi001.stdout: boost-program-options-1.75.0-8.el9.x86_64 2024-03-25T16:34:10.160 INFO:teuthology.orchestra.run.smithi001.stdout: ceph-base-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T16:34:10.160 INFO:teuthology.orchestra.run.smithi001.stdout: ceph-common-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T16:34:10.160 INFO:teuthology.orchestra.run.smithi001.stdout: ceph-radosgw-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T16:34:10.160 INFO:teuthology.orchestra.run.smithi001.stdout: ceph-selinux-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T16:34:10.160 INFO:teuthology.orchestra.run.smithi001.stdout: gperftools-libs-2.9.1-2.el9.x86_64 2024-03-25T16:34:10.160 INFO:teuthology.orchestra.run.smithi001.stdout: libarrow-9.0.0-10.el9.x86_64 2024-03-25T16:34:10.160 INFO:teuthology.orchestra.run.smithi001.stdout: libarrow-doc-9.0.0-10.el9.noarch 2024-03-25T16:34:10.161 INFO:teuthology.orchestra.run.smithi001.stdout: libcephfs2-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T16:34:10.161 INFO:teuthology.orchestra.run.smithi001.stdout: liboath-2.6.7-2.el9.x86_64 2024-03-25T16:34:10.161 INFO:teuthology.orchestra.run.smithi001.stdout: libpmemobj-1.12.1-1.el9.x86_64 2024-03-25T16:34:10.161 INFO:teuthology.orchestra.run.smithi001.stdout: librabbitmq-0.11.0-7.el9.x86_64 2024-03-25T16:34:10.161 INFO:teuthology.orchestra.run.smithi001.stdout: libradosstriper1-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T16:34:10.161 INFO:teuthology.orchestra.run.smithi001.stdout: librdkafka-1.6.1-102.el9.x86_64 2024-03-25T16:34:10.161 INFO:teuthology.orchestra.run.smithi001.stdout: librgw2-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T16:34:10.161 INFO:teuthology.orchestra.run.smithi001.stdout: libunwind-1.6.2-1.el9.x86_64 2024-03-25T16:34:10.161 INFO:teuthology.orchestra.run.smithi001.stdout: lttng-ust-2.12.0-6.el9.x86_64 2024-03-25T16:34:10.161 INFO:teuthology.orchestra.run.smithi001.stdout: parquet-libs-9.0.0-10.el9.x86_64 2024-03-25T16:34:10.161 INFO:teuthology.orchestra.run.smithi001.stdout: python3-ceph-argparse-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T16:34:10.161 INFO:teuthology.orchestra.run.smithi001.stdout: python3-ceph-common-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T16:34:10.161 INFO:teuthology.orchestra.run.smithi001.stdout: python3-cephfs-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T16:34:10.161 INFO:teuthology.orchestra.run.smithi001.stdout: python3-prettytable-0.7.2-27.el9.noarch 2024-03-25T16:34:10.162 INFO:teuthology.orchestra.run.smithi001.stdout: python3-rados-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T16:34:10.162 INFO:teuthology.orchestra.run.smithi001.stdout: python3-rbd-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T16:34:10.162 INFO:teuthology.orchestra.run.smithi001.stdout: python3-rgw-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T16:34:10.162 INFO:teuthology.orchestra.run.smithi001.stdout: qatlib-23.11.0-1.el9.x86_64 2024-03-25T16:34:10.162 INFO:teuthology.orchestra.run.smithi001.stdout: qatlib-service-23.11.0-1.el9.x86_64 2024-03-25T16:34:10.162 INFO:teuthology.orchestra.run.smithi001.stdout: qatzip-libs-1.1.2-2.el9.x86_64 2024-03-25T16:34:10.162 INFO:teuthology.orchestra.run.smithi001.stdout: re2-1:20211101-3.el9.x86_64 2024-03-25T16:34:10.162 INFO:teuthology.orchestra.run.smithi001.stdout: thrift-0.15.0-2.el9.x86_64 2024-03-25T16:34:10.162 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-25T16:34:10.162 INFO:teuthology.orchestra.run.smithi001.stdout:Complete! 2024-03-25T16:34:10.479 DEBUG:teuthology.orchestra.run.smithi001:> sudo yum -y install ceph-test 2024-03-25T16:34:10.777 INFO:teuthology.orchestra.run.smithi115.stdout: Running scriptlet: ceph-selinux-2:19.0.0-1439.g5dcb190d.el9.x86_64 31/34 2024-03-25T16:34:10.777 INFO:teuthology.orchestra.run.smithi115.stdout:skipping the directory /sys 2024-03-25T16:34:10.777 INFO:teuthology.orchestra.run.smithi115.stdout:skipping the directory /proc 2024-03-25T16:34:10.777 INFO:teuthology.orchestra.run.smithi115.stdout:skipping the directory /mnt 2024-03-25T16:34:10.777 INFO:teuthology.orchestra.run.smithi115.stdout:skipping the directory /var/tmp 2024-03-25T16:34:10.778 INFO:teuthology.orchestra.run.smithi115.stdout:skipping the directory /home 2024-03-25T16:34:10.778 INFO:teuthology.orchestra.run.smithi115.stdout:skipping the directory /root 2024-03-25T16:34:10.778 INFO:teuthology.orchestra.run.smithi115.stdout:skipping the directory /tmp 2024-03-25T16:34:10.778 INFO:teuthology.orchestra.run.smithi115.stdout: 2024-03-25T16:34:11.051 INFO:teuthology.orchestra.run.smithi001.stdout:Last metadata expiration check: 0:01:22 ago on Mon 25 Mar 2024 04:32:49 PM UTC. 2024-03-25T16:34:11.205 INFO:teuthology.orchestra.run.smithi001.stdout:Dependencies resolved. 2024-03-25T16:34:11.205 INFO:teuthology.orchestra.run.smithi001.stdout:================================================================================ 2024-03-25T16:34:11.205 INFO:teuthology.orchestra.run.smithi001.stdout: Package Arch Version Repository Size 2024-03-25T16:34:11.206 INFO:teuthology.orchestra.run.smithi001.stdout:================================================================================ 2024-03-25T16:34:11.206 INFO:teuthology.orchestra.run.smithi001.stdout:Installing: 2024-03-25T16:34:11.206 INFO:teuthology.orchestra.run.smithi001.stdout: ceph-test x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 47 M 2024-03-25T16:34:11.206 INFO:teuthology.orchestra.run.smithi001.stdout:Installing dependencies: 2024-03-25T16:34:11.206 INFO:teuthology.orchestra.run.smithi001.stdout: jq x86_64 1.6-16.el9 baseos 187 k 2024-03-25T16:34:11.206 INFO:teuthology.orchestra.run.smithi001.stdout: libcephsqlite x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 163 k 2024-03-25T16:34:11.206 INFO:teuthology.orchestra.run.smithi001.stdout: oniguruma x86_64 6.9.6-1.el9.5 CentOS-AppStream 217 k 2024-03-25T16:34:11.206 INFO:teuthology.orchestra.run.smithi001.stdout: socat x86_64 1.7.4.1-5.el9 CentOS-AppStream 300 k 2024-03-25T16:34:11.206 INFO:teuthology.orchestra.run.smithi001.stdout: xmlstarlet x86_64 1.6.1-20.el9 CentOS-AppStream 63 k 2024-03-25T16:34:11.206 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-25T16:34:11.206 INFO:teuthology.orchestra.run.smithi001.stdout:Transaction Summary 2024-03-25T16:34:11.206 INFO:teuthology.orchestra.run.smithi001.stdout:================================================================================ 2024-03-25T16:34:11.206 INFO:teuthology.orchestra.run.smithi001.stdout:Install 6 Packages 2024-03-25T16:34:11.207 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-25T16:34:11.207 INFO:teuthology.orchestra.run.smithi001.stdout:Total download size: 48 M 2024-03-25T16:34:11.207 INFO:teuthology.orchestra.run.smithi001.stdout:Installed size: 204 M 2024-03-25T16:34:11.207 INFO:teuthology.orchestra.run.smithi001.stdout:Downloading Packages: 2024-03-25T16:34:11.645 INFO:teuthology.orchestra.run.smithi001.stdout:(1/6): oniguruma-6.9.6-1.el9.5.x86_64.rpm 1.3 MB/s | 217 kB 00:00 2024-03-25T16:34:11.675 INFO:teuthology.orchestra.run.smithi115.stdout: Installing : ceph-radosgw-2:19.0.0-1439.g5dcb190d.el9.x86_64 32/34 2024-03-25T16:34:11.699 INFO:teuthology.orchestra.run.smithi115.stdout: Running scriptlet: ceph-radosgw-2:19.0.0-1439.g5dcb190d.el9.x86_64 32/34 2024-03-25T16:34:11.699 INFO:teuthology.orchestra.run.smithi115.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-25T16:34:11.700 INFO:teuthology.orchestra.run.smithi115.stdout:Invalid unit name "ceph-radosgw@*.service" escaped as "ceph-radosgw@\x2a.service". 2024-03-25T16:34:11.700 INFO:teuthology.orchestra.run.smithi115.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-radosgw.target → /usr/lib/systemd/system/ceph-radosgw.target. 2024-03-25T16:34:11.700 INFO:teuthology.orchestra.run.smithi115.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-radosgw.target → /usr/lib/systemd/system/ceph-radosgw.target. 2024-03-25T16:34:11.700 INFO:teuthology.orchestra.run.smithi115.stdout: 2024-03-25T16:34:11.701 INFO:teuthology.orchestra.run.smithi115.stdout: Cleanup : librbd1-2:16.2.4-5.el9.x86_64 33/34 2024-03-25T16:34:11.704 INFO:teuthology.orchestra.run.smithi001.stdout:(2/6): socat-1.7.4.1-5.el9.x86_64.rpm 5.0 MB/s | 300 kB 00:00 2024-03-25T16:34:11.737 INFO:teuthology.orchestra.run.smithi001.stdout:(3/6): xmlstarlet-1.6.1-20.el9.x86_64.rpm 1.9 MB/s | 63 kB 00:00 2024-03-25T16:34:11.769 INFO:teuthology.orchestra.run.smithi115.stdout: Running scriptlet: librbd1-2:16.2.4-5.el9.x86_64 33/34 2024-03-25T16:34:11.770 INFO:teuthology.orchestra.run.smithi115.stdout: Cleanup : librados2-2:16.2.4-5.el9.x86_64 34/34 2024-03-25T16:34:11.946 INFO:teuthology.orchestra.run.smithi001.stdout:(4/6): jq-1.6-16.el9.x86_64.rpm 898 kB/s | 187 kB 00:00 2024-03-25T16:34:12.480 INFO:teuthology.orchestra.run.smithi001.stdout:(5/6): libcephsqlite-19.0.0-1439.g5dcb190d.el9. 164 kB/s | 163 kB 00:00 2024-03-25T16:34:12.925 INFO:teuthology.orchestra.run.smithi115.stdout: Running scriptlet: librados2-2:16.2.4-5.el9.x86_64 34/34 2024-03-25T16:34:12.925 INFO:teuthology.orchestra.run.smithi115.stdout: Verifying : ceph-base-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/34 2024-03-25T16:34:12.925 INFO:teuthology.orchestra.run.smithi115.stdout: Verifying : ceph-common-2:19.0.0-1439.g5dcb190d.el9.x86_64 2/34 2024-03-25T16:34:12.925 INFO:teuthology.orchestra.run.smithi115.stdout: Verifying : ceph-radosgw-2:19.0.0-1439.g5dcb190d.el9.x86_64 3/34 2024-03-25T16:34:12.925 INFO:teuthology.orchestra.run.smithi115.stdout: Verifying : ceph-selinux-2:19.0.0-1439.g5dcb190d.el9.x86_64 4/34 2024-03-25T16:34:12.925 INFO:teuthology.orchestra.run.smithi115.stdout: Verifying : libcephfs2-2:19.0.0-1439.g5dcb190d.el9.x86_64 5/34 2024-03-25T16:34:12.925 INFO:teuthology.orchestra.run.smithi115.stdout: Verifying : libradosstriper1-2:19.0.0-1439.g5dcb190d.el9.x86_6 6/34 2024-03-25T16:34:12.925 INFO:teuthology.orchestra.run.smithi115.stdout: Verifying : librgw2-2:19.0.0-1439.g5dcb190d.el9.x86_64 7/34 2024-03-25T16:34:12.925 INFO:teuthology.orchestra.run.smithi115.stdout: Verifying : python3-ceph-argparse-2:19.0.0-1439.g5dcb190d.el9. 8/34 2024-03-25T16:34:12.925 INFO:teuthology.orchestra.run.smithi115.stdout: Verifying : python3-ceph-common-2:19.0.0-1439.g5dcb190d.el9.x8 9/34 2024-03-25T16:34:12.925 INFO:teuthology.orchestra.run.smithi115.stdout: Verifying : python3-cephfs-2:19.0.0-1439.g5dcb190d.el9.x86_64 10/34 2024-03-25T16:34:12.925 INFO:teuthology.orchestra.run.smithi115.stdout: Verifying : python3-rados-2:19.0.0-1439.g5dcb190d.el9.x86_64 11/34 2024-03-25T16:34:12.925 INFO:teuthology.orchestra.run.smithi115.stdout: Verifying : python3-rbd-2:19.0.0-1439.g5dcb190d.el9.x86_64 12/34 2024-03-25T16:34:12.925 INFO:teuthology.orchestra.run.smithi115.stdout: Verifying : python3-rgw-2:19.0.0-1439.g5dcb190d.el9.x86_64 13/34 2024-03-25T16:34:12.926 INFO:teuthology.orchestra.run.smithi115.stdout: Verifying : boost-program-options-1.75.0-8.el9.x86_64 14/34 2024-03-25T16:34:12.926 INFO:teuthology.orchestra.run.smithi115.stdout: Verifying : libpmemobj-1.12.1-1.el9.x86_64 15/34 2024-03-25T16:34:12.926 INFO:teuthology.orchestra.run.smithi115.stdout: Verifying : librabbitmq-0.11.0-7.el9.x86_64 16/34 2024-03-25T16:34:12.926 INFO:teuthology.orchestra.run.smithi115.stdout: Verifying : librdkafka-1.6.1-102.el9.x86_64 17/34 2024-03-25T16:34:12.926 INFO:teuthology.orchestra.run.smithi115.stdout: Verifying : lttng-ust-2.12.0-6.el9.x86_64 18/34 2024-03-25T16:34:12.926 INFO:teuthology.orchestra.run.smithi115.stdout: Verifying : python3-prettytable-0.7.2-27.el9.noarch 19/34 2024-03-25T16:34:12.926 INFO:teuthology.orchestra.run.smithi115.stdout: Verifying : qatlib-23.11.0-1.el9.x86_64 20/34 2024-03-25T16:34:12.926 INFO:teuthology.orchestra.run.smithi115.stdout: Verifying : qatlib-service-23.11.0-1.el9.x86_64 21/34 2024-03-25T16:34:12.926 INFO:teuthology.orchestra.run.smithi115.stdout: Verifying : qatzip-libs-1.1.2-2.el9.x86_64 22/34 2024-03-25T16:34:12.926 INFO:teuthology.orchestra.run.smithi115.stdout: Verifying : gperftools-libs-2.9.1-2.el9.x86_64 23/34 2024-03-25T16:34:12.926 INFO:teuthology.orchestra.run.smithi115.stdout: Verifying : libarrow-9.0.0-10.el9.x86_64 24/34 2024-03-25T16:34:12.926 INFO:teuthology.orchestra.run.smithi115.stdout: Verifying : libarrow-doc-9.0.0-10.el9.noarch 25/34 2024-03-25T16:34:12.926 INFO:teuthology.orchestra.run.smithi115.stdout: Verifying : liboath-2.6.7-2.el9.x86_64 26/34 2024-03-25T16:34:12.927 INFO:teuthology.orchestra.run.smithi115.stdout: Verifying : libunwind-1.6.2-1.el9.x86_64 27/34 2024-03-25T16:34:12.927 INFO:teuthology.orchestra.run.smithi115.stdout: Verifying : parquet-libs-9.0.0-10.el9.x86_64 28/34 2024-03-25T16:34:12.927 INFO:teuthology.orchestra.run.smithi115.stdout: Verifying : re2-1:20211101-3.el9.x86_64 29/34 2024-03-25T16:34:12.927 INFO:teuthology.orchestra.run.smithi115.stdout: Verifying : thrift-0.15.0-2.el9.x86_64 30/34 2024-03-25T16:34:12.927 INFO:teuthology.orchestra.run.smithi115.stdout: Verifying : librados2-2:19.0.0-1439.g5dcb190d.el9.x86_64 31/34 2024-03-25T16:34:12.928 INFO:teuthology.orchestra.run.smithi115.stdout: Verifying : librados2-2:16.2.4-5.el9.x86_64 32/34 2024-03-25T16:34:12.928 INFO:teuthology.orchestra.run.smithi115.stdout: Verifying : librbd1-2:19.0.0-1439.g5dcb190d.el9.x86_64 33/34 2024-03-25T16:34:13.974 INFO:teuthology.orchestra.run.smithi115.stdout: Verifying : librbd1-2:16.2.4-5.el9.x86_64 34/34 2024-03-25T16:34:13.974 INFO:teuthology.orchestra.run.smithi115.stdout: 2024-03-25T16:34:13.974 INFO:teuthology.orchestra.run.smithi115.stdout:Upgraded: 2024-03-25T16:34:13.974 INFO:teuthology.orchestra.run.smithi115.stdout: librados2-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T16:34:13.975 INFO:teuthology.orchestra.run.smithi115.stdout: librbd1-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T16:34:13.975 INFO:teuthology.orchestra.run.smithi115.stdout:Installed: 2024-03-25T16:34:13.975 INFO:teuthology.orchestra.run.smithi115.stdout: boost-program-options-1.75.0-8.el9.x86_64 2024-03-25T16:34:13.975 INFO:teuthology.orchestra.run.smithi115.stdout: ceph-base-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T16:34:13.975 INFO:teuthology.orchestra.run.smithi115.stdout: ceph-common-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T16:34:13.975 INFO:teuthology.orchestra.run.smithi115.stdout: ceph-radosgw-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T16:34:13.975 INFO:teuthology.orchestra.run.smithi115.stdout: ceph-selinux-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T16:34:13.975 INFO:teuthology.orchestra.run.smithi115.stdout: gperftools-libs-2.9.1-2.el9.x86_64 2024-03-25T16:34:13.975 INFO:teuthology.orchestra.run.smithi115.stdout: libarrow-9.0.0-10.el9.x86_64 2024-03-25T16:34:13.975 INFO:teuthology.orchestra.run.smithi115.stdout: libarrow-doc-9.0.0-10.el9.noarch 2024-03-25T16:34:13.975 INFO:teuthology.orchestra.run.smithi115.stdout: libcephfs2-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T16:34:13.975 INFO:teuthology.orchestra.run.smithi115.stdout: liboath-2.6.7-2.el9.x86_64 2024-03-25T16:34:13.975 INFO:teuthology.orchestra.run.smithi115.stdout: libpmemobj-1.12.1-1.el9.x86_64 2024-03-25T16:34:13.975 INFO:teuthology.orchestra.run.smithi115.stdout: librabbitmq-0.11.0-7.el9.x86_64 2024-03-25T16:34:13.975 INFO:teuthology.orchestra.run.smithi115.stdout: libradosstriper1-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T16:34:13.976 INFO:teuthology.orchestra.run.smithi115.stdout: librdkafka-1.6.1-102.el9.x86_64 2024-03-25T16:34:13.976 INFO:teuthology.orchestra.run.smithi115.stdout: librgw2-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T16:34:13.976 INFO:teuthology.orchestra.run.smithi115.stdout: libunwind-1.6.2-1.el9.x86_64 2024-03-25T16:34:13.976 INFO:teuthology.orchestra.run.smithi115.stdout: lttng-ust-2.12.0-6.el9.x86_64 2024-03-25T16:34:13.976 INFO:teuthology.orchestra.run.smithi115.stdout: parquet-libs-9.0.0-10.el9.x86_64 2024-03-25T16:34:13.976 INFO:teuthology.orchestra.run.smithi115.stdout: python3-ceph-argparse-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T16:34:13.976 INFO:teuthology.orchestra.run.smithi115.stdout: python3-ceph-common-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T16:34:13.976 INFO:teuthology.orchestra.run.smithi115.stdout: python3-cephfs-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T16:34:13.976 INFO:teuthology.orchestra.run.smithi115.stdout: python3-prettytable-0.7.2-27.el9.noarch 2024-03-25T16:34:13.976 INFO:teuthology.orchestra.run.smithi115.stdout: python3-rados-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T16:34:13.976 INFO:teuthology.orchestra.run.smithi115.stdout: python3-rbd-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T16:34:13.976 INFO:teuthology.orchestra.run.smithi115.stdout: python3-rgw-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T16:34:13.976 INFO:teuthology.orchestra.run.smithi115.stdout: qatlib-23.11.0-1.el9.x86_64 2024-03-25T16:34:13.976 INFO:teuthology.orchestra.run.smithi115.stdout: qatlib-service-23.11.0-1.el9.x86_64 2024-03-25T16:34:13.976 INFO:teuthology.orchestra.run.smithi115.stdout: qatzip-libs-1.1.2-2.el9.x86_64 2024-03-25T16:34:13.977 INFO:teuthology.orchestra.run.smithi115.stdout: re2-1:20211101-3.el9.x86_64 2024-03-25T16:34:13.977 INFO:teuthology.orchestra.run.smithi115.stdout: thrift-0.15.0-2.el9.x86_64 2024-03-25T16:34:13.977 INFO:teuthology.orchestra.run.smithi115.stdout: 2024-03-25T16:34:13.977 INFO:teuthology.orchestra.run.smithi115.stdout:Complete! 2024-03-25T16:34:14.321 DEBUG:teuthology.orchestra.run.smithi115:> sudo yum -y install ceph-test 2024-03-25T16:34:14.885 INFO:teuthology.orchestra.run.smithi115.stdout:Last metadata expiration check: 0:01:29 ago on Mon 25 Mar 2024 04:32:45 PM UTC. 2024-03-25T16:34:15.045 INFO:teuthology.orchestra.run.smithi115.stdout:Dependencies resolved. 2024-03-25T16:34:15.046 INFO:teuthology.orchestra.run.smithi115.stdout:================================================================================ 2024-03-25T16:34:15.046 INFO:teuthology.orchestra.run.smithi115.stdout: Package Arch Version Repository Size 2024-03-25T16:34:15.046 INFO:teuthology.orchestra.run.smithi115.stdout:================================================================================ 2024-03-25T16:34:15.047 INFO:teuthology.orchestra.run.smithi115.stdout:Installing: 2024-03-25T16:34:15.047 INFO:teuthology.orchestra.run.smithi115.stdout: ceph-test x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 47 M 2024-03-25T16:34:15.047 INFO:teuthology.orchestra.run.smithi115.stdout:Installing dependencies: 2024-03-25T16:34:15.047 INFO:teuthology.orchestra.run.smithi115.stdout: jq x86_64 1.6-16.el9 baseos 187 k 2024-03-25T16:34:15.047 INFO:teuthology.orchestra.run.smithi115.stdout: libcephsqlite x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 163 k 2024-03-25T16:34:15.047 INFO:teuthology.orchestra.run.smithi115.stdout: oniguruma x86_64 6.9.6-1.el9.5 CentOS-AppStream 217 k 2024-03-25T16:34:15.047 INFO:teuthology.orchestra.run.smithi115.stdout: socat x86_64 1.7.4.1-5.el9 CentOS-AppStream 300 k 2024-03-25T16:34:15.047 INFO:teuthology.orchestra.run.smithi115.stdout: xmlstarlet x86_64 1.6.1-20.el9 CentOS-AppStream 63 k 2024-03-25T16:34:15.047 INFO:teuthology.orchestra.run.smithi115.stdout: 2024-03-25T16:34:15.047 INFO:teuthology.orchestra.run.smithi115.stdout:Transaction Summary 2024-03-25T16:34:15.047 INFO:teuthology.orchestra.run.smithi115.stdout:================================================================================ 2024-03-25T16:34:15.047 INFO:teuthology.orchestra.run.smithi115.stdout:Install 6 Packages 2024-03-25T16:34:15.047 INFO:teuthology.orchestra.run.smithi115.stdout: 2024-03-25T16:34:15.048 INFO:teuthology.orchestra.run.smithi115.stdout:Total download size: 48 M 2024-03-25T16:34:15.048 INFO:teuthology.orchestra.run.smithi115.stdout:Installed size: 204 M 2024-03-25T16:34:15.048 INFO:teuthology.orchestra.run.smithi115.stdout:Downloading Packages: 2024-03-25T16:34:15.481 INFO:teuthology.orchestra.run.smithi115.stdout:(1/6): oniguruma-6.9.6-1.el9.5.x86_64.rpm 1.3 MB/s | 217 kB 00:00 2024-03-25T16:34:15.539 INFO:teuthology.orchestra.run.smithi115.stdout:(2/6): socat-1.7.4.1-5.el9.x86_64.rpm 5.0 MB/s | 300 kB 00:00 2024-03-25T16:34:15.565 INFO:teuthology.orchestra.run.smithi115.stdout:(3/6): xmlstarlet-1.6.1-20.el9.x86_64.rpm 2.5 MB/s | 63 kB 00:00 2024-03-25T16:34:15.706 INFO:teuthology.orchestra.run.smithi115.stdout:(4/6): jq-1.6-16.el9.x86_64.rpm 1.3 MB/s | 187 kB 00:00 2024-03-25T16:34:16.648 INFO:teuthology.orchestra.run.smithi115.stdout:(5/6): libcephsqlite-19.0.0-1439.g5dcb190d.el9. 122 kB/s | 163 kB 00:01 2024-03-25T16:35:46.505 INFO:teuthology.orchestra.run.smithi001.stdout:(6/6): ceph-test-19.0.0-1439.g5dcb190d.el9.x86_ 509 kB/s | 47 MB 01:35 2024-03-25T16:35:46.507 INFO:teuthology.orchestra.run.smithi001.stdout:-------------------------------------------------------------------------------- 2024-03-25T16:35:46.507 INFO:teuthology.orchestra.run.smithi001.stdout:Total 518 kB/s | 48 MB 01:35 2024-03-25T16:35:46.519 INFO:teuthology.orchestra.run.smithi001.stdout:Running transaction check 2024-03-25T16:35:46.541 INFO:teuthology.orchestra.run.smithi001.stdout:Transaction check succeeded. 2024-03-25T16:35:46.541 INFO:teuthology.orchestra.run.smithi001.stdout:Running transaction test 2024-03-25T16:35:46.933 INFO:teuthology.orchestra.run.smithi001.stdout:Transaction test succeeded. 2024-03-25T16:35:46.933 INFO:teuthology.orchestra.run.smithi001.stdout:Running transaction 2024-03-25T16:35:47.550 INFO:teuthology.orchestra.run.smithi001.stdout: Preparing : 1/1 2024-03-25T16:35:47.631 INFO:teuthology.orchestra.run.smithi001.stdout: Installing : xmlstarlet-1.6.1-20.el9.x86_64 1/6 2024-03-25T16:35:47.666 INFO:teuthology.orchestra.run.smithi001.stdout: Installing : socat-1.7.4.1-5.el9.x86_64 2/6 2024-03-25T16:35:47.698 INFO:teuthology.orchestra.run.smithi001.stdout: Installing : oniguruma-6.9.6-1.el9.5.x86_64 3/6 2024-03-25T16:35:47.742 INFO:teuthology.orchestra.run.smithi001.stdout: Installing : jq-1.6-16.el9.x86_64 4/6 2024-03-25T16:35:47.766 INFO:teuthology.orchestra.run.smithi001.stdout: Installing : libcephsqlite-2:19.0.0-1439.g5dcb190d.el9.x86_64 5/6 2024-03-25T16:35:52.491 INFO:teuthology.orchestra.run.smithi001.stdout: Running scriptlet: libcephsqlite-2:19.0.0-1439.g5dcb190d.el9.x86_64 5/6 2024-03-25T16:35:52.586 INFO:teuthology.orchestra.run.smithi001.stdout: Installing : ceph-test-2:19.0.0-1439.g5dcb190d.el9.x86_64 6/6 2024-03-25T16:35:54.854 INFO:teuthology.orchestra.run.smithi001.stdout: Running scriptlet: ceph-test-2:19.0.0-1439.g5dcb190d.el9.x86_64 6/6 2024-03-25T16:35:54.854 INFO:teuthology.orchestra.run.smithi001.stdout: Verifying : ceph-test-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/6 2024-03-25T16:35:54.854 INFO:teuthology.orchestra.run.smithi001.stdout: Verifying : libcephsqlite-2:19.0.0-1439.g5dcb190d.el9.x86_64 2/6 2024-03-25T16:35:54.854 INFO:teuthology.orchestra.run.smithi001.stdout: Verifying : oniguruma-6.9.6-1.el9.5.x86_64 3/6 2024-03-25T16:35:54.854 INFO:teuthology.orchestra.run.smithi001.stdout: Verifying : socat-1.7.4.1-5.el9.x86_64 4/6 2024-03-25T16:35:54.854 INFO:teuthology.orchestra.run.smithi001.stdout: Verifying : xmlstarlet-1.6.1-20.el9.x86_64 5/6 2024-03-25T16:35:55.316 INFO:teuthology.orchestra.run.smithi001.stdout: Verifying : jq-1.6-16.el9.x86_64 6/6 2024-03-25T16:35:55.316 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-25T16:35:55.316 INFO:teuthology.orchestra.run.smithi001.stdout:Installed: 2024-03-25T16:35:55.316 INFO:teuthology.orchestra.run.smithi001.stdout: ceph-test-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T16:35:55.316 INFO:teuthology.orchestra.run.smithi001.stdout: jq-1.6-16.el9.x86_64 2024-03-25T16:35:55.316 INFO:teuthology.orchestra.run.smithi001.stdout: libcephsqlite-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T16:35:55.316 INFO:teuthology.orchestra.run.smithi001.stdout: oniguruma-6.9.6-1.el9.5.x86_64 2024-03-25T16:35:55.316 INFO:teuthology.orchestra.run.smithi001.stdout: socat-1.7.4.1-5.el9.x86_64 2024-03-25T16:35:55.316 INFO:teuthology.orchestra.run.smithi001.stdout: xmlstarlet-1.6.1-20.el9.x86_64 2024-03-25T16:35:55.316 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-25T16:35:55.316 INFO:teuthology.orchestra.run.smithi001.stdout:Complete! 2024-03-25T16:35:55.645 DEBUG:teuthology.orchestra.run.smithi001:> sudo yum -y install ceph 2024-03-25T16:35:56.218 INFO:teuthology.orchestra.run.smithi001.stdout:Last metadata expiration check: 0:03:07 ago on Mon 25 Mar 2024 04:32:49 PM UTC. 2024-03-25T16:35:56.382 INFO:teuthology.orchestra.run.smithi001.stdout:Dependencies resolved. 2024-03-25T16:35:56.385 INFO:teuthology.orchestra.run.smithi001.stdout:========================================================================================= 2024-03-25T16:35:56.385 INFO:teuthology.orchestra.run.smithi001.stdout: Package Arch Version Repository Size 2024-03-25T16:35:56.385 INFO:teuthology.orchestra.run.smithi001.stdout:========================================================================================= 2024-03-25T16:35:56.385 INFO:teuthology.orchestra.run.smithi001.stdout:Installing: 2024-03-25T16:35:56.385 INFO:teuthology.orchestra.run.smithi001.stdout: ceph x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 6.6 k 2024-03-25T16:35:56.385 INFO:teuthology.orchestra.run.smithi001.stdout:Installing dependencies: 2024-03-25T16:35:56.385 INFO:teuthology.orchestra.run.smithi001.stdout: ceph-mds x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 2.1 M 2024-03-25T16:35:56.385 INFO:teuthology.orchestra.run.smithi001.stdout: ceph-mgr x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 1.5 M 2024-03-25T16:35:56.385 INFO:teuthology.orchestra.run.smithi001.stdout: ceph-mgr-modules-core noarch 2:19.0.0-1439.g5dcb190d.el9 ceph-noarch 247 k 2024-03-25T16:35:56.385 INFO:teuthology.orchestra.run.smithi001.stdout: ceph-mon x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 4.7 M 2024-03-25T16:35:56.386 INFO:teuthology.orchestra.run.smithi001.stdout: ceph-osd x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 17 M 2024-03-25T16:35:56.386 INFO:teuthology.orchestra.run.smithi001.stdout: lua x86_64 5.4.4-4.el9 CentOS-AppStream 187 k 2024-03-25T16:35:56.386 INFO:teuthology.orchestra.run.smithi001.stdout: lua-devel x86_64 5.4.4-4.el9 crb 22 k 2024-03-25T16:35:56.386 INFO:teuthology.orchestra.run.smithi001.stdout: luarocks noarch 3.9.2-1.el9 epel 151 k 2024-03-25T16:35:56.386 INFO:teuthology.orchestra.run.smithi001.stdout: python3-bcrypt x86_64 3.2.2-1.el9 epel 43 k 2024-03-25T16:35:56.386 INFO:teuthology.orchestra.run.smithi001.stdout: python3-cffi x86_64 1.14.5-5.el9 CentOS-BaseOS 241 k 2024-03-25T16:35:56.386 INFO:teuthology.orchestra.run.smithi001.stdout: python3-cheroot noarch 8.6.0-4.el9 epel 172 k 2024-03-25T16:35:56.386 INFO:teuthology.orchestra.run.smithi001.stdout: python3-cherrypy noarch 18.6.1-2.el9 epel 358 k 2024-03-25T16:35:56.386 INFO:teuthology.orchestra.run.smithi001.stdout: python3-cryptography x86_64 36.0.1-4.el9 CentOS-BaseOS 1.1 M 2024-03-25T16:35:56.386 INFO:teuthology.orchestra.run.smithi001.stdout: python3-jaraco noarch 8.2.1-3.el9 epel 11 k 2024-03-25T16:35:56.386 INFO:teuthology.orchestra.run.smithi001.stdout: python3-jaraco-classes noarch 3.2.1-5.el9 epel 18 k 2024-03-25T16:35:56.386 INFO:teuthology.orchestra.run.smithi001.stdout: python3-jaraco-collections noarch 3.0.0-8.el9 epel 23 k 2024-03-25T16:35:56.387 INFO:teuthology.orchestra.run.smithi001.stdout: python3-jaraco-functools noarch 3.5.0-2.el9 epel 19 k 2024-03-25T16:35:56.387 INFO:teuthology.orchestra.run.smithi001.stdout: python3-jaraco-text noarch 3.2.0-6.el9 epel 20 k 2024-03-25T16:35:56.387 INFO:teuthology.orchestra.run.smithi001.stdout: python3-logutils noarch 0.3.5-21.el9 epel 46 k 2024-03-25T16:35:56.387 INFO:teuthology.orchestra.run.smithi001.stdout: python3-mako noarch 1.1.4-6.el9 CentOS-AppStream 144 k 2024-03-25T16:35:56.387 INFO:teuthology.orchestra.run.smithi001.stdout: python3-markupsafe x86_64 1.1.1-12.el9 CentOS-AppStream 32 k 2024-03-25T16:35:56.387 INFO:teuthology.orchestra.run.smithi001.stdout: python3-more-itertools noarch 8.12.0-2.el9 epel 79 k 2024-03-25T16:35:56.387 INFO:teuthology.orchestra.run.smithi001.stdout: python3-pecan noarch 1.4.2-3.el9 epel 272 k 2024-03-25T16:35:56.387 INFO:teuthology.orchestra.run.smithi001.stdout: python3-ply noarch 3.11-14.el9 CentOS-BaseOS 103 k 2024-03-25T16:35:56.387 INFO:teuthology.orchestra.run.smithi001.stdout: python3-portend noarch 3.1.0-2.el9 epel 16 k 2024-03-25T16:35:56.387 INFO:teuthology.orchestra.run.smithi001.stdout: python3-pyOpenSSL noarch 21.0.0-1.el9 epel 90 k 2024-03-25T16:35:56.387 INFO:teuthology.orchestra.run.smithi001.stdout: python3-pycparser noarch 2.20-6.el9 CentOS-BaseOS 124 k 2024-03-25T16:35:56.387 INFO:teuthology.orchestra.run.smithi001.stdout: python3-pytz noarch 2021.1-5.el9 CentOS-AppStream 47 k 2024-03-25T16:35:56.387 INFO:teuthology.orchestra.run.smithi001.stdout: python3-requests noarch 2.25.1-8.el9 baseos 125 k 2024-03-25T16:35:56.387 INFO:teuthology.orchestra.run.smithi001.stdout: python3-tempora noarch 5.0.0-2.el9 epel 36 k 2024-03-25T16:35:56.387 INFO:teuthology.orchestra.run.smithi001.stdout: python3-toml noarch 0.10.2-6.el9 CentOS-AppStream 39 k 2024-03-25T16:35:56.388 INFO:teuthology.orchestra.run.smithi001.stdout: python3-urllib3 noarch 1.26.5-5.el9 CentOS-BaseOS 187 k 2024-03-25T16:35:56.388 INFO:teuthology.orchestra.run.smithi001.stdout: python3-webob noarch 1.8.7-6.el9 epel 230 k 2024-03-25T16:35:56.388 INFO:teuthology.orchestra.run.smithi001.stdout: python3-werkzeug noarch 2.0.3-3.el9.1 epel 427 k 2024-03-25T16:35:56.388 INFO:teuthology.orchestra.run.smithi001.stdout: python3-zc-lockfile noarch 2.0-10.el9 epel 20 k 2024-03-25T16:35:56.388 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-25T16:35:56.388 INFO:teuthology.orchestra.run.smithi001.stdout:Transaction Summary 2024-03-25T16:35:56.388 INFO:teuthology.orchestra.run.smithi001.stdout:========================================================================================= 2024-03-25T16:35:56.388 INFO:teuthology.orchestra.run.smithi001.stdout:Install 36 Packages 2024-03-25T16:35:56.388 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-25T16:35:56.389 INFO:teuthology.orchestra.run.smithi001.stdout:Total download size: 30 M 2024-03-25T16:35:56.389 INFO:teuthology.orchestra.run.smithi001.stdout:Installed size: 107 M 2024-03-25T16:35:56.389 INFO:teuthology.orchestra.run.smithi001.stdout:Downloading Packages: 2024-03-25T16:35:57.234 INFO:teuthology.orchestra.run.smithi001.stdout:(1/36): ceph-19.0.0-1439.g5dcb190d.el9.x86_64.r 14 kB/s | 6.6 kB 00:00 2024-03-25T16:36:02.884 INFO:teuthology.orchestra.run.smithi001.stdout:(2/36): ceph-mgr-19.0.0-1439.g5dcb190d.el9.x86_ 249 kB/s | 1.5 MB 00:06 2024-03-25T16:36:03.634 INFO:teuthology.orchestra.run.smithi001.stdout:(3/36): ceph-mds-19.0.0-1439.g5dcb190d.el9.x86_ 317 kB/s | 2.1 MB 00:06 2024-03-25T16:36:04.243 INFO:teuthology.orchestra.run.smithi001.stdout:(4/36): ceph-mgr-modules-core-19.0.0-1439.g5dcb 407 kB/s | 247 kB 00:00 2024-03-25T16:36:04.426 INFO:teuthology.orchestra.run.smithi001.stdout:(5/36): lua-5.4.4-4.el9.x86_64.rpm 1.0 MB/s | 187 kB 00:00 2024-03-25T16:36:04.493 INFO:teuthology.orchestra.run.smithi001.stdout:(6/36): python3-mako-1.1.4-6.el9.noarch.rpm 2.1 MB/s | 144 kB 00:00 2024-03-25T16:36:04.527 INFO:teuthology.orchestra.run.smithi001.stdout:(7/36): python3-markupsafe-1.1.1-12.el9.x86_64. 951 kB/s | 32 kB 00:00 2024-03-25T16:36:04.569 INFO:teuthology.orchestra.run.smithi001.stdout:(8/36): python3-pytz-2021.1-5.el9.noarch.rpm 1.1 MB/s | 47 kB 00:00 2024-03-25T16:36:04.611 INFO:teuthology.orchestra.run.smithi001.stdout:(9/36): python3-toml-0.10.2-6.el9.noarch.rpm 930 kB/s | 39 kB 00:00 2024-03-25T16:36:04.678 INFO:teuthology.orchestra.run.smithi001.stdout:(10/36): python3-cffi-1.14.5-5.el9.x86_64.rpm 3.5 MB/s | 241 kB 00:00 2024-03-25T16:36:04.786 INFO:teuthology.orchestra.run.smithi001.stdout:(11/36): python3-cryptography-36.0.1-4.el9.x86_ 11 MB/s | 1.1 MB 00:00 2024-03-25T16:36:04.828 INFO:teuthology.orchestra.run.smithi001.stdout:(12/36): python3-ply-3.11-14.el9.noarch.rpm 2.4 MB/s | 103 kB 00:00 2024-03-25T16:36:04.870 INFO:teuthology.orchestra.run.smithi001.stdout:(13/36): python3-pycparser-2.20-6.el9.noarch.rp 2.9 MB/s | 124 kB 00:00 2024-03-25T16:36:04.912 INFO:teuthology.orchestra.run.smithi001.stdout:(14/36): python3-urllib3-1.26.5-5.el9.noarch.rp 4.4 MB/s | 187 kB 00:00 2024-03-25T16:36:05.137 INFO:teuthology.orchestra.run.smithi001.stdout:(15/36): python3-requests-2.25.1-8.el9.noarch.r 555 kB/s | 125 kB 00:00 2024-03-25T16:36:05.735 INFO:teuthology.orchestra.run.smithi115.stdout:(6/6): ceph-test-19.0.0-1439.g5dcb190d.el9.x86_ 438 kB/s | 47 MB 01:50 2024-03-25T16:36:05.737 INFO:teuthology.orchestra.run.smithi115.stdout:-------------------------------------------------------------------------------- 2024-03-25T16:36:05.737 INFO:teuthology.orchestra.run.smithi115.stdout:Total 446 kB/s | 48 MB 01:50 2024-03-25T16:36:05.749 INFO:teuthology.orchestra.run.smithi115.stdout:Running transaction check 2024-03-25T16:36:05.763 INFO:teuthology.orchestra.run.smithi001.stdout:(16/36): lua-devel-5.4.4-4.el9.x86_64.rpm 36 kB/s | 22 kB 00:00 2024-03-25T16:36:05.768 INFO:teuthology.orchestra.run.smithi115.stdout:Transaction check succeeded. 2024-03-25T16:36:05.769 INFO:teuthology.orchestra.run.smithi115.stdout:Running transaction test 2024-03-25T16:36:05.980 INFO:teuthology.orchestra.run.smithi001.stdout:(17/36): luarocks-3.9.2-1.el9.noarch.rpm 699 kB/s | 151 kB 00:00 2024-03-25T16:36:06.063 INFO:teuthology.orchestra.run.smithi001.stdout:(18/36): python3-bcrypt-3.2.2-1.el9.x86_64.rpm 522 kB/s | 43 kB 00:00 2024-03-25T16:36:06.139 INFO:teuthology.orchestra.run.smithi001.stdout:(19/36): python3-cheroot-8.6.0-4.el9.noarch.rpm 2.2 MB/s | 172 kB 00:00 2024-03-25T16:36:06.148 INFO:teuthology.orchestra.run.smithi115.stdout:Transaction test succeeded. 2024-03-25T16:36:06.149 INFO:teuthology.orchestra.run.smithi115.stdout:Running transaction 2024-03-25T16:36:06.206 INFO:teuthology.orchestra.run.smithi001.stdout:(20/36): python3-cherrypy-18.6.1-2.el9.noarch.r 5.2 MB/s | 358 kB 00:00 2024-03-25T16:36:06.381 INFO:teuthology.orchestra.run.smithi001.stdout:(21/36): python3-jaraco-8.2.1-3.el9.noarch.rpm 61 kB/s | 11 kB 00:00 2024-03-25T16:36:06.431 INFO:teuthology.orchestra.run.smithi001.stdout:(22/36): python3-jaraco-classes-3.2.1-5.el9.noa 355 kB/s | 18 kB 00:00 2024-03-25T16:36:06.490 INFO:teuthology.orchestra.run.smithi001.stdout:(23/36): python3-jaraco-collections-3.0.0-8.el9 397 kB/s | 23 kB 00:00 2024-03-25T16:36:06.540 INFO:teuthology.orchestra.run.smithi001.stdout:(24/36): python3-jaraco-functools-3.5.0-2.el9.n 389 kB/s | 19 kB 00:00 2024-03-25T16:36:06.607 INFO:teuthology.orchestra.run.smithi001.stdout:(25/36): python3-jaraco-text-3.2.0-6.el9.noarch 295 kB/s | 20 kB 00:00 2024-03-25T16:36:06.674 INFO:teuthology.orchestra.run.smithi001.stdout:(26/36): python3-logutils-0.3.5-21.el9.noarch.r 698 kB/s | 46 kB 00:00 2024-03-25T16:36:06.724 INFO:teuthology.orchestra.run.smithi001.stdout:(27/36): python3-more-itertools-8.12.0-2.el9.no 1.5 MB/s | 79 kB 00:00 2024-03-25T16:36:06.731 INFO:teuthology.orchestra.run.smithi115.stdout: Preparing : 1/1 2024-03-25T16:36:06.775 INFO:teuthology.orchestra.run.smithi001.stdout:(28/36): python3-pecan-1.4.2-3.el9.noarch.rpm 5.3 MB/s | 272 kB 00:00 2024-03-25T16:36:06.801 INFO:teuthology.orchestra.run.smithi115.stdout: Installing : xmlstarlet-1.6.1-20.el9.x86_64 1/6 2024-03-25T16:36:06.825 INFO:teuthology.orchestra.run.smithi001.stdout:(29/36): python3-portend-3.1.0-2.el9.noarch.rpm 330 kB/s | 16 kB 00:00 2024-03-25T16:36:06.843 INFO:teuthology.orchestra.run.smithi115.stdout: Installing : socat-1.7.4.1-5.el9.x86_64 2/6 2024-03-25T16:36:06.875 INFO:teuthology.orchestra.run.smithi001.stdout:(30/36): python3-pyOpenSSL-21.0.0-1.el9.noarch. 1.8 MB/s | 90 kB 00:00 2024-03-25T16:36:06.883 INFO:teuthology.orchestra.run.smithi115.stdout: Installing : oniguruma-6.9.6-1.el9.5.x86_64 3/6 2024-03-25T16:36:06.917 INFO:teuthology.orchestra.run.smithi001.stdout:(31/36): python3-tempora-5.0.0-2.el9.noarch.rpm 863 kB/s | 36 kB 00:00 2024-03-25T16:36:06.931 INFO:teuthology.orchestra.run.smithi115.stdout: Installing : jq-1.6-16.el9.x86_64 4/6 2024-03-25T16:36:06.954 INFO:teuthology.orchestra.run.smithi115.stdout: Installing : libcephsqlite-2:19.0.0-1439.g5dcb190d.el9.x86_64 5/6 2024-03-25T16:36:06.976 INFO:teuthology.orchestra.run.smithi001.stdout:(32/36): python3-webob-1.8.7-6.el9.noarch.rpm 3.8 MB/s | 230 kB 00:00 2024-03-25T16:36:07.034 INFO:teuthology.orchestra.run.smithi001.stdout:(33/36): python3-werkzeug-2.0.3-3.el9.1.noarch. 7.2 MB/s | 427 kB 00:00 2024-03-25T16:36:07.076 INFO:teuthology.orchestra.run.smithi001.stdout:(34/36): python3-zc-lockfile-2.0-10.el9.noarch. 480 kB/s | 20 kB 00:00 2024-03-25T16:36:11.680 INFO:teuthology.orchestra.run.smithi115.stdout: Running scriptlet: libcephsqlite-2:19.0.0-1439.g5dcb190d.el9.x86_64 5/6 2024-03-25T16:36:11.762 INFO:teuthology.orchestra.run.smithi115.stdout: Installing : ceph-test-2:19.0.0-1439.g5dcb190d.el9.x86_64 6/6 2024-03-25T16:36:12.368 INFO:teuthology.orchestra.run.smithi001.stdout:(35/36): ceph-mon-19.0.0-1439.g5dcb190d.el9.x86 317 kB/s | 4.7 MB 00:15 2024-03-25T16:36:14.112 INFO:teuthology.orchestra.run.smithi115.stdout: Running scriptlet: ceph-test-2:19.0.0-1439.g5dcb190d.el9.x86_64 6/6 2024-03-25T16:36:14.112 INFO:teuthology.orchestra.run.smithi115.stdout: Verifying : ceph-test-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/6 2024-03-25T16:36:14.112 INFO:teuthology.orchestra.run.smithi115.stdout: Verifying : libcephsqlite-2:19.0.0-1439.g5dcb190d.el9.x86_64 2/6 2024-03-25T16:36:14.112 INFO:teuthology.orchestra.run.smithi115.stdout: Verifying : oniguruma-6.9.6-1.el9.5.x86_64 3/6 2024-03-25T16:36:14.112 INFO:teuthology.orchestra.run.smithi115.stdout: Verifying : socat-1.7.4.1-5.el9.x86_64 4/6 2024-03-25T16:36:14.113 INFO:teuthology.orchestra.run.smithi115.stdout: Verifying : xmlstarlet-1.6.1-20.el9.x86_64 5/6 2024-03-25T16:36:14.525 INFO:teuthology.orchestra.run.smithi115.stdout: Verifying : jq-1.6-16.el9.x86_64 6/6 2024-03-25T16:36:14.526 INFO:teuthology.orchestra.run.smithi115.stdout: 2024-03-25T16:36:14.526 INFO:teuthology.orchestra.run.smithi115.stdout:Installed: 2024-03-25T16:36:14.526 INFO:teuthology.orchestra.run.smithi115.stdout: ceph-test-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T16:36:14.526 INFO:teuthology.orchestra.run.smithi115.stdout: jq-1.6-16.el9.x86_64 2024-03-25T16:36:14.526 INFO:teuthology.orchestra.run.smithi115.stdout: libcephsqlite-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T16:36:14.526 INFO:teuthology.orchestra.run.smithi115.stdout: oniguruma-6.9.6-1.el9.5.x86_64 2024-03-25T16:36:14.526 INFO:teuthology.orchestra.run.smithi115.stdout: socat-1.7.4.1-5.el9.x86_64 2024-03-25T16:36:14.526 INFO:teuthology.orchestra.run.smithi115.stdout: xmlstarlet-1.6.1-20.el9.x86_64 2024-03-25T16:36:14.526 INFO:teuthology.orchestra.run.smithi115.stdout: 2024-03-25T16:36:14.526 INFO:teuthology.orchestra.run.smithi115.stdout:Complete! 2024-03-25T16:36:14.839 DEBUG:teuthology.orchestra.run.smithi115:> sudo yum -y install ceph 2024-03-25T16:36:15.411 INFO:teuthology.orchestra.run.smithi115.stdout:Last metadata expiration check: 0:03:30 ago on Mon 25 Mar 2024 04:32:45 PM UTC. 2024-03-25T16:36:15.572 INFO:teuthology.orchestra.run.smithi115.stdout:Dependencies resolved. 2024-03-25T16:36:15.575 INFO:teuthology.orchestra.run.smithi115.stdout:========================================================================================= 2024-03-25T16:36:15.575 INFO:teuthology.orchestra.run.smithi115.stdout: Package Arch Version Repository Size 2024-03-25T16:36:15.575 INFO:teuthology.orchestra.run.smithi115.stdout:========================================================================================= 2024-03-25T16:36:15.575 INFO:teuthology.orchestra.run.smithi115.stdout:Installing: 2024-03-25T16:36:15.575 INFO:teuthology.orchestra.run.smithi115.stdout: ceph x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 6.6 k 2024-03-25T16:36:15.575 INFO:teuthology.orchestra.run.smithi115.stdout:Installing dependencies: 2024-03-25T16:36:15.575 INFO:teuthology.orchestra.run.smithi115.stdout: ceph-mds x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 2.1 M 2024-03-25T16:36:15.575 INFO:teuthology.orchestra.run.smithi115.stdout: ceph-mgr x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 1.5 M 2024-03-25T16:36:15.575 INFO:teuthology.orchestra.run.smithi115.stdout: ceph-mgr-modules-core noarch 2:19.0.0-1439.g5dcb190d.el9 ceph-noarch 247 k 2024-03-25T16:36:15.575 INFO:teuthology.orchestra.run.smithi115.stdout: ceph-mon x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 4.7 M 2024-03-25T16:36:15.575 INFO:teuthology.orchestra.run.smithi115.stdout: ceph-osd x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 17 M 2024-03-25T16:36:15.576 INFO:teuthology.orchestra.run.smithi115.stdout: lua x86_64 5.4.4-4.el9 CentOS-AppStream 187 k 2024-03-25T16:36:15.576 INFO:teuthology.orchestra.run.smithi115.stdout: lua-devel x86_64 5.4.4-4.el9 crb 22 k 2024-03-25T16:36:15.576 INFO:teuthology.orchestra.run.smithi115.stdout: luarocks noarch 3.9.2-1.el9 epel 151 k 2024-03-25T16:36:15.576 INFO:teuthology.orchestra.run.smithi115.stdout: python3-bcrypt x86_64 3.2.2-1.el9 epel 43 k 2024-03-25T16:36:15.576 INFO:teuthology.orchestra.run.smithi115.stdout: python3-cffi x86_64 1.14.5-5.el9 CentOS-BaseOS 241 k 2024-03-25T16:36:15.576 INFO:teuthology.orchestra.run.smithi115.stdout: python3-cheroot noarch 8.6.0-4.el9 epel 172 k 2024-03-25T16:36:15.576 INFO:teuthology.orchestra.run.smithi115.stdout: python3-cherrypy noarch 18.6.1-2.el9 epel 358 k 2024-03-25T16:36:15.576 INFO:teuthology.orchestra.run.smithi115.stdout: python3-cryptography x86_64 36.0.1-4.el9 CentOS-BaseOS 1.1 M 2024-03-25T16:36:15.576 INFO:teuthology.orchestra.run.smithi115.stdout: python3-jaraco noarch 8.2.1-3.el9 epel 11 k 2024-03-25T16:36:15.576 INFO:teuthology.orchestra.run.smithi115.stdout: python3-jaraco-classes noarch 3.2.1-5.el9 epel 18 k 2024-03-25T16:36:15.576 INFO:teuthology.orchestra.run.smithi115.stdout: python3-jaraco-collections noarch 3.0.0-8.el9 epel 23 k 2024-03-25T16:36:15.576 INFO:teuthology.orchestra.run.smithi115.stdout: python3-jaraco-functools noarch 3.5.0-2.el9 epel 19 k 2024-03-25T16:36:15.576 INFO:teuthology.orchestra.run.smithi115.stdout: python3-jaraco-text noarch 3.2.0-6.el9 epel 20 k 2024-03-25T16:36:15.576 INFO:teuthology.orchestra.run.smithi115.stdout: python3-logutils noarch 0.3.5-21.el9 epel 46 k 2024-03-25T16:36:15.577 INFO:teuthology.orchestra.run.smithi115.stdout: python3-mako noarch 1.1.4-6.el9 CentOS-AppStream 144 k 2024-03-25T16:36:15.577 INFO:teuthology.orchestra.run.smithi115.stdout: python3-markupsafe x86_64 1.1.1-12.el9 CentOS-AppStream 32 k 2024-03-25T16:36:15.577 INFO:teuthology.orchestra.run.smithi115.stdout: python3-more-itertools noarch 8.12.0-2.el9 epel 79 k 2024-03-25T16:36:15.577 INFO:teuthology.orchestra.run.smithi115.stdout: python3-pecan noarch 1.4.2-3.el9 epel 272 k 2024-03-25T16:36:15.577 INFO:teuthology.orchestra.run.smithi115.stdout: python3-ply noarch 3.11-14.el9 CentOS-BaseOS 103 k 2024-03-25T16:36:15.577 INFO:teuthology.orchestra.run.smithi115.stdout: python3-portend noarch 3.1.0-2.el9 epel 16 k 2024-03-25T16:36:15.577 INFO:teuthology.orchestra.run.smithi115.stdout: python3-pyOpenSSL noarch 21.0.0-1.el9 epel 90 k 2024-03-25T16:36:15.577 INFO:teuthology.orchestra.run.smithi115.stdout: python3-pycparser noarch 2.20-6.el9 CentOS-BaseOS 124 k 2024-03-25T16:36:15.577 INFO:teuthology.orchestra.run.smithi115.stdout: python3-pytz noarch 2021.1-5.el9 CentOS-AppStream 47 k 2024-03-25T16:36:15.577 INFO:teuthology.orchestra.run.smithi115.stdout: python3-requests noarch 2.25.1-8.el9 baseos 125 k 2024-03-25T16:36:15.577 INFO:teuthology.orchestra.run.smithi115.stdout: python3-tempora noarch 5.0.0-2.el9 epel 36 k 2024-03-25T16:36:15.577 INFO:teuthology.orchestra.run.smithi115.stdout: python3-toml noarch 0.10.2-6.el9 CentOS-AppStream 39 k 2024-03-25T16:36:15.577 INFO:teuthology.orchestra.run.smithi115.stdout: python3-urllib3 noarch 1.26.5-5.el9 CentOS-BaseOS 187 k 2024-03-25T16:36:15.577 INFO:teuthology.orchestra.run.smithi115.stdout: python3-webob noarch 1.8.7-6.el9 epel 230 k 2024-03-25T16:36:15.577 INFO:teuthology.orchestra.run.smithi115.stdout: python3-werkzeug noarch 2.0.3-3.el9.1 epel 427 k 2024-03-25T16:36:15.577 INFO:teuthology.orchestra.run.smithi115.stdout: python3-zc-lockfile noarch 2.0-10.el9 epel 20 k 2024-03-25T16:36:15.578 INFO:teuthology.orchestra.run.smithi115.stdout: 2024-03-25T16:36:15.578 INFO:teuthology.orchestra.run.smithi115.stdout:Transaction Summary 2024-03-25T16:36:15.578 INFO:teuthology.orchestra.run.smithi115.stdout:========================================================================================= 2024-03-25T16:36:15.578 INFO:teuthology.orchestra.run.smithi115.stdout:Install 36 Packages 2024-03-25T16:36:15.578 INFO:teuthology.orchestra.run.smithi115.stdout: 2024-03-25T16:36:15.578 INFO:teuthology.orchestra.run.smithi115.stdout:Total download size: 30 M 2024-03-25T16:36:15.578 INFO:teuthology.orchestra.run.smithi115.stdout:Installed size: 107 M 2024-03-25T16:36:15.578 INFO:teuthology.orchestra.run.smithi115.stdout:Downloading Packages: 2024-03-25T16:36:16.514 INFO:teuthology.orchestra.run.smithi115.stdout:(1/36): ceph-19.0.0-1439.g5dcb190d.el9.x86_64.r 14 kB/s | 6.6 kB 00:00 2024-03-25T16:36:19.497 INFO:teuthology.orchestra.run.smithi115.stdout:(2/36): ceph-mgr-19.0.0-1439.g5dcb190d.el9.x86_ 443 kB/s | 1.5 MB 00:03 2024-03-25T16:36:19.722 INFO:teuthology.orchestra.run.smithi115.stdout:(3/36): ceph-mds-19.0.0-1439.g5dcb190d.el9.x86_ 593 kB/s | 2.1 MB 00:03 2024-03-25T16:36:20.322 INFO:teuthology.orchestra.run.smithi115.stdout:(4/36): ceph-mgr-modules-core-19.0.0-1439.g5dcb 412 kB/s | 247 kB 00:00 2024-03-25T16:36:20.439 INFO:teuthology.orchestra.run.smithi115.stdout:(5/36): lua-5.4.4-4.el9.x86_64.rpm 1.6 MB/s | 187 kB 00:00 2024-03-25T16:36:20.473 INFO:teuthology.orchestra.run.smithi115.stdout:(6/36): python3-mako-1.1.4-6.el9.noarch.rpm 4.2 MB/s | 144 kB 00:00 2024-03-25T16:36:20.498 INFO:teuthology.orchestra.run.smithi115.stdout:(7/36): python3-markupsafe-1.1.1-12.el9.x86_64. 1.2 MB/s | 32 kB 00:00 2024-03-25T16:36:20.523 INFO:teuthology.orchestra.run.smithi115.stdout:(8/36): python3-pytz-2021.1-5.el9.noarch.rpm 1.9 MB/s | 47 kB 00:00 2024-03-25T16:36:20.548 INFO:teuthology.orchestra.run.smithi115.stdout:(9/36): python3-toml-0.10.2-6.el9.noarch.rpm 1.5 MB/s | 39 kB 00:00 2024-03-25T16:36:20.591 INFO:teuthology.orchestra.run.smithi115.stdout:(10/36): python3-cffi-1.14.5-5.el9.x86_64.rpm 5.6 MB/s | 241 kB 00:00 2024-03-25T16:36:20.657 INFO:teuthology.orchestra.run.smithi115.stdout:(11/36): python3-cryptography-36.0.1-4.el9.x86_ 17 MB/s | 1.1 MB 00:00 2024-03-25T16:36:20.691 INFO:teuthology.orchestra.run.smithi115.stdout:(12/36): python3-ply-3.11-14.el9.noarch.rpm 3.0 MB/s | 103 kB 00:00 2024-03-25T16:36:20.725 INFO:teuthology.orchestra.run.smithi115.stdout:(13/36): python3-pycparser-2.20-6.el9.noarch.rp 3.6 MB/s | 124 kB 00:00 2024-03-25T16:36:20.758 INFO:teuthology.orchestra.run.smithi115.stdout:(14/36): python3-urllib3-1.26.5-5.el9.noarch.rp 5.5 MB/s | 187 kB 00:00 2024-03-25T16:36:20.917 INFO:teuthology.orchestra.run.smithi115.stdout:(15/36): python3-requests-2.25.1-8.el9.noarch.r 789 kB/s | 125 kB 00:00 2024-03-25T16:36:21.484 INFO:teuthology.orchestra.run.smithi115.stdout:(16/36): lua-devel-5.4.4-4.el9.x86_64.rpm 39 kB/s | 22 kB 00:00 2024-03-25T16:36:21.867 INFO:teuthology.orchestra.run.smithi115.stdout:(17/36): luarocks-3.9.2-1.el9.noarch.rpm 395 kB/s | 151 kB 00:00 2024-03-25T16:36:21.951 INFO:teuthology.orchestra.run.smithi115.stdout:(18/36): python3-bcrypt-3.2.2-1.el9.x86_64.rpm 522 kB/s | 43 kB 00:00 2024-03-25T16:36:22.059 INFO:teuthology.orchestra.run.smithi115.stdout:(19/36): python3-cheroot-8.6.0-4.el9.noarch.rpm 1.5 MB/s | 172 kB 00:00 2024-03-25T16:36:22.193 INFO:teuthology.orchestra.run.smithi115.stdout:(20/36): python3-cherrypy-18.6.1-2.el9.noarch.r 2.6 MB/s | 358 kB 00:00 2024-03-25T16:36:22.285 INFO:teuthology.orchestra.run.smithi115.stdout:(21/36): python3-jaraco-8.2.1-3.el9.noarch.rpm 116 kB/s | 11 kB 00:00 2024-03-25T16:36:22.352 INFO:teuthology.orchestra.run.smithi115.stdout:(22/36): python3-jaraco-classes-3.2.1-5.el9.noa 265 kB/s | 18 kB 00:00 2024-03-25T16:36:22.419 INFO:teuthology.orchestra.run.smithi115.stdout:(23/36): python3-jaraco-collections-3.0.0-8.el9 348 kB/s | 23 kB 00:00 2024-03-25T16:36:22.486 INFO:teuthology.orchestra.run.smithi115.stdout:(24/36): python3-jaraco-functools-3.5.0-2.el9.n 291 kB/s | 19 kB 00:00 2024-03-25T16:36:22.553 INFO:teuthology.orchestra.run.smithi115.stdout:(25/36): python3-jaraco-text-3.2.0-6.el9.noarch 295 kB/s | 20 kB 00:00 2024-03-25T16:36:22.620 INFO:teuthology.orchestra.run.smithi115.stdout:(26/36): python3-logutils-0.3.5-21.el9.noarch.r 695 kB/s | 46 kB 00:00 2024-03-25T16:36:22.712 INFO:teuthology.orchestra.run.smithi115.stdout:(27/36): python3-more-itertools-8.12.0-2.el9.no 860 kB/s | 79 kB 00:00 2024-03-25T16:36:22.813 INFO:teuthology.orchestra.run.smithi115.stdout:(28/36): python3-pecan-1.4.2-3.el9.noarch.rpm 2.7 MB/s | 272 kB 00:00 2024-03-25T16:36:22.879 INFO:teuthology.orchestra.run.smithi115.stdout:(29/36): python3-portend-3.1.0-2.el9.noarch.rpm 247 kB/s | 16 kB 00:00 2024-03-25T16:36:22.955 INFO:teuthology.orchestra.run.smithi115.stdout:(30/36): python3-pyOpenSSL-21.0.0-1.el9.noarch. 1.2 MB/s | 90 kB 00:00 2024-03-25T16:36:23.030 INFO:teuthology.orchestra.run.smithi115.stdout:(31/36): python3-tempora-5.0.0-2.el9.noarch.rpm 477 kB/s | 36 kB 00:00 2024-03-25T16:36:23.114 INFO:teuthology.orchestra.run.smithi115.stdout:(32/36): python3-webob-1.8.7-6.el9.noarch.rpm 2.7 MB/s | 230 kB 00:00 2024-03-25T16:36:23.214 INFO:teuthology.orchestra.run.smithi115.stdout:(33/36): python3-werkzeug-2.0.3-3.el9.1.noarch. 4.2 MB/s | 427 kB 00:00 2024-03-25T16:36:23.298 INFO:teuthology.orchestra.run.smithi115.stdout:(34/36): python3-zc-lockfile-2.0-10.el9.noarch. 239 kB/s | 20 kB 00:00 2024-03-25T16:36:29.938 INFO:teuthology.orchestra.run.smithi115.stdout:(35/36): ceph-mon-19.0.0-1439.g5dcb190d.el9.x86 357 kB/s | 4.7 MB 00:13 2024-03-25T16:36:52.201 INFO:teuthology.orchestra.run.smithi115.stdout:(36/36): ceph-osd-19.0.0-1439.g5dcb190d.el9.x86 525 kB/s | 17 MB 00:32 2024-03-25T16:36:52.206 INFO:teuthology.orchestra.run.smithi115.stdout:-------------------------------------------------------------------------------- 2024-03-25T16:36:52.206 INFO:teuthology.orchestra.run.smithi115.stdout:Total 829 kB/s | 30 MB 00:36 2024-03-25T16:36:52.228 INFO:teuthology.orchestra.run.smithi115.stdout:Running transaction check 2024-03-25T16:36:52.271 INFO:teuthology.orchestra.run.smithi115.stdout:Transaction check succeeded. 2024-03-25T16:36:52.271 INFO:teuthology.orchestra.run.smithi115.stdout:Running transaction test 2024-03-25T16:36:52.694 INFO:teuthology.orchestra.run.smithi115.stdout:Transaction test succeeded. 2024-03-25T16:36:52.694 INFO:teuthology.orchestra.run.smithi115.stdout:Running transaction 2024-03-25T16:36:54.014 INFO:teuthology.orchestra.run.smithi115.stdout: Preparing : 1/1 2024-03-25T16:36:54.078 INFO:teuthology.orchestra.run.smithi115.stdout: Installing : python3-more-itertools-8.12.0-2.el9.noarch 1/36 2024-03-25T16:36:54.115 INFO:teuthology.orchestra.run.smithi115.stdout: Installing : python3-jaraco-8.2.1-3.el9.noarch 2/36 2024-03-25T16:36:54.175 INFO:teuthology.orchestra.run.smithi115.stdout: Installing : lua-5.4.4-4.el9.x86_64 3/36 2024-03-25T16:36:54.250 INFO:teuthology.orchestra.run.smithi115.stdout: Installing : lua-devel-5.4.4-4.el9.x86_64 4/36 2024-03-25T16:36:54.288 INFO:teuthology.orchestra.run.smithi115.stdout: Installing : luarocks-3.9.2-1.el9.noarch 5/36 2024-03-25T16:36:54.320 INFO:teuthology.orchestra.run.smithi115.stdout: Installing : python3-jaraco-classes-3.2.1-5.el9.noarch 6/36 2024-03-25T16:36:54.411 INFO:teuthology.orchestra.run.smithi115.stdout: Installing : python3-zc-lockfile-2.0-10.el9.noarch 7/36 2024-03-25T16:36:54.499 INFO:teuthology.orchestra.run.smithi115.stdout: Installing : python3-werkzeug-2.0.3-3.el9.1.noarch 8/36 2024-03-25T16:36:54.540 INFO:teuthology.orchestra.run.smithi115.stdout: Installing : python3-webob-1.8.7-6.el9.noarch 9/36 2024-03-25T16:36:54.611 INFO:teuthology.orchestra.run.smithi115.stdout: Installing : python3-logutils-0.3.5-21.el9.noarch 10/36 2024-03-25T16:36:54.665 INFO:teuthology.orchestra.run.smithi115.stdout: Installing : python3-urllib3-1.26.5-5.el9.noarch 11/36 2024-03-25T16:36:54.706 INFO:teuthology.orchestra.run.smithi115.stdout: Installing : python3-requests-2.25.1-8.el9.noarch 12/36 2024-03-25T16:36:54.752 INFO:teuthology.orchestra.run.smithi115.stdout: Installing : python3-ply-3.11-14.el9.noarch 13/36 2024-03-25T16:36:54.808 INFO:teuthology.orchestra.run.smithi115.stdout: Installing : python3-pycparser-2.20-6.el9.noarch 14/36 2024-03-25T16:36:54.993 INFO:teuthology.orchestra.run.smithi115.stdout: Installing : python3-cffi-1.14.5-5.el9.x86_64 15/36 2024-03-25T16:36:55.050 INFO:teuthology.orchestra.run.smithi115.stdout: Installing : python3-cryptography-36.0.1-4.el9.x86_64 16/36 2024-03-25T16:36:55.088 INFO:teuthology.orchestra.run.smithi115.stdout: Installing : python3-pyOpenSSL-21.0.0-1.el9.noarch 17/36 2024-03-25T16:36:55.132 INFO:teuthology.orchestra.run.smithi115.stdout: Installing : python3-bcrypt-3.2.2-1.el9.x86_64 18/36 2024-03-25T16:36:55.162 INFO:teuthology.orchestra.run.smithi115.stdout: Installing : python3-toml-0.10.2-6.el9.noarch 19/36 2024-03-25T16:36:55.237 INFO:teuthology.orchestra.run.smithi115.stdout: Installing : python3-jaraco-functools-3.5.0-2.el9.noarch 20/36 2024-03-25T16:36:55.272 INFO:teuthology.orchestra.run.smithi115.stdout: Installing : python3-cheroot-8.6.0-4.el9.noarch 21/36 2024-03-25T16:36:55.306 INFO:teuthology.orchestra.run.smithi115.stdout: Installing : python3-jaraco-text-3.2.0-6.el9.noarch 22/36 2024-03-25T16:36:55.346 INFO:teuthology.orchestra.run.smithi115.stdout: Installing : python3-jaraco-collections-3.0.0-8.el9.noarch 23/36 2024-03-25T16:36:55.381 INFO:teuthology.orchestra.run.smithi115.stdout: Installing : python3-pytz-2021.1-5.el9.noarch 24/36 2024-03-25T16:36:55.416 INFO:teuthology.orchestra.run.smithi115.stdout: Installing : python3-tempora-5.0.0-2.el9.noarch 25/36 2024-03-25T16:36:55.430 INFO:teuthology.orchestra.run.smithi001.stdout:(36/36): ceph-osd-19.0.0-1439.g5dcb190d.el9.x86 327 kB/s | 17 MB 00:52 2024-03-25T16:36:55.435 INFO:teuthology.orchestra.run.smithi001.stdout:-------------------------------------------------------------------------------- 2024-03-25T16:36:55.435 INFO:teuthology.orchestra.run.smithi001.stdout:Total 514 kB/s | 30 MB 00:59 2024-03-25T16:36:55.460 INFO:teuthology.orchestra.run.smithi001.stdout:Running transaction check 2024-03-25T16:36:55.504 INFO:teuthology.orchestra.run.smithi115.stdout: Installing : python3-portend-3.1.0-2.el9.noarch 26/36 2024-03-25T16:36:55.506 INFO:teuthology.orchestra.run.smithi001.stdout:Transaction check succeeded. 2024-03-25T16:36:55.506 INFO:teuthology.orchestra.run.smithi001.stdout:Running transaction test 2024-03-25T16:36:55.538 INFO:teuthology.orchestra.run.smithi115.stdout: Installing : python3-cherrypy-18.6.1-2.el9.noarch 27/36 2024-03-25T16:36:55.610 INFO:teuthology.orchestra.run.smithi115.stdout: Installing : python3-markupsafe-1.1.1-12.el9.x86_64 28/36 2024-03-25T16:36:55.747 INFO:teuthology.orchestra.run.smithi115.stdout: Installing : python3-mako-1.1.4-6.el9.noarch 29/36 2024-03-25T16:36:55.865 INFO:teuthology.orchestra.run.smithi115.stdout: Installing : python3-pecan-1.4.2-3.el9.noarch 30/36 2024-03-25T16:36:55.925 INFO:teuthology.orchestra.run.smithi001.stdout:Transaction test succeeded. 2024-03-25T16:36:55.926 INFO:teuthology.orchestra.run.smithi001.stdout:Running transaction 2024-03-25T16:36:56.022 INFO:teuthology.orchestra.run.smithi115.stdout: Installing : ceph-mgr-modules-core-2:19.0.0-1439.g5dcb190d.el9. 31/36 2024-03-25T16:36:56.043 INFO:teuthology.orchestra.run.smithi115.stdout: Installing : ceph-mgr-2:19.0.0-1439.g5dcb190d.el9.x86_64 32/36 2024-03-25T16:36:56.068 INFO:teuthology.orchestra.run.smithi115.stdout: Running scriptlet: ceph-mgr-2:19.0.0-1439.g5dcb190d.el9.x86_64 32/36 2024-03-25T16:36:56.068 INFO:teuthology.orchestra.run.smithi115.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-25T16:36:56.068 INFO:teuthology.orchestra.run.smithi115.stdout:Invalid unit name "ceph-mgr@*.service" escaped as "ceph-mgr@\x2a.service". 2024-03-25T16:36:56.068 INFO:teuthology.orchestra.run.smithi115.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mgr.target → /usr/lib/systemd/system/ceph-mgr.target. 2024-03-25T16:36:56.068 INFO:teuthology.orchestra.run.smithi115.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mgr.target → /usr/lib/systemd/system/ceph-mgr.target. 2024-03-25T16:36:56.068 INFO:teuthology.orchestra.run.smithi115.stdout: 2024-03-25T16:36:57.405 INFO:teuthology.orchestra.run.smithi001.stdout: Preparing : 1/1 2024-03-25T16:36:57.475 INFO:teuthology.orchestra.run.smithi001.stdout: Installing : python3-more-itertools-8.12.0-2.el9.noarch 1/36 2024-03-25T16:36:57.508 INFO:teuthology.orchestra.run.smithi001.stdout: Installing : python3-jaraco-8.2.1-3.el9.noarch 2/36 2024-03-25T16:36:57.535 INFO:teuthology.orchestra.run.smithi001.stdout: Installing : lua-5.4.4-4.el9.x86_64 3/36 2024-03-25T16:36:57.569 INFO:teuthology.orchestra.run.smithi115.stdout: Installing : ceph-osd-2:19.0.0-1439.g5dcb190d.el9.x86_64 33/36 2024-03-25T16:36:57.596 INFO:teuthology.orchestra.run.smithi115.stdout: Running scriptlet: ceph-osd-2:19.0.0-1439.g5dcb190d.el9.x86_64 33/36 2024-03-25T16:36:57.596 INFO:teuthology.orchestra.run.smithi115.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-25T16:36:57.596 INFO:teuthology.orchestra.run.smithi115.stdout:Invalid unit name "ceph-osd@*.service" escaped as "ceph-osd@\x2a.service". 2024-03-25T16:36:57.596 INFO:teuthology.orchestra.run.smithi115.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-osd.target → /usr/lib/systemd/system/ceph-osd.target. 2024-03-25T16:36:57.597 INFO:teuthology.orchestra.run.smithi115.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-osd.target → /usr/lib/systemd/system/ceph-osd.target. 2024-03-25T16:36:57.597 INFO:teuthology.orchestra.run.smithi115.stdout: 2024-03-25T16:36:57.608 INFO:teuthology.orchestra.run.smithi001.stdout: Installing : lua-devel-5.4.4-4.el9.x86_64 4/36 2024-03-25T16:36:57.643 INFO:teuthology.orchestra.run.smithi001.stdout: Installing : luarocks-3.9.2-1.el9.noarch 5/36 2024-03-25T16:36:57.675 INFO:teuthology.orchestra.run.smithi001.stdout: Installing : python3-jaraco-classes-3.2.1-5.el9.noarch 6/36 2024-03-25T16:36:57.770 INFO:teuthology.orchestra.run.smithi001.stdout: Installing : python3-zc-lockfile-2.0-10.el9.noarch 7/36 2024-03-25T16:36:57.964 INFO:teuthology.orchestra.run.smithi001.stdout: Installing : python3-werkzeug-2.0.3-3.el9.1.noarch 8/36 2024-03-25T16:36:58.020 INFO:teuthology.orchestra.run.smithi001.stdout: Installing : python3-webob-1.8.7-6.el9.noarch 9/36 2024-03-25T16:36:58.037 INFO:teuthology.orchestra.run.smithi115.stdout: Installing : ceph-mon-2:19.0.0-1439.g5dcb190d.el9.x86_64 34/36 2024-03-25T16:36:58.060 INFO:teuthology.orchestra.run.smithi115.stdout: Running scriptlet: ceph-mon-2:19.0.0-1439.g5dcb190d.el9.x86_64 34/36 2024-03-25T16:36:58.060 INFO:teuthology.orchestra.run.smithi115.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-25T16:36:58.060 INFO:teuthology.orchestra.run.smithi115.stdout:Invalid unit name "ceph-mon@*.service" escaped as "ceph-mon@\x2a.service". 2024-03-25T16:36:58.061 INFO:teuthology.orchestra.run.smithi115.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mon.target → /usr/lib/systemd/system/ceph-mon.target. 2024-03-25T16:36:58.061 INFO:teuthology.orchestra.run.smithi115.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mon.target → /usr/lib/systemd/system/ceph-mon.target. 2024-03-25T16:36:58.061 INFO:teuthology.orchestra.run.smithi115.stdout: 2024-03-25T16:36:58.102 INFO:teuthology.orchestra.run.smithi001.stdout: Installing : python3-logutils-0.3.5-21.el9.noarch 10/36 2024-03-25T16:36:58.162 INFO:teuthology.orchestra.run.smithi001.stdout: Installing : python3-urllib3-1.26.5-5.el9.noarch 11/36 2024-03-25T16:36:58.200 INFO:teuthology.orchestra.run.smithi001.stdout: Installing : python3-requests-2.25.1-8.el9.noarch 12/36 2024-03-25T16:36:58.259 INFO:teuthology.orchestra.run.smithi001.stdout: Installing : python3-ply-3.11-14.el9.noarch 13/36 2024-03-25T16:36:58.270 INFO:teuthology.orchestra.run.smithi115.stdout: Installing : ceph-mds-2:19.0.0-1439.g5dcb190d.el9.x86_64 35/36 2024-03-25T16:36:58.293 INFO:teuthology.orchestra.run.smithi115.stdout: Running scriptlet: ceph-mds-2:19.0.0-1439.g5dcb190d.el9.x86_64 35/36 2024-03-25T16:36:58.293 INFO:teuthology.orchestra.run.smithi115.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-25T16:36:58.293 INFO:teuthology.orchestra.run.smithi115.stdout:Invalid unit name "ceph-mds@*.service" escaped as "ceph-mds@\x2a.service". 2024-03-25T16:36:58.293 INFO:teuthology.orchestra.run.smithi115.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mds.target → /usr/lib/systemd/system/ceph-mds.target. 2024-03-25T16:36:58.293 INFO:teuthology.orchestra.run.smithi115.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mds.target → /usr/lib/systemd/system/ceph-mds.target. 2024-03-25T16:36:58.293 INFO:teuthology.orchestra.run.smithi115.stdout: 2024-03-25T16:36:58.322 INFO:teuthology.orchestra.run.smithi001.stdout: Installing : python3-pycparser-2.20-6.el9.noarch 14/36 2024-03-25T16:36:58.360 INFO:teuthology.orchestra.run.smithi115.stdout: Installing : ceph-2:19.0.0-1439.g5dcb190d.el9.x86_64 36/36 2024-03-25T16:36:58.515 INFO:teuthology.orchestra.run.smithi001.stdout: Installing : python3-cffi-1.14.5-5.el9.x86_64 15/36 2024-03-25T16:36:58.563 INFO:teuthology.orchestra.run.smithi001.stdout: Installing : python3-cryptography-36.0.1-4.el9.x86_64 16/36 2024-03-25T16:36:58.602 INFO:teuthology.orchestra.run.smithi001.stdout: Installing : python3-pyOpenSSL-21.0.0-1.el9.noarch 17/36 2024-03-25T16:36:58.636 INFO:teuthology.orchestra.run.smithi001.stdout: Installing : python3-bcrypt-3.2.2-1.el9.x86_64 18/36 2024-03-25T16:36:58.666 INFO:teuthology.orchestra.run.smithi001.stdout: Installing : python3-toml-0.10.2-6.el9.noarch 19/36 2024-03-25T16:36:58.745 INFO:teuthology.orchestra.run.smithi001.stdout: Installing : python3-jaraco-functools-3.5.0-2.el9.noarch 20/36 2024-03-25T16:36:58.776 INFO:teuthology.orchestra.run.smithi001.stdout: Installing : python3-cheroot-8.6.0-4.el9.noarch 21/36 2024-03-25T16:36:58.809 INFO:teuthology.orchestra.run.smithi001.stdout: Installing : python3-jaraco-text-3.2.0-6.el9.noarch 22/36 2024-03-25T16:36:58.858 INFO:teuthology.orchestra.run.smithi001.stdout: Installing : python3-jaraco-collections-3.0.0-8.el9.noarch 23/36 2024-03-25T16:36:58.910 INFO:teuthology.orchestra.run.smithi001.stdout: Installing : python3-pytz-2021.1-5.el9.noarch 24/36 2024-03-25T16:36:58.944 INFO:teuthology.orchestra.run.smithi001.stdout: Installing : python3-tempora-5.0.0-2.el9.noarch 25/36 2024-03-25T16:36:59.038 INFO:teuthology.orchestra.run.smithi001.stdout: Installing : python3-portend-3.1.0-2.el9.noarch 26/36 2024-03-25T16:36:59.075 INFO:teuthology.orchestra.run.smithi001.stdout: Installing : python3-cherrypy-18.6.1-2.el9.noarch 27/36 2024-03-25T16:36:59.151 INFO:teuthology.orchestra.run.smithi001.stdout: Installing : python3-markupsafe-1.1.1-12.el9.x86_64 28/36 2024-03-25T16:36:59.292 INFO:teuthology.orchestra.run.smithi001.stdout: Installing : python3-mako-1.1.4-6.el9.noarch 29/36 2024-03-25T16:36:59.424 INFO:teuthology.orchestra.run.smithi001.stdout: Installing : python3-pecan-1.4.2-3.el9.noarch 30/36 2024-03-25T16:36:59.599 INFO:teuthology.orchestra.run.smithi001.stdout: Installing : ceph-mgr-modules-core-2:19.0.0-1439.g5dcb190d.el9. 31/36 2024-03-25T16:36:59.630 INFO:teuthology.orchestra.run.smithi001.stdout: Installing : ceph-mgr-2:19.0.0-1439.g5dcb190d.el9.x86_64 32/36 2024-03-25T16:36:59.655 INFO:teuthology.orchestra.run.smithi001.stdout: Running scriptlet: ceph-mgr-2:19.0.0-1439.g5dcb190d.el9.x86_64 32/36 2024-03-25T16:36:59.655 INFO:teuthology.orchestra.run.smithi001.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-25T16:36:59.655 INFO:teuthology.orchestra.run.smithi001.stdout:Invalid unit name "ceph-mgr@*.service" escaped as "ceph-mgr@\x2a.service". 2024-03-25T16:36:59.655 INFO:teuthology.orchestra.run.smithi001.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mgr.target → /usr/lib/systemd/system/ceph-mgr.target. 2024-03-25T16:36:59.655 INFO:teuthology.orchestra.run.smithi001.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mgr.target → /usr/lib/systemd/system/ceph-mgr.target. 2024-03-25T16:36:59.655 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-25T16:37:00.077 INFO:teuthology.orchestra.run.smithi115.stdout: Running scriptlet: ceph-2:19.0.0-1439.g5dcb190d.el9.x86_64 36/36 2024-03-25T16:37:00.077 INFO:teuthology.orchestra.run.smithi115.stdout: Verifying : ceph-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/36 2024-03-25T16:37:00.077 INFO:teuthology.orchestra.run.smithi115.stdout: Verifying : ceph-mds-2:19.0.0-1439.g5dcb190d.el9.x86_64 2/36 2024-03-25T16:37:00.077 INFO:teuthology.orchestra.run.smithi115.stdout: Verifying : ceph-mgr-2:19.0.0-1439.g5dcb190d.el9.x86_64 3/36 2024-03-25T16:37:00.077 INFO:teuthology.orchestra.run.smithi115.stdout: Verifying : ceph-mon-2:19.0.0-1439.g5dcb190d.el9.x86_64 4/36 2024-03-25T16:37:00.077 INFO:teuthology.orchestra.run.smithi115.stdout: Verifying : ceph-osd-2:19.0.0-1439.g5dcb190d.el9.x86_64 5/36 2024-03-25T16:37:00.077 INFO:teuthology.orchestra.run.smithi115.stdout: Verifying : ceph-mgr-modules-core-2:19.0.0-1439.g5dcb190d.el9. 6/36 2024-03-25T16:37:00.078 INFO:teuthology.orchestra.run.smithi115.stdout: Verifying : lua-5.4.4-4.el9.x86_64 7/36 2024-03-25T16:37:00.078 INFO:teuthology.orchestra.run.smithi115.stdout: Verifying : python3-mako-1.1.4-6.el9.noarch 8/36 2024-03-25T16:37:00.078 INFO:teuthology.orchestra.run.smithi115.stdout: Verifying : python3-markupsafe-1.1.1-12.el9.x86_64 9/36 2024-03-25T16:37:00.078 INFO:teuthology.orchestra.run.smithi115.stdout: Verifying : python3-pytz-2021.1-5.el9.noarch 10/36 2024-03-25T16:37:00.078 INFO:teuthology.orchestra.run.smithi115.stdout: Verifying : python3-toml-0.10.2-6.el9.noarch 11/36 2024-03-25T16:37:00.078 INFO:teuthology.orchestra.run.smithi115.stdout: Verifying : python3-cffi-1.14.5-5.el9.x86_64 12/36 2024-03-25T16:37:00.078 INFO:teuthology.orchestra.run.smithi115.stdout: Verifying : python3-cryptography-36.0.1-4.el9.x86_64 13/36 2024-03-25T16:37:00.078 INFO:teuthology.orchestra.run.smithi115.stdout: Verifying : python3-ply-3.11-14.el9.noarch 14/36 2024-03-25T16:37:00.078 INFO:teuthology.orchestra.run.smithi115.stdout: Verifying : python3-pycparser-2.20-6.el9.noarch 15/36 2024-03-25T16:37:00.078 INFO:teuthology.orchestra.run.smithi115.stdout: Verifying : python3-urllib3-1.26.5-5.el9.noarch 16/36 2024-03-25T16:37:00.078 INFO:teuthology.orchestra.run.smithi115.stdout: Verifying : python3-requests-2.25.1-8.el9.noarch 17/36 2024-03-25T16:37:00.078 INFO:teuthology.orchestra.run.smithi115.stdout: Verifying : lua-devel-5.4.4-4.el9.x86_64 18/36 2024-03-25T16:37:00.078 INFO:teuthology.orchestra.run.smithi115.stdout: Verifying : luarocks-3.9.2-1.el9.noarch 19/36 2024-03-25T16:37:00.079 INFO:teuthology.orchestra.run.smithi115.stdout: Verifying : python3-bcrypt-3.2.2-1.el9.x86_64 20/36 2024-03-25T16:37:00.079 INFO:teuthology.orchestra.run.smithi115.stdout: Verifying : python3-cheroot-8.6.0-4.el9.noarch 21/36 2024-03-25T16:37:00.079 INFO:teuthology.orchestra.run.smithi115.stdout: Verifying : python3-cherrypy-18.6.1-2.el9.noarch 22/36 2024-03-25T16:37:00.079 INFO:teuthology.orchestra.run.smithi115.stdout: Verifying : python3-jaraco-8.2.1-3.el9.noarch 23/36 2024-03-25T16:37:00.079 INFO:teuthology.orchestra.run.smithi115.stdout: Verifying : python3-jaraco-classes-3.2.1-5.el9.noarch 24/36 2024-03-25T16:37:00.079 INFO:teuthology.orchestra.run.smithi115.stdout: Verifying : python3-jaraco-collections-3.0.0-8.el9.noarch 25/36 2024-03-25T16:37:00.080 INFO:teuthology.orchestra.run.smithi115.stdout: Verifying : python3-jaraco-functools-3.5.0-2.el9.noarch 26/36 2024-03-25T16:37:00.080 INFO:teuthology.orchestra.run.smithi115.stdout: Verifying : python3-jaraco-text-3.2.0-6.el9.noarch 27/36 2024-03-25T16:37:00.080 INFO:teuthology.orchestra.run.smithi115.stdout: Verifying : python3-logutils-0.3.5-21.el9.noarch 28/36 2024-03-25T16:37:00.080 INFO:teuthology.orchestra.run.smithi115.stdout: Verifying : python3-more-itertools-8.12.0-2.el9.noarch 29/36 2024-03-25T16:37:00.080 INFO:teuthology.orchestra.run.smithi115.stdout: Verifying : python3-pecan-1.4.2-3.el9.noarch 30/36 2024-03-25T16:37:00.081 INFO:teuthology.orchestra.run.smithi115.stdout: Verifying : python3-portend-3.1.0-2.el9.noarch 31/36 2024-03-25T16:37:00.081 INFO:teuthology.orchestra.run.smithi115.stdout: Verifying : python3-pyOpenSSL-21.0.0-1.el9.noarch 32/36 2024-03-25T16:37:00.081 INFO:teuthology.orchestra.run.smithi115.stdout: Verifying : python3-tempora-5.0.0-2.el9.noarch 33/36 2024-03-25T16:37:00.081 INFO:teuthology.orchestra.run.smithi115.stdout: Verifying : python3-webob-1.8.7-6.el9.noarch 34/36 2024-03-25T16:37:00.081 INFO:teuthology.orchestra.run.smithi115.stdout: Verifying : python3-werkzeug-2.0.3-3.el9.1.noarch 35/36 2024-03-25T16:37:00.955 INFO:teuthology.orchestra.run.smithi115.stdout: Verifying : python3-zc-lockfile-2.0-10.el9.noarch 36/36 2024-03-25T16:37:00.955 INFO:teuthology.orchestra.run.smithi115.stdout: 2024-03-25T16:37:00.956 INFO:teuthology.orchestra.run.smithi115.stdout:Installed: 2024-03-25T16:37:00.956 INFO:teuthology.orchestra.run.smithi115.stdout: ceph-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T16:37:00.956 INFO:teuthology.orchestra.run.smithi115.stdout: ceph-mds-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T16:37:00.956 INFO:teuthology.orchestra.run.smithi115.stdout: ceph-mgr-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T16:37:00.956 INFO:teuthology.orchestra.run.smithi115.stdout: ceph-mgr-modules-core-2:19.0.0-1439.g5dcb190d.el9.noarch 2024-03-25T16:37:00.956 INFO:teuthology.orchestra.run.smithi115.stdout: ceph-mon-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T16:37:00.956 INFO:teuthology.orchestra.run.smithi115.stdout: ceph-osd-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T16:37:00.956 INFO:teuthology.orchestra.run.smithi115.stdout: lua-5.4.4-4.el9.x86_64 2024-03-25T16:37:00.956 INFO:teuthology.orchestra.run.smithi115.stdout: lua-devel-5.4.4-4.el9.x86_64 2024-03-25T16:37:00.956 INFO:teuthology.orchestra.run.smithi115.stdout: luarocks-3.9.2-1.el9.noarch 2024-03-25T16:37:00.956 INFO:teuthology.orchestra.run.smithi115.stdout: python3-bcrypt-3.2.2-1.el9.x86_64 2024-03-25T16:37:00.956 INFO:teuthology.orchestra.run.smithi115.stdout: python3-cffi-1.14.5-5.el9.x86_64 2024-03-25T16:37:00.956 INFO:teuthology.orchestra.run.smithi115.stdout: python3-cheroot-8.6.0-4.el9.noarch 2024-03-25T16:37:00.956 INFO:teuthology.orchestra.run.smithi115.stdout: python3-cherrypy-18.6.1-2.el9.noarch 2024-03-25T16:37:00.957 INFO:teuthology.orchestra.run.smithi115.stdout: python3-cryptography-36.0.1-4.el9.x86_64 2024-03-25T16:37:00.957 INFO:teuthology.orchestra.run.smithi115.stdout: python3-jaraco-8.2.1-3.el9.noarch 2024-03-25T16:37:00.957 INFO:teuthology.orchestra.run.smithi115.stdout: python3-jaraco-classes-3.2.1-5.el9.noarch 2024-03-25T16:37:00.957 INFO:teuthology.orchestra.run.smithi115.stdout: python3-jaraco-collections-3.0.0-8.el9.noarch 2024-03-25T16:37:00.957 INFO:teuthology.orchestra.run.smithi115.stdout: python3-jaraco-functools-3.5.0-2.el9.noarch 2024-03-25T16:37:00.957 INFO:teuthology.orchestra.run.smithi115.stdout: python3-jaraco-text-3.2.0-6.el9.noarch 2024-03-25T16:37:00.957 INFO:teuthology.orchestra.run.smithi115.stdout: python3-logutils-0.3.5-21.el9.noarch 2024-03-25T16:37:00.957 INFO:teuthology.orchestra.run.smithi115.stdout: python3-mako-1.1.4-6.el9.noarch 2024-03-25T16:37:00.957 INFO:teuthology.orchestra.run.smithi115.stdout: python3-markupsafe-1.1.1-12.el9.x86_64 2024-03-25T16:37:00.957 INFO:teuthology.orchestra.run.smithi115.stdout: python3-more-itertools-8.12.0-2.el9.noarch 2024-03-25T16:37:00.957 INFO:teuthology.orchestra.run.smithi115.stdout: python3-pecan-1.4.2-3.el9.noarch 2024-03-25T16:37:00.957 INFO:teuthology.orchestra.run.smithi115.stdout: python3-ply-3.11-14.el9.noarch 2024-03-25T16:37:00.957 INFO:teuthology.orchestra.run.smithi115.stdout: python3-portend-3.1.0-2.el9.noarch 2024-03-25T16:37:00.957 INFO:teuthology.orchestra.run.smithi115.stdout: python3-pyOpenSSL-21.0.0-1.el9.noarch 2024-03-25T16:37:00.957 INFO:teuthology.orchestra.run.smithi115.stdout: python3-pycparser-2.20-6.el9.noarch 2024-03-25T16:37:00.957 INFO:teuthology.orchestra.run.smithi115.stdout: python3-pytz-2021.1-5.el9.noarch 2024-03-25T16:37:00.958 INFO:teuthology.orchestra.run.smithi115.stdout: python3-requests-2.25.1-8.el9.noarch 2024-03-25T16:37:00.958 INFO:teuthology.orchestra.run.smithi115.stdout: python3-tempora-5.0.0-2.el9.noarch 2024-03-25T16:37:00.958 INFO:teuthology.orchestra.run.smithi115.stdout: python3-toml-0.10.2-6.el9.noarch 2024-03-25T16:37:00.958 INFO:teuthology.orchestra.run.smithi115.stdout: python3-urllib3-1.26.5-5.el9.noarch 2024-03-25T16:37:00.958 INFO:teuthology.orchestra.run.smithi115.stdout: python3-webob-1.8.7-6.el9.noarch 2024-03-25T16:37:00.958 INFO:teuthology.orchestra.run.smithi115.stdout: python3-werkzeug-2.0.3-3.el9.1.noarch 2024-03-25T16:37:00.958 INFO:teuthology.orchestra.run.smithi115.stdout: python3-zc-lockfile-2.0-10.el9.noarch 2024-03-25T16:37:00.958 INFO:teuthology.orchestra.run.smithi115.stdout: 2024-03-25T16:37:00.958 INFO:teuthology.orchestra.run.smithi115.stdout:Complete! 2024-03-25T16:37:01.172 INFO:teuthology.orchestra.run.smithi001.stdout: Installing : ceph-osd-2:19.0.0-1439.g5dcb190d.el9.x86_64 33/36 2024-03-25T16:37:01.198 INFO:teuthology.orchestra.run.smithi001.stdout: Running scriptlet: ceph-osd-2:19.0.0-1439.g5dcb190d.el9.x86_64 33/36 2024-03-25T16:37:01.198 INFO:teuthology.orchestra.run.smithi001.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-25T16:37:01.199 INFO:teuthology.orchestra.run.smithi001.stdout:Invalid unit name "ceph-osd@*.service" escaped as "ceph-osd@\x2a.service". 2024-03-25T16:37:01.199 INFO:teuthology.orchestra.run.smithi001.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-osd.target → /usr/lib/systemd/system/ceph-osd.target. 2024-03-25T16:37:01.199 INFO:teuthology.orchestra.run.smithi001.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-osd.target → /usr/lib/systemd/system/ceph-osd.target. 2024-03-25T16:37:01.199 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-25T16:37:01.279 DEBUG:teuthology.orchestra.run.smithi115:> sudo yum -y install ceph-base 2024-03-25T16:37:01.665 INFO:teuthology.orchestra.run.smithi001.stdout: Installing : ceph-mon-2:19.0.0-1439.g5dcb190d.el9.x86_64 34/36 2024-03-25T16:37:01.687 INFO:teuthology.orchestra.run.smithi001.stdout: Running scriptlet: ceph-mon-2:19.0.0-1439.g5dcb190d.el9.x86_64 34/36 2024-03-25T16:37:01.688 INFO:teuthology.orchestra.run.smithi001.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-25T16:37:01.688 INFO:teuthology.orchestra.run.smithi001.stdout:Invalid unit name "ceph-mon@*.service" escaped as "ceph-mon@\x2a.service". 2024-03-25T16:37:01.688 INFO:teuthology.orchestra.run.smithi001.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mon.target → /usr/lib/systemd/system/ceph-mon.target. 2024-03-25T16:37:01.688 INFO:teuthology.orchestra.run.smithi001.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mon.target → /usr/lib/systemd/system/ceph-mon.target. 2024-03-25T16:37:01.688 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-25T16:37:01.840 INFO:teuthology.orchestra.run.smithi115.stdout:Last metadata expiration check: 0:04:16 ago on Mon 25 Mar 2024 04:32:45 PM UTC. 2024-03-25T16:37:01.907 INFO:teuthology.orchestra.run.smithi001.stdout: Installing : ceph-mds-2:19.0.0-1439.g5dcb190d.el9.x86_64 35/36 2024-03-25T16:37:01.930 INFO:teuthology.orchestra.run.smithi001.stdout: Running scriptlet: ceph-mds-2:19.0.0-1439.g5dcb190d.el9.x86_64 35/36 2024-03-25T16:37:01.931 INFO:teuthology.orchestra.run.smithi001.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-25T16:37:01.931 INFO:teuthology.orchestra.run.smithi001.stdout:Invalid unit name "ceph-mds@*.service" escaped as "ceph-mds@\x2a.service". 2024-03-25T16:37:01.931 INFO:teuthology.orchestra.run.smithi001.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mds.target → /usr/lib/systemd/system/ceph-mds.target. 2024-03-25T16:37:01.931 INFO:teuthology.orchestra.run.smithi001.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mds.target → /usr/lib/systemd/system/ceph-mds.target. 2024-03-25T16:37:01.931 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-25T16:37:01.936 INFO:teuthology.orchestra.run.smithi115.stdout:Package ceph-base-2:19.0.0-1439.g5dcb190d.el9.x86_64 is already installed. 2024-03-25T16:37:01.988 INFO:teuthology.orchestra.run.smithi115.stdout:Dependencies resolved. 2024-03-25T16:37:01.989 INFO:teuthology.orchestra.run.smithi115.stdout:Nothing to do. 2024-03-25T16:37:01.989 INFO:teuthology.orchestra.run.smithi115.stdout:Complete! 2024-03-25T16:37:01.998 INFO:teuthology.orchestra.run.smithi001.stdout: Installing : ceph-2:19.0.0-1439.g5dcb190d.el9.x86_64 36/36 2024-03-25T16:37:02.050 DEBUG:teuthology.orchestra.run.smithi115:> sudo yum -y install cephadm 2024-03-25T16:37:02.614 INFO:teuthology.orchestra.run.smithi115.stdout:Last metadata expiration check: 0:04:17 ago on Mon 25 Mar 2024 04:32:45 PM UTC. 2024-03-25T16:37:02.763 INFO:teuthology.orchestra.run.smithi115.stdout:Dependencies resolved. 2024-03-25T16:37:02.764 INFO:teuthology.orchestra.run.smithi115.stdout:================================================================================ 2024-03-25T16:37:02.764 INFO:teuthology.orchestra.run.smithi115.stdout: Package Arch Version Repository Size 2024-03-25T16:37:02.764 INFO:teuthology.orchestra.run.smithi115.stdout:================================================================================ 2024-03-25T16:37:02.764 INFO:teuthology.orchestra.run.smithi115.stdout:Installing: 2024-03-25T16:37:02.764 INFO:teuthology.orchestra.run.smithi115.stdout: cephadm noarch 2:19.0.0-1439.g5dcb190d.el9 ceph-noarch 747 k 2024-03-25T16:37:02.764 INFO:teuthology.orchestra.run.smithi115.stdout: 2024-03-25T16:37:02.764 INFO:teuthology.orchestra.run.smithi115.stdout:Transaction Summary 2024-03-25T16:37:02.764 INFO:teuthology.orchestra.run.smithi115.stdout:================================================================================ 2024-03-25T16:37:02.764 INFO:teuthology.orchestra.run.smithi115.stdout:Install 1 Package 2024-03-25T16:37:02.764 INFO:teuthology.orchestra.run.smithi115.stdout: 2024-03-25T16:37:02.764 INFO:teuthology.orchestra.run.smithi115.stdout:Total download size: 747 k 2024-03-25T16:37:02.764 INFO:teuthology.orchestra.run.smithi115.stdout:Installed size: 754 k 2024-03-25T16:37:02.764 INFO:teuthology.orchestra.run.smithi115.stdout:Downloading Packages: 2024-03-25T16:37:03.702 INFO:teuthology.orchestra.run.smithi001.stdout: Running scriptlet: ceph-2:19.0.0-1439.g5dcb190d.el9.x86_64 36/36 2024-03-25T16:37:03.703 INFO:teuthology.orchestra.run.smithi001.stdout: Verifying : ceph-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/36 2024-03-25T16:37:03.703 INFO:teuthology.orchestra.run.smithi001.stdout: Verifying : ceph-mds-2:19.0.0-1439.g5dcb190d.el9.x86_64 2/36 2024-03-25T16:37:03.703 INFO:teuthology.orchestra.run.smithi001.stdout: Verifying : ceph-mgr-2:19.0.0-1439.g5dcb190d.el9.x86_64 3/36 2024-03-25T16:37:03.703 INFO:teuthology.orchestra.run.smithi001.stdout: Verifying : ceph-mon-2:19.0.0-1439.g5dcb190d.el9.x86_64 4/36 2024-03-25T16:37:03.703 INFO:teuthology.orchestra.run.smithi001.stdout: Verifying : ceph-osd-2:19.0.0-1439.g5dcb190d.el9.x86_64 5/36 2024-03-25T16:37:03.703 INFO:teuthology.orchestra.run.smithi001.stdout: Verifying : ceph-mgr-modules-core-2:19.0.0-1439.g5dcb190d.el9. 6/36 2024-03-25T16:37:03.703 INFO:teuthology.orchestra.run.smithi001.stdout: Verifying : lua-5.4.4-4.el9.x86_64 7/36 2024-03-25T16:37:03.703 INFO:teuthology.orchestra.run.smithi001.stdout: Verifying : python3-mako-1.1.4-6.el9.noarch 8/36 2024-03-25T16:37:03.703 INFO:teuthology.orchestra.run.smithi001.stdout: Verifying : python3-markupsafe-1.1.1-12.el9.x86_64 9/36 2024-03-25T16:37:03.703 INFO:teuthology.orchestra.run.smithi001.stdout: Verifying : python3-pytz-2021.1-5.el9.noarch 10/36 2024-03-25T16:37:03.703 INFO:teuthology.orchestra.run.smithi001.stdout: Verifying : python3-toml-0.10.2-6.el9.noarch 11/36 2024-03-25T16:37:03.704 INFO:teuthology.orchestra.run.smithi001.stdout: Verifying : python3-cffi-1.14.5-5.el9.x86_64 12/36 2024-03-25T16:37:03.704 INFO:teuthology.orchestra.run.smithi001.stdout: Verifying : python3-cryptography-36.0.1-4.el9.x86_64 13/36 2024-03-25T16:37:03.704 INFO:teuthology.orchestra.run.smithi001.stdout: Verifying : python3-ply-3.11-14.el9.noarch 14/36 2024-03-25T16:37:03.704 INFO:teuthology.orchestra.run.smithi001.stdout: Verifying : python3-pycparser-2.20-6.el9.noarch 15/36 2024-03-25T16:37:03.704 INFO:teuthology.orchestra.run.smithi001.stdout: Verifying : python3-urllib3-1.26.5-5.el9.noarch 16/36 2024-03-25T16:37:03.704 INFO:teuthology.orchestra.run.smithi001.stdout: Verifying : python3-requests-2.25.1-8.el9.noarch 17/36 2024-03-25T16:37:03.704 INFO:teuthology.orchestra.run.smithi001.stdout: Verifying : lua-devel-5.4.4-4.el9.x86_64 18/36 2024-03-25T16:37:03.704 INFO:teuthology.orchestra.run.smithi001.stdout: Verifying : luarocks-3.9.2-1.el9.noarch 19/36 2024-03-25T16:37:03.704 INFO:teuthology.orchestra.run.smithi001.stdout: Verifying : python3-bcrypt-3.2.2-1.el9.x86_64 20/36 2024-03-25T16:37:03.704 INFO:teuthology.orchestra.run.smithi001.stdout: Verifying : python3-cheroot-8.6.0-4.el9.noarch 21/36 2024-03-25T16:37:03.704 INFO:teuthology.orchestra.run.smithi001.stdout: Verifying : python3-cherrypy-18.6.1-2.el9.noarch 22/36 2024-03-25T16:37:03.704 INFO:teuthology.orchestra.run.smithi001.stdout: Verifying : python3-jaraco-8.2.1-3.el9.noarch 23/36 2024-03-25T16:37:03.704 INFO:teuthology.orchestra.run.smithi001.stdout: Verifying : python3-jaraco-classes-3.2.1-5.el9.noarch 24/36 2024-03-25T16:37:03.705 INFO:teuthology.orchestra.run.smithi001.stdout: Verifying : python3-jaraco-collections-3.0.0-8.el9.noarch 25/36 2024-03-25T16:37:03.706 INFO:teuthology.orchestra.run.smithi001.stdout: Verifying : python3-jaraco-functools-3.5.0-2.el9.noarch 26/36 2024-03-25T16:37:03.706 INFO:teuthology.orchestra.run.smithi001.stdout: Verifying : python3-jaraco-text-3.2.0-6.el9.noarch 27/36 2024-03-25T16:37:03.706 INFO:teuthology.orchestra.run.smithi001.stdout: Verifying : python3-logutils-0.3.5-21.el9.noarch 28/36 2024-03-25T16:37:03.706 INFO:teuthology.orchestra.run.smithi001.stdout: Verifying : python3-more-itertools-8.12.0-2.el9.noarch 29/36 2024-03-25T16:37:03.706 INFO:teuthology.orchestra.run.smithi001.stdout: Verifying : python3-pecan-1.4.2-3.el9.noarch 30/36 2024-03-25T16:37:03.706 INFO:teuthology.orchestra.run.smithi001.stdout: Verifying : python3-portend-3.1.0-2.el9.noarch 31/36 2024-03-25T16:37:03.706 INFO:teuthology.orchestra.run.smithi001.stdout: Verifying : python3-pyOpenSSL-21.0.0-1.el9.noarch 32/36 2024-03-25T16:37:03.707 INFO:teuthology.orchestra.run.smithi001.stdout: Verifying : python3-tempora-5.0.0-2.el9.noarch 33/36 2024-03-25T16:37:03.707 INFO:teuthology.orchestra.run.smithi001.stdout: Verifying : python3-webob-1.8.7-6.el9.noarch 34/36 2024-03-25T16:37:03.707 INFO:teuthology.orchestra.run.smithi001.stdout: Verifying : python3-werkzeug-2.0.3-3.el9.1.noarch 35/36 2024-03-25T16:37:04.844 INFO:teuthology.orchestra.run.smithi001.stdout: Verifying : python3-zc-lockfile-2.0-10.el9.noarch 36/36 2024-03-25T16:37:04.845 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-25T16:37:04.845 INFO:teuthology.orchestra.run.smithi001.stdout:Installed: 2024-03-25T16:37:04.845 INFO:teuthology.orchestra.run.smithi001.stdout: ceph-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T16:37:04.845 INFO:teuthology.orchestra.run.smithi001.stdout: ceph-mds-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T16:37:04.845 INFO:teuthology.orchestra.run.smithi001.stdout: ceph-mgr-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T16:37:04.845 INFO:teuthology.orchestra.run.smithi001.stdout: ceph-mgr-modules-core-2:19.0.0-1439.g5dcb190d.el9.noarch 2024-03-25T16:37:04.845 INFO:teuthology.orchestra.run.smithi001.stdout: ceph-mon-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T16:37:04.845 INFO:teuthology.orchestra.run.smithi001.stdout: ceph-osd-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T16:37:04.845 INFO:teuthology.orchestra.run.smithi001.stdout: lua-5.4.4-4.el9.x86_64 2024-03-25T16:37:04.845 INFO:teuthology.orchestra.run.smithi001.stdout: lua-devel-5.4.4-4.el9.x86_64 2024-03-25T16:37:04.845 INFO:teuthology.orchestra.run.smithi001.stdout: luarocks-3.9.2-1.el9.noarch 2024-03-25T16:37:04.845 INFO:teuthology.orchestra.run.smithi001.stdout: python3-bcrypt-3.2.2-1.el9.x86_64 2024-03-25T16:37:04.845 INFO:teuthology.orchestra.run.smithi001.stdout: python3-cffi-1.14.5-5.el9.x86_64 2024-03-25T16:37:04.845 INFO:teuthology.orchestra.run.smithi001.stdout: python3-cheroot-8.6.0-4.el9.noarch 2024-03-25T16:37:04.846 INFO:teuthology.orchestra.run.smithi001.stdout: python3-cherrypy-18.6.1-2.el9.noarch 2024-03-25T16:37:04.846 INFO:teuthology.orchestra.run.smithi001.stdout: python3-cryptography-36.0.1-4.el9.x86_64 2024-03-25T16:37:04.846 INFO:teuthology.orchestra.run.smithi001.stdout: python3-jaraco-8.2.1-3.el9.noarch 2024-03-25T16:37:04.846 INFO:teuthology.orchestra.run.smithi001.stdout: python3-jaraco-classes-3.2.1-5.el9.noarch 2024-03-25T16:37:04.846 INFO:teuthology.orchestra.run.smithi001.stdout: python3-jaraco-collections-3.0.0-8.el9.noarch 2024-03-25T16:37:04.846 INFO:teuthology.orchestra.run.smithi001.stdout: python3-jaraco-functools-3.5.0-2.el9.noarch 2024-03-25T16:37:04.846 INFO:teuthology.orchestra.run.smithi001.stdout: python3-jaraco-text-3.2.0-6.el9.noarch 2024-03-25T16:37:04.846 INFO:teuthology.orchestra.run.smithi001.stdout: python3-logutils-0.3.5-21.el9.noarch 2024-03-25T16:37:04.846 INFO:teuthology.orchestra.run.smithi001.stdout: python3-mako-1.1.4-6.el9.noarch 2024-03-25T16:37:04.846 INFO:teuthology.orchestra.run.smithi001.stdout: python3-markupsafe-1.1.1-12.el9.x86_64 2024-03-25T16:37:04.846 INFO:teuthology.orchestra.run.smithi001.stdout: python3-more-itertools-8.12.0-2.el9.noarch 2024-03-25T16:37:04.846 INFO:teuthology.orchestra.run.smithi001.stdout: python3-pecan-1.4.2-3.el9.noarch 2024-03-25T16:37:04.847 INFO:teuthology.orchestra.run.smithi001.stdout: python3-ply-3.11-14.el9.noarch 2024-03-25T16:37:04.847 INFO:teuthology.orchestra.run.smithi001.stdout: python3-portend-3.1.0-2.el9.noarch 2024-03-25T16:37:04.847 INFO:teuthology.orchestra.run.smithi001.stdout: python3-pyOpenSSL-21.0.0-1.el9.noarch 2024-03-25T16:37:04.847 INFO:teuthology.orchestra.run.smithi001.stdout: python3-pycparser-2.20-6.el9.noarch 2024-03-25T16:37:04.847 INFO:teuthology.orchestra.run.smithi001.stdout: python3-pytz-2021.1-5.el9.noarch 2024-03-25T16:37:04.847 INFO:teuthology.orchestra.run.smithi001.stdout: python3-requests-2.25.1-8.el9.noarch 2024-03-25T16:37:04.847 INFO:teuthology.orchestra.run.smithi001.stdout: python3-tempora-5.0.0-2.el9.noarch 2024-03-25T16:37:04.847 INFO:teuthology.orchestra.run.smithi001.stdout: python3-toml-0.10.2-6.el9.noarch 2024-03-25T16:37:04.847 INFO:teuthology.orchestra.run.smithi001.stdout: python3-urllib3-1.26.5-5.el9.noarch 2024-03-25T16:37:04.847 INFO:teuthology.orchestra.run.smithi001.stdout: python3-webob-1.8.7-6.el9.noarch 2024-03-25T16:37:04.847 INFO:teuthology.orchestra.run.smithi001.stdout: python3-werkzeug-2.0.3-3.el9.1.noarch 2024-03-25T16:37:04.847 INFO:teuthology.orchestra.run.smithi001.stdout: python3-zc-lockfile-2.0-10.el9.noarch 2024-03-25T16:37:04.847 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-25T16:37:04.847 INFO:teuthology.orchestra.run.smithi001.stdout:Complete! 2024-03-25T16:37:05.221 DEBUG:teuthology.orchestra.run.smithi001:> sudo yum -y install ceph-base 2024-03-25T16:37:05.794 INFO:teuthology.orchestra.run.smithi001.stdout:Last metadata expiration check: 0:04:16 ago on Mon 25 Mar 2024 04:32:49 PM UTC. 2024-03-25T16:37:05.893 INFO:teuthology.orchestra.run.smithi001.stdout:Package ceph-base-2:19.0.0-1439.g5dcb190d.el9.x86_64 is already installed. 2024-03-25T16:37:05.947 INFO:teuthology.orchestra.run.smithi001.stdout:Dependencies resolved. 2024-03-25T16:37:05.948 INFO:teuthology.orchestra.run.smithi001.stdout:Nothing to do. 2024-03-25T16:37:05.949 INFO:teuthology.orchestra.run.smithi001.stdout:Complete! 2024-03-25T16:37:06.019 DEBUG:teuthology.orchestra.run.smithi001:> sudo yum -y install cephadm 2024-03-25T16:37:06.592 INFO:teuthology.orchestra.run.smithi001.stdout:Last metadata expiration check: 0:04:17 ago on Mon 25 Mar 2024 04:32:49 PM UTC. 2024-03-25T16:37:06.746 INFO:teuthology.orchestra.run.smithi001.stdout:Dependencies resolved. 2024-03-25T16:37:06.747 INFO:teuthology.orchestra.run.smithi001.stdout:================================================================================ 2024-03-25T16:37:06.747 INFO:teuthology.orchestra.run.smithi001.stdout: Package Arch Version Repository Size 2024-03-25T16:37:06.747 INFO:teuthology.orchestra.run.smithi001.stdout:================================================================================ 2024-03-25T16:37:06.747 INFO:teuthology.orchestra.run.smithi001.stdout:Installing: 2024-03-25T16:37:06.748 INFO:teuthology.orchestra.run.smithi001.stdout: cephadm noarch 2:19.0.0-1439.g5dcb190d.el9 ceph-noarch 747 k 2024-03-25T16:37:06.748 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-25T16:37:06.748 INFO:teuthology.orchestra.run.smithi001.stdout:Transaction Summary 2024-03-25T16:37:06.748 INFO:teuthology.orchestra.run.smithi001.stdout:================================================================================ 2024-03-25T16:37:06.748 INFO:teuthology.orchestra.run.smithi001.stdout:Install 1 Package 2024-03-25T16:37:06.748 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-25T16:37:06.748 INFO:teuthology.orchestra.run.smithi001.stdout:Total download size: 747 k 2024-03-25T16:37:06.748 INFO:teuthology.orchestra.run.smithi001.stdout:Installed size: 754 k 2024-03-25T16:37:06.748 INFO:teuthology.orchestra.run.smithi001.stdout:Downloading Packages: 2024-03-25T16:37:06.987 INFO:teuthology.orchestra.run.smithi115.stdout:cephadm-19.0.0-1439.g5dcb190d.el9.noarch.rpm 177 kB/s | 747 kB 00:04 2024-03-25T16:37:06.988 INFO:teuthology.orchestra.run.smithi115.stdout:-------------------------------------------------------------------------------- 2024-03-25T16:37:06.989 INFO:teuthology.orchestra.run.smithi115.stdout:Total 177 kB/s | 747 kB 00:04 2024-03-25T16:37:06.989 INFO:teuthology.orchestra.run.smithi115.stdout:Running transaction check 2024-03-25T16:37:06.997 INFO:teuthology.orchestra.run.smithi115.stdout:Transaction check succeeded. 2024-03-25T16:37:06.997 INFO:teuthology.orchestra.run.smithi115.stdout:Running transaction test 2024-03-25T16:37:07.001 INFO:teuthology.orchestra.run.smithi115.stdout:Transaction test succeeded. 2024-03-25T16:37:07.001 INFO:teuthology.orchestra.run.smithi115.stdout:Running transaction 2024-03-25T16:37:07.079 INFO:teuthology.orchestra.run.smithi115.stdout: Preparing : 1/1 2024-03-25T16:37:07.580 INFO:teuthology.orchestra.run.smithi115.stdout: Running scriptlet: cephadm-2:19.0.0-1439.g5dcb190d.el9.noarch 1/1 2024-03-25T16:37:07.693 INFO:teuthology.orchestra.run.smithi115.stdout: Installing : cephadm-2:19.0.0-1439.g5dcb190d.el9.noarch 1/1 2024-03-25T16:37:08.092 INFO:teuthology.orchestra.run.smithi115.stdout: Running scriptlet: cephadm-2:19.0.0-1439.g5dcb190d.el9.noarch 1/1 2024-03-25T16:37:08.355 INFO:teuthology.orchestra.run.smithi115.stdout: Verifying : cephadm-2:19.0.0-1439.g5dcb190d.el9.noarch 1/1 2024-03-25T16:37:08.355 INFO:teuthology.orchestra.run.smithi115.stdout: 2024-03-25T16:37:08.355 INFO:teuthology.orchestra.run.smithi115.stdout:Installed: 2024-03-25T16:37:08.355 INFO:teuthology.orchestra.run.smithi115.stdout: cephadm-2:19.0.0-1439.g5dcb190d.el9.noarch 2024-03-25T16:37:08.355 INFO:teuthology.orchestra.run.smithi115.stdout: 2024-03-25T16:37:08.355 INFO:teuthology.orchestra.run.smithi115.stdout:Complete! 2024-03-25T16:37:08.544 DEBUG:teuthology.orchestra.run.smithi115:> sudo yum -y install ceph-immutable-object-cache 2024-03-25T16:37:09.119 INFO:teuthology.orchestra.run.smithi115.stdout:Last metadata expiration check: 0:04:24 ago on Mon 25 Mar 2024 04:32:45 PM UTC. 2024-03-25T16:37:09.274 INFO:teuthology.orchestra.run.smithi115.stdout:Dependencies resolved. 2024-03-25T16:37:09.275 INFO:teuthology.orchestra.run.smithi115.stdout:================================================================================ 2024-03-25T16:37:09.275 INFO:teuthology.orchestra.run.smithi115.stdout: Package Arch Version Repo Size 2024-03-25T16:37:09.275 INFO:teuthology.orchestra.run.smithi115.stdout:================================================================================ 2024-03-25T16:37:09.275 INFO:teuthology.orchestra.run.smithi115.stdout:Installing: 2024-03-25T16:37:09.275 INFO:teuthology.orchestra.run.smithi115.stdout: ceph-immutable-object-cache x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 145 k 2024-03-25T16:37:09.275 INFO:teuthology.orchestra.run.smithi115.stdout: 2024-03-25T16:37:09.275 INFO:teuthology.orchestra.run.smithi115.stdout:Transaction Summary 2024-03-25T16:37:09.275 INFO:teuthology.orchestra.run.smithi115.stdout:================================================================================ 2024-03-25T16:37:09.275 INFO:teuthology.orchestra.run.smithi115.stdout:Install 1 Package 2024-03-25T16:37:09.275 INFO:teuthology.orchestra.run.smithi115.stdout: 2024-03-25T16:37:09.275 INFO:teuthology.orchestra.run.smithi115.stdout:Total download size: 145 k 2024-03-25T16:37:09.275 INFO:teuthology.orchestra.run.smithi115.stdout:Installed size: 436 k 2024-03-25T16:37:09.275 INFO:teuthology.orchestra.run.smithi115.stdout:Downloading Packages: 2024-03-25T16:37:10.369 INFO:teuthology.orchestra.run.smithi115.stdout:ceph-immutable-object-cache-19.0.0-1439.g5dcb19 133 kB/s | 145 kB 00:01 2024-03-25T16:37:10.369 INFO:teuthology.orchestra.run.smithi115.stdout:-------------------------------------------------------------------------------- 2024-03-25T16:37:10.370 INFO:teuthology.orchestra.run.smithi115.stdout:Total 133 kB/s | 145 kB 00:01 2024-03-25T16:37:10.370 INFO:teuthology.orchestra.run.smithi115.stdout:Running transaction check 2024-03-25T16:37:10.379 INFO:teuthology.orchestra.run.smithi115.stdout:Transaction check succeeded. 2024-03-25T16:37:10.379 INFO:teuthology.orchestra.run.smithi115.stdout:Running transaction test 2024-03-25T16:37:10.436 INFO:teuthology.orchestra.run.smithi115.stdout:Transaction test succeeded. 2024-03-25T16:37:10.436 INFO:teuthology.orchestra.run.smithi115.stdout:Running transaction 2024-03-25T16:37:10.584 INFO:teuthology.orchestra.run.smithi115.stdout: Preparing : 1/1 2024-03-25T16:37:10.645 INFO:teuthology.orchestra.run.smithi115.stdout: Installing : ceph-immutable-object-cache-2:19.0.0-1439.g5dcb190d. 1/1 2024-03-25T16:37:10.669 INFO:teuthology.orchestra.run.smithi115.stdout: Running scriptlet: ceph-immutable-object-cache-2:19.0.0-1439.g5dcb190d. 1/1 2024-03-25T16:37:10.669 INFO:teuthology.orchestra.run.smithi115.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-25T16:37:10.669 INFO:teuthology.orchestra.run.smithi115.stdout:Invalid unit name "ceph-immutable-object-cache@*.service" escaped as "ceph-immutable-object-cache@\x2a.service". 2024-03-25T16:37:10.669 INFO:teuthology.orchestra.run.smithi115.stdout: 2024-03-25T16:37:11.080 INFO:teuthology.orchestra.run.smithi001.stdout:cephadm-19.0.0-1439.g5dcb190d.el9.noarch.rpm 173 kB/s | 747 kB 00:04 2024-03-25T16:37:11.081 INFO:teuthology.orchestra.run.smithi001.stdout:-------------------------------------------------------------------------------- 2024-03-25T16:37:11.081 INFO:teuthology.orchestra.run.smithi001.stdout:Total 172 kB/s | 747 kB 00:04 2024-03-25T16:37:11.082 INFO:teuthology.orchestra.run.smithi001.stdout:Running transaction check 2024-03-25T16:37:11.088 INFO:teuthology.orchestra.run.smithi001.stdout:Transaction check succeeded. 2024-03-25T16:37:11.088 INFO:teuthology.orchestra.run.smithi001.stdout:Running transaction test 2024-03-25T16:37:11.094 INFO:teuthology.orchestra.run.smithi001.stdout:Transaction test succeeded. 2024-03-25T16:37:11.095 INFO:teuthology.orchestra.run.smithi001.stdout:Running transaction 2024-03-25T16:37:11.175 INFO:teuthology.orchestra.run.smithi001.stdout: Preparing : 1/1 2024-03-25T16:37:11.559 INFO:teuthology.orchestra.run.smithi001.stdout: Running scriptlet: cephadm-2:19.0.0-1439.g5dcb190d.el9.noarch 1/1 2024-03-25T16:37:11.608 INFO:teuthology.orchestra.run.smithi115.stdout: Verifying : ceph-immutable-object-cache-2:19.0.0-1439.g5dcb190d. 1/1 2024-03-25T16:37:11.609 INFO:teuthology.orchestra.run.smithi115.stdout: 2024-03-25T16:37:11.609 INFO:teuthology.orchestra.run.smithi115.stdout:Installed: 2024-03-25T16:37:11.609 INFO:teuthology.orchestra.run.smithi115.stdout: ceph-immutable-object-cache-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T16:37:11.609 INFO:teuthology.orchestra.run.smithi115.stdout: 2024-03-25T16:37:11.609 INFO:teuthology.orchestra.run.smithi115.stdout:Complete! 2024-03-25T16:37:11.664 INFO:teuthology.orchestra.run.smithi001.stdout: Installing : cephadm-2:19.0.0-1439.g5dcb190d.el9.noarch 1/1 2024-03-25T16:37:11.872 DEBUG:teuthology.orchestra.run.smithi115:> sudo yum -y install ceph-mgr 2024-03-25T16:37:12.022 INFO:teuthology.orchestra.run.smithi001.stdout: Running scriptlet: cephadm-2:19.0.0-1439.g5dcb190d.el9.noarch 1/1 2024-03-25T16:37:12.309 INFO:teuthology.orchestra.run.smithi001.stdout: Verifying : cephadm-2:19.0.0-1439.g5dcb190d.el9.noarch 1/1 2024-03-25T16:37:12.309 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-25T16:37:12.309 INFO:teuthology.orchestra.run.smithi001.stdout:Installed: 2024-03-25T16:37:12.309 INFO:teuthology.orchestra.run.smithi001.stdout: cephadm-2:19.0.0-1439.g5dcb190d.el9.noarch 2024-03-25T16:37:12.309 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-25T16:37:12.309 INFO:teuthology.orchestra.run.smithi001.stdout:Complete! 2024-03-25T16:37:12.447 INFO:teuthology.orchestra.run.smithi115.stdout:Last metadata expiration check: 0:04:27 ago on Mon 25 Mar 2024 04:32:45 PM UTC. 2024-03-25T16:37:12.543 INFO:teuthology.orchestra.run.smithi115.stdout:Package ceph-mgr-2:19.0.0-1439.g5dcb190d.el9.x86_64 is already installed. 2024-03-25T16:37:12.572 DEBUG:teuthology.orchestra.run.smithi001:> sudo yum -y install ceph-immutable-object-cache 2024-03-25T16:37:12.595 INFO:teuthology.orchestra.run.smithi115.stdout:Dependencies resolved. 2024-03-25T16:37:12.596 INFO:teuthology.orchestra.run.smithi115.stdout:Nothing to do. 2024-03-25T16:37:12.596 INFO:teuthology.orchestra.run.smithi115.stdout:Complete! 2024-03-25T16:37:12.664 DEBUG:teuthology.orchestra.run.smithi115:> sudo yum -y install ceph-mgr-dashboard 2024-03-25T16:37:13.152 INFO:teuthology.orchestra.run.smithi001.stdout:Last metadata expiration check: 0:04:24 ago on Mon 25 Mar 2024 04:32:49 PM UTC. 2024-03-25T16:37:13.222 INFO:teuthology.orchestra.run.smithi115.stdout:Last metadata expiration check: 0:04:28 ago on Mon 25 Mar 2024 04:32:45 PM UTC. 2024-03-25T16:37:13.308 INFO:teuthology.orchestra.run.smithi001.stdout:Dependencies resolved. 2024-03-25T16:37:13.309 INFO:teuthology.orchestra.run.smithi001.stdout:================================================================================ 2024-03-25T16:37:13.309 INFO:teuthology.orchestra.run.smithi001.stdout: Package Arch Version Repo Size 2024-03-25T16:37:13.309 INFO:teuthology.orchestra.run.smithi001.stdout:================================================================================ 2024-03-25T16:37:13.309 INFO:teuthology.orchestra.run.smithi001.stdout:Installing: 2024-03-25T16:37:13.309 INFO:teuthology.orchestra.run.smithi001.stdout: ceph-immutable-object-cache x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 145 k 2024-03-25T16:37:13.309 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-25T16:37:13.309 INFO:teuthology.orchestra.run.smithi001.stdout:Transaction Summary 2024-03-25T16:37:13.309 INFO:teuthology.orchestra.run.smithi001.stdout:================================================================================ 2024-03-25T16:37:13.309 INFO:teuthology.orchestra.run.smithi001.stdout:Install 1 Package 2024-03-25T16:37:13.309 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-25T16:37:13.310 INFO:teuthology.orchestra.run.smithi001.stdout:Total download size: 145 k 2024-03-25T16:37:13.310 INFO:teuthology.orchestra.run.smithi001.stdout:Installed size: 436 k 2024-03-25T16:37:13.310 INFO:teuthology.orchestra.run.smithi001.stdout:Downloading Packages: 2024-03-25T16:37:13.372 INFO:teuthology.orchestra.run.smithi115.stdout:Dependencies resolved. 2024-03-25T16:37:13.373 INFO:teuthology.orchestra.run.smithi115.stdout:====================================================================================== 2024-03-25T16:37:13.374 INFO:teuthology.orchestra.run.smithi115.stdout: Package Arch Version Repository Size 2024-03-25T16:37:13.374 INFO:teuthology.orchestra.run.smithi115.stdout:====================================================================================== 2024-03-25T16:37:13.374 INFO:teuthology.orchestra.run.smithi115.stdout:Installing: 2024-03-25T16:37:13.374 INFO:teuthology.orchestra.run.smithi115.stdout: ceph-mgr-dashboard noarch 2:19.0.0-1439.g5dcb190d.el9 ceph-noarch 1.8 M 2024-03-25T16:37:13.374 INFO:teuthology.orchestra.run.smithi115.stdout:Installing dependencies: 2024-03-25T16:37:13.374 INFO:teuthology.orchestra.run.smithi115.stdout: abseil-cpp x86_64 20211102.0-3.el9 epel 548 k 2024-03-25T16:37:13.374 INFO:teuthology.orchestra.run.smithi115.stdout: ceph-grafana-dashboards noarch 2:19.0.0-1439.g5dcb190d.el9 ceph-noarch 31 k 2024-03-25T16:37:13.374 INFO:teuthology.orchestra.run.smithi115.stdout: ceph-prometheus-alerts noarch 2:19.0.0-1439.g5dcb190d.el9 ceph-noarch 15 k 2024-03-25T16:37:13.374 INFO:teuthology.orchestra.run.smithi115.stdout: grpc-data noarch 1.46.7-10.el9 epel 19 k 2024-03-25T16:37:13.375 INFO:teuthology.orchestra.run.smithi115.stdout: protobuf x86_64 3.14.0-13.el9 CentOS-AppStream 1.0 M 2024-03-25T16:37:13.375 INFO:teuthology.orchestra.run.smithi115.stdout: protobuf-compiler x86_64 3.14.0-13.el9 crb 863 k 2024-03-25T16:37:13.375 INFO:teuthology.orchestra.run.smithi115.stdout: python3-grpcio x86_64 1.46.7-10.el9 epel 2.0 M 2024-03-25T16:37:13.375 INFO:teuthology.orchestra.run.smithi115.stdout: python3-grpcio-tools x86_64 1.46.7-10.el9 epel 144 k 2024-03-25T16:37:13.375 INFO:teuthology.orchestra.run.smithi115.stdout: python3-protobuf noarch 3.14.0-13.el9 CentOS-AppStream 238 k 2024-03-25T16:37:13.375 INFO:teuthology.orchestra.run.smithi115.stdout: python3-repoze-lru noarch 0.7-16.el9 epel 31 k 2024-03-25T16:37:13.375 INFO:teuthology.orchestra.run.smithi115.stdout: python3-routes noarch 2.5.1-5.el9 epel 188 k 2024-03-25T16:37:13.375 INFO:teuthology.orchestra.run.smithi115.stdout: 2024-03-25T16:37:13.375 INFO:teuthology.orchestra.run.smithi115.stdout:Transaction Summary 2024-03-25T16:37:13.375 INFO:teuthology.orchestra.run.smithi115.stdout:====================================================================================== 2024-03-25T16:37:13.375 INFO:teuthology.orchestra.run.smithi115.stdout:Install 12 Packages 2024-03-25T16:37:13.375 INFO:teuthology.orchestra.run.smithi115.stdout: 2024-03-25T16:37:13.376 INFO:teuthology.orchestra.run.smithi115.stdout:Total download size: 6.9 M 2024-03-25T16:37:13.376 INFO:teuthology.orchestra.run.smithi115.stdout:Installed size: 26 M 2024-03-25T16:37:13.376 INFO:teuthology.orchestra.run.smithi115.stdout:Downloading Packages: 2024-03-25T16:37:14.064 INFO:teuthology.orchestra.run.smithi115.stdout:(1/12): ceph-prometheus-alerts-19.0.0-1439.g5dc 27 kB/s | 15 kB 00:00 2024-03-25T16:37:14.081 INFO:teuthology.orchestra.run.smithi115.stdout:(2/12): ceph-grafana-dashboards-19.0.0-1439.g5d 53 kB/s | 31 kB 00:00 2024-03-25T16:37:14.273 INFO:teuthology.orchestra.run.smithi115.stdout:(3/12): python3-protobuf-3.14.0-13.el9.noarch.r 1.2 MB/s | 238 kB 00:00 2024-03-25T16:37:14.340 INFO:teuthology.orchestra.run.smithi115.stdout:(4/12): protobuf-3.14.0-13.el9.x86_64.rpm 3.7 MB/s | 1.0 MB 00:00 2024-03-25T16:37:14.370 INFO:teuthology.orchestra.run.smithi001.stdout:ceph-immutable-object-cache-19.0.0-1439.g5dcb19 137 kB/s | 145 kB 00:01 2024-03-25T16:37:14.370 INFO:teuthology.orchestra.run.smithi001.stdout:-------------------------------------------------------------------------------- 2024-03-25T16:37:14.370 INFO:teuthology.orchestra.run.smithi001.stdout:Total 137 kB/s | 145 kB 00:01 2024-03-25T16:37:14.371 INFO:teuthology.orchestra.run.smithi001.stdout:Running transaction check 2024-03-25T16:37:14.380 INFO:teuthology.orchestra.run.smithi001.stdout:Transaction check succeeded. 2024-03-25T16:37:14.380 INFO:teuthology.orchestra.run.smithi001.stdout:Running transaction test 2024-03-25T16:37:14.439 INFO:teuthology.orchestra.run.smithi001.stdout:Transaction test succeeded. 2024-03-25T16:37:14.439 INFO:teuthology.orchestra.run.smithi001.stdout:Running transaction 2024-03-25T16:37:14.565 INFO:teuthology.orchestra.run.smithi115.stdout:(5/12): protobuf-compiler-3.14.0-13.el9.x86_64. 2.9 MB/s | 863 kB 00:00 2024-03-25T16:37:14.583 INFO:teuthology.orchestra.run.smithi001.stdout: Preparing : 1/1 2024-03-25T16:37:14.654 INFO:teuthology.orchestra.run.smithi001.stdout: Installing : ceph-immutable-object-cache-2:19.0.0-1439.g5dcb190d. 1/1 2024-03-25T16:37:14.679 INFO:teuthology.orchestra.run.smithi001.stdout: Running scriptlet: ceph-immutable-object-cache-2:19.0.0-1439.g5dcb190d. 1/1 2024-03-25T16:37:14.679 INFO:teuthology.orchestra.run.smithi001.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-25T16:37:14.679 INFO:teuthology.orchestra.run.smithi001.stdout:Invalid unit name "ceph-immutable-object-cache@*.service" escaped as "ceph-immutable-object-cache@\x2a.service". 2024-03-25T16:37:14.679 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-25T16:37:14.807 INFO:teuthology.orchestra.run.smithi115.stdout:(6/12): abseil-cpp-20211102.0-3.el9.x86_64.rpm 1.1 MB/s | 548 kB 00:00 2024-03-25T16:37:14.823 INFO:teuthology.orchestra.run.smithi115.stdout:(7/12): grpc-data-1.46.7-10.el9.noarch.rpm 75 kB/s | 19 kB 00:00 2024-03-25T16:37:15.007 INFO:teuthology.orchestra.run.smithi115.stdout:(8/12): python3-grpcio-1.46.7-10.el9.x86_64.rpm 10 MB/s | 2.0 MB 00:00 2024-03-25T16:37:15.033 INFO:teuthology.orchestra.run.smithi115.stdout:(9/12): python3-grpcio-tools-1.46.7-10.el9.x86_ 692 kB/s | 144 kB 00:00 2024-03-25T16:37:15.083 INFO:teuthology.orchestra.run.smithi115.stdout:(10/12): python3-repoze-lru-0.7-16.el9.noarch.r 409 kB/s | 31 kB 00:00 2024-03-25T16:37:15.158 INFO:teuthology.orchestra.run.smithi115.stdout:(11/12): python3-routes-2.5.1-5.el9.noarch.rpm 1.5 MB/s | 188 kB 00:00 2024-03-25T16:37:15.606 INFO:teuthology.orchestra.run.smithi001.stdout: Verifying : ceph-immutable-object-cache-2:19.0.0-1439.g5dcb190d. 1/1 2024-03-25T16:37:15.606 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-25T16:37:15.607 INFO:teuthology.orchestra.run.smithi001.stdout:Installed: 2024-03-25T16:37:15.607 INFO:teuthology.orchestra.run.smithi001.stdout: ceph-immutable-object-cache-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T16:37:15.607 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-25T16:37:15.607 INFO:teuthology.orchestra.run.smithi001.stdout:Complete! 2024-03-25T16:37:15.907 DEBUG:teuthology.orchestra.run.smithi001:> sudo yum -y install ceph-mgr 2024-03-25T16:37:16.483 INFO:teuthology.orchestra.run.smithi001.stdout:Last metadata expiration check: 0:04:27 ago on Mon 25 Mar 2024 04:32:49 PM UTC. 2024-03-25T16:37:16.583 INFO:teuthology.orchestra.run.smithi001.stdout:Package ceph-mgr-2:19.0.0-1439.g5dcb190d.el9.x86_64 is already installed. 2024-03-25T16:37:16.637 INFO:teuthology.orchestra.run.smithi001.stdout:Dependencies resolved. 2024-03-25T16:37:16.638 INFO:teuthology.orchestra.run.smithi001.stdout:Nothing to do. 2024-03-25T16:37:16.638 INFO:teuthology.orchestra.run.smithi001.stdout:Complete! 2024-03-25T16:37:16.720 DEBUG:teuthology.orchestra.run.smithi001:> sudo yum -y install ceph-mgr-dashboard 2024-03-25T16:37:17.291 INFO:teuthology.orchestra.run.smithi001.stdout:Last metadata expiration check: 0:04:28 ago on Mon 25 Mar 2024 04:32:49 PM UTC. 2024-03-25T16:37:17.447 INFO:teuthology.orchestra.run.smithi001.stdout:Dependencies resolved. 2024-03-25T16:37:17.448 INFO:teuthology.orchestra.run.smithi001.stdout:====================================================================================== 2024-03-25T16:37:17.448 INFO:teuthology.orchestra.run.smithi001.stdout: Package Arch Version Repository Size 2024-03-25T16:37:17.448 INFO:teuthology.orchestra.run.smithi001.stdout:====================================================================================== 2024-03-25T16:37:17.448 INFO:teuthology.orchestra.run.smithi001.stdout:Installing: 2024-03-25T16:37:17.448 INFO:teuthology.orchestra.run.smithi001.stdout: ceph-mgr-dashboard noarch 2:19.0.0-1439.g5dcb190d.el9 ceph-noarch 1.8 M 2024-03-25T16:37:17.449 INFO:teuthology.orchestra.run.smithi001.stdout:Installing dependencies: 2024-03-25T16:37:17.449 INFO:teuthology.orchestra.run.smithi001.stdout: abseil-cpp x86_64 20211102.0-3.el9 epel 548 k 2024-03-25T16:37:17.449 INFO:teuthology.orchestra.run.smithi001.stdout: ceph-grafana-dashboards noarch 2:19.0.0-1439.g5dcb190d.el9 ceph-noarch 31 k 2024-03-25T16:37:17.449 INFO:teuthology.orchestra.run.smithi001.stdout: ceph-prometheus-alerts noarch 2:19.0.0-1439.g5dcb190d.el9 ceph-noarch 15 k 2024-03-25T16:37:17.449 INFO:teuthology.orchestra.run.smithi001.stdout: grpc-data noarch 1.46.7-10.el9 epel 19 k 2024-03-25T16:37:17.449 INFO:teuthology.orchestra.run.smithi001.stdout: protobuf x86_64 3.14.0-13.el9 CentOS-AppStream 1.0 M 2024-03-25T16:37:17.449 INFO:teuthology.orchestra.run.smithi001.stdout: protobuf-compiler x86_64 3.14.0-13.el9 crb 863 k 2024-03-25T16:37:17.449 INFO:teuthology.orchestra.run.smithi001.stdout: python3-grpcio x86_64 1.46.7-10.el9 epel 2.0 M 2024-03-25T16:37:17.449 INFO:teuthology.orchestra.run.smithi001.stdout: python3-grpcio-tools x86_64 1.46.7-10.el9 epel 144 k 2024-03-25T16:37:17.449 INFO:teuthology.orchestra.run.smithi001.stdout: python3-protobuf noarch 3.14.0-13.el9 CentOS-AppStream 238 k 2024-03-25T16:37:17.449 INFO:teuthology.orchestra.run.smithi001.stdout: python3-repoze-lru noarch 0.7-16.el9 epel 31 k 2024-03-25T16:37:17.449 INFO:teuthology.orchestra.run.smithi001.stdout: python3-routes noarch 2.5.1-5.el9 epel 188 k 2024-03-25T16:37:17.449 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-25T16:37:17.450 INFO:teuthology.orchestra.run.smithi001.stdout:Transaction Summary 2024-03-25T16:37:17.450 INFO:teuthology.orchestra.run.smithi001.stdout:====================================================================================== 2024-03-25T16:37:17.450 INFO:teuthology.orchestra.run.smithi001.stdout:Install 12 Packages 2024-03-25T16:37:17.450 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-25T16:37:17.450 INFO:teuthology.orchestra.run.smithi001.stdout:Total download size: 6.9 M 2024-03-25T16:37:17.450 INFO:teuthology.orchestra.run.smithi001.stdout:Installed size: 26 M 2024-03-25T16:37:17.450 INFO:teuthology.orchestra.run.smithi001.stdout:Downloading Packages: 2024-03-25T16:37:18.415 INFO:teuthology.orchestra.run.smithi001.stdout:(1/12): ceph-prometheus-alerts-19.0.0-1439.g5dc 34 kB/s | 15 kB 00:00 2024-03-25T16:37:18.491 INFO:teuthology.orchestra.run.smithi001.stdout:(2/12): ceph-grafana-dashboards-19.0.0-1439.g5d 59 kB/s | 31 kB 00:00 2024-03-25T16:37:18.649 INFO:teuthology.orchestra.run.smithi001.stdout:(3/12): protobuf-3.14.0-13.el9.x86_64.rpm 4.3 MB/s | 1.0 MB 00:00 2024-03-25T16:37:18.675 INFO:teuthology.orchestra.run.smithi001.stdout:(4/12): python3-protobuf-3.14.0-13.el9.noarch.r 1.3 MB/s | 238 kB 00:00 2024-03-25T16:37:18.875 INFO:teuthology.orchestra.run.smithi001.stdout:(5/12): protobuf-compiler-3.14.0-13.el9.x86_64. 3.7 MB/s | 863 kB 00:00 2024-03-25T16:37:19.100 INFO:teuthology.orchestra.run.smithi001.stdout:(6/12): grpc-data-1.46.7-10.el9.noarch.rpm 86 kB/s | 19 kB 00:00 2024-03-25T16:37:19.200 INFO:teuthology.orchestra.run.smithi001.stdout:(7/12): abseil-cpp-20211102.0-3.el9.x86_64.rpm 1.0 MB/s | 548 kB 00:00 2024-03-25T16:37:19.321 INFO:teuthology.orchestra.run.smithi001.stdout:(8/12): python3-grpcio-tools-1.46.7-10.el9.x86_ 1.2 MB/s | 144 kB 00:00 2024-03-25T16:37:19.426 INFO:teuthology.orchestra.run.smithi001.stdout:(9/12): python3-repoze-lru-0.7-16.el9.noarch.rp 294 kB/s | 31 kB 00:00 2024-03-25T16:37:19.526 INFO:teuthology.orchestra.run.smithi001.stdout:(10/12): python3-routes-2.5.1-5.el9.noarch.rpm 1.8 MB/s | 188 kB 00:00 2024-03-25T16:37:19.643 INFO:teuthology.orchestra.run.smithi001.stdout:(11/12): python3-grpcio-1.46.7-10.el9.x86_64.rp 3.8 MB/s | 2.0 MB 00:00 2024-03-25T16:37:22.765 INFO:teuthology.orchestra.run.smithi115.stdout:(12/12): ceph-mgr-dashboard-19.0.0-1439.g5dcb19 197 kB/s | 1.8 MB 00:09 2024-03-25T16:37:22.771 INFO:teuthology.orchestra.run.smithi115.stdout:-------------------------------------------------------------------------------- 2024-03-25T16:37:22.772 INFO:teuthology.orchestra.run.smithi115.stdout:Total 747 kB/s | 6.9 MB 00:09 2024-03-25T16:37:22.791 INFO:teuthology.orchestra.run.smithi115.stdout:Running transaction check 2024-03-25T16:37:22.811 INFO:teuthology.orchestra.run.smithi115.stdout:Transaction check succeeded. 2024-03-25T16:37:22.811 INFO:teuthology.orchestra.run.smithi115.stdout:Running transaction test 2024-03-25T16:37:22.992 INFO:teuthology.orchestra.run.smithi115.stdout:Transaction test succeeded. 2024-03-25T16:37:22.993 INFO:teuthology.orchestra.run.smithi115.stdout:Running transaction 2024-03-25T16:37:23.519 INFO:teuthology.orchestra.run.smithi115.stdout: Preparing : 1/1 2024-03-25T16:37:23.607 INFO:teuthology.orchestra.run.smithi115.stdout: Installing : protobuf-3.14.0-13.el9.x86_64 1/12 2024-03-25T16:37:23.643 INFO:teuthology.orchestra.run.smithi115.stdout: Installing : protobuf-compiler-3.14.0-13.el9.x86_64 2/12 2024-03-25T16:37:23.689 INFO:teuthology.orchestra.run.smithi115.stdout: Installing : python3-repoze-lru-0.7-16.el9.noarch 3/12 2024-03-25T16:37:23.720 INFO:teuthology.orchestra.run.smithi115.stdout: Installing : python3-routes-2.5.1-5.el9.noarch 4/12 2024-03-25T16:37:23.823 INFO:teuthology.orchestra.run.smithi115.stdout: Installing : grpc-data-1.46.7-10.el9.noarch 5/12 2024-03-25T16:37:23.953 INFO:teuthology.orchestra.run.smithi115.stdout: Installing : abseil-cpp-20211102.0-3.el9.x86_64 6/12 2024-03-25T16:37:24.040 INFO:teuthology.orchestra.run.smithi115.stdout: Installing : python3-grpcio-1.46.7-10.el9.x86_64 7/12 2024-03-25T16:37:24.083 INFO:teuthology.orchestra.run.smithi115.stdout: Installing : python3-protobuf-3.14.0-13.el9.noarch 8/12 2024-03-25T16:37:24.111 INFO:teuthology.orchestra.run.smithi115.stdout: Installing : python3-grpcio-tools-1.46.7-10.el9.x86_64 9/12 2024-03-25T16:37:24.143 INFO:teuthology.orchestra.run.smithi115.stdout: Installing : ceph-prometheus-alerts-2:19.0.0-1439.g5dcb190d.el9 10/12 2024-03-25T16:37:24.386 INFO:teuthology.orchestra.run.smithi115.stdout: Installing : ceph-grafana-dashboards-2:19.0.0-1439.g5dcb190d.el 11/12 2024-03-25T16:37:24.418 INFO:teuthology.orchestra.run.smithi115.stdout: Installing : ceph-mgr-dashboard-2:19.0.0-1439.g5dcb190d.el9.noa 12/12 2024-03-25T16:37:25.175 INFO:teuthology.orchestra.run.smithi115.stdout: Running scriptlet: ceph-mgr-dashboard-2:19.0.0-1439.g5dcb190d.el9.noa 12/12 2024-03-25T16:37:25.175 INFO:teuthology.orchestra.run.smithi115.stdout: Verifying : ceph-grafana-dashboards-2:19.0.0-1439.g5dcb190d.el 1/12 2024-03-25T16:37:25.176 INFO:teuthology.orchestra.run.smithi115.stdout: Verifying : ceph-mgr-dashboard-2:19.0.0-1439.g5dcb190d.el9.noa 2/12 2024-03-25T16:37:25.176 INFO:teuthology.orchestra.run.smithi115.stdout: Verifying : ceph-prometheus-alerts-2:19.0.0-1439.g5dcb190d.el9 3/12 2024-03-25T16:37:25.176 INFO:teuthology.orchestra.run.smithi115.stdout: Verifying : protobuf-3.14.0-13.el9.x86_64 4/12 2024-03-25T16:37:25.176 INFO:teuthology.orchestra.run.smithi115.stdout: Verifying : python3-protobuf-3.14.0-13.el9.noarch 5/12 2024-03-25T16:37:25.176 INFO:teuthology.orchestra.run.smithi115.stdout: Verifying : protobuf-compiler-3.14.0-13.el9.x86_64 6/12 2024-03-25T16:37:25.176 INFO:teuthology.orchestra.run.smithi115.stdout: Verifying : abseil-cpp-20211102.0-3.el9.x86_64 7/12 2024-03-25T16:37:25.176 INFO:teuthology.orchestra.run.smithi115.stdout: Verifying : grpc-data-1.46.7-10.el9.noarch 8/12 2024-03-25T16:37:25.176 INFO:teuthology.orchestra.run.smithi115.stdout: Verifying : python3-grpcio-1.46.7-10.el9.x86_64 9/12 2024-03-25T16:37:25.176 INFO:teuthology.orchestra.run.smithi115.stdout: Verifying : python3-grpcio-tools-1.46.7-10.el9.x86_64 10/12 2024-03-25T16:37:25.176 INFO:teuthology.orchestra.run.smithi115.stdout: Verifying : python3-repoze-lru-0.7-16.el9.noarch 11/12 2024-03-25T16:37:25.517 INFO:teuthology.orchestra.run.smithi001.stdout:(12/12): ceph-mgr-dashboard-19.0.0-1439.g5dcb19 241 kB/s | 1.8 MB 00:07 2024-03-25T16:37:25.521 INFO:teuthology.orchestra.run.smithi001.stdout:-------------------------------------------------------------------------------- 2024-03-25T16:37:25.521 INFO:teuthology.orchestra.run.smithi001.stdout:Total 870 kB/s | 6.9 MB 00:08 2024-03-25T16:37:25.525 INFO:teuthology.orchestra.run.smithi115.stdout: Verifying : python3-routes-2.5.1-5.el9.noarch 12/12 2024-03-25T16:37:25.525 INFO:teuthology.orchestra.run.smithi115.stdout: 2024-03-25T16:37:25.525 INFO:teuthology.orchestra.run.smithi115.stdout:Installed: 2024-03-25T16:37:25.525 INFO:teuthology.orchestra.run.smithi115.stdout: abseil-cpp-20211102.0-3.el9.x86_64 2024-03-25T16:37:25.525 INFO:teuthology.orchestra.run.smithi115.stdout: ceph-grafana-dashboards-2:19.0.0-1439.g5dcb190d.el9.noarch 2024-03-25T16:37:25.525 INFO:teuthology.orchestra.run.smithi115.stdout: ceph-mgr-dashboard-2:19.0.0-1439.g5dcb190d.el9.noarch 2024-03-25T16:37:25.525 INFO:teuthology.orchestra.run.smithi115.stdout: ceph-prometheus-alerts-2:19.0.0-1439.g5dcb190d.el9.noarch 2024-03-25T16:37:25.525 INFO:teuthology.orchestra.run.smithi115.stdout: grpc-data-1.46.7-10.el9.noarch 2024-03-25T16:37:25.525 INFO:teuthology.orchestra.run.smithi115.stdout: protobuf-3.14.0-13.el9.x86_64 2024-03-25T16:37:25.525 INFO:teuthology.orchestra.run.smithi115.stdout: protobuf-compiler-3.14.0-13.el9.x86_64 2024-03-25T16:37:25.525 INFO:teuthology.orchestra.run.smithi115.stdout: python3-grpcio-1.46.7-10.el9.x86_64 2024-03-25T16:37:25.525 INFO:teuthology.orchestra.run.smithi115.stdout: python3-grpcio-tools-1.46.7-10.el9.x86_64 2024-03-25T16:37:25.526 INFO:teuthology.orchestra.run.smithi115.stdout: python3-protobuf-3.14.0-13.el9.noarch 2024-03-25T16:37:25.526 INFO:teuthology.orchestra.run.smithi115.stdout: python3-repoze-lru-0.7-16.el9.noarch 2024-03-25T16:37:25.526 INFO:teuthology.orchestra.run.smithi115.stdout: python3-routes-2.5.1-5.el9.noarch 2024-03-25T16:37:25.526 INFO:teuthology.orchestra.run.smithi115.stdout: 2024-03-25T16:37:25.526 INFO:teuthology.orchestra.run.smithi115.stdout:Complete! 2024-03-25T16:37:25.541 INFO:teuthology.orchestra.run.smithi001.stdout:Running transaction check 2024-03-25T16:37:25.562 INFO:teuthology.orchestra.run.smithi001.stdout:Transaction check succeeded. 2024-03-25T16:37:25.562 INFO:teuthology.orchestra.run.smithi001.stdout:Running transaction test 2024-03-25T16:37:25.752 INFO:teuthology.orchestra.run.smithi001.stdout:Transaction test succeeded. 2024-03-25T16:37:25.753 INFO:teuthology.orchestra.run.smithi001.stdout:Running transaction 2024-03-25T16:37:25.760 DEBUG:teuthology.orchestra.run.smithi115:> sudo yum -y install ceph-mgr-diskprediction-local 2024-03-25T16:37:26.266 INFO:teuthology.orchestra.run.smithi001.stdout: Preparing : 1/1 2024-03-25T16:37:26.323 INFO:teuthology.orchestra.run.smithi115.stdout:Last metadata expiration check: 0:04:41 ago on Mon 25 Mar 2024 04:32:45 PM UTC. 2024-03-25T16:37:26.352 INFO:teuthology.orchestra.run.smithi001.stdout: Installing : protobuf-3.14.0-13.el9.x86_64 1/12 2024-03-25T16:37:26.404 INFO:teuthology.orchestra.run.smithi001.stdout: Installing : protobuf-compiler-3.14.0-13.el9.x86_64 2/12 2024-03-25T16:37:26.451 INFO:teuthology.orchestra.run.smithi001.stdout: Installing : python3-repoze-lru-0.7-16.el9.noarch 3/12 2024-03-25T16:37:26.478 INFO:teuthology.orchestra.run.smithi115.stdout:Dependencies resolved. 2024-03-25T16:37:26.480 INFO:teuthology.orchestra.run.smithi115.stdout:============================================================================================ 2024-03-25T16:37:26.480 INFO:teuthology.orchestra.run.smithi115.stdout: Package Arch Version Repository Size 2024-03-25T16:37:26.480 INFO:teuthology.orchestra.run.smithi115.stdout:============================================================================================ 2024-03-25T16:37:26.480 INFO:teuthology.orchestra.run.smithi115.stdout:Installing: 2024-03-25T16:37:26.480 INFO:teuthology.orchestra.run.smithi115.stdout: ceph-mgr-diskprediction-local noarch 2:19.0.0-1439.g5dcb190d.el9 ceph-noarch 7.4 M 2024-03-25T16:37:26.480 INFO:teuthology.orchestra.run.smithi115.stdout:Installing dependencies: 2024-03-25T16:37:26.480 INFO:teuthology.orchestra.run.smithi115.stdout: flexiblas x86_64 3.0.4-8.el9 CentOS-AppStream 32 k 2024-03-25T16:37:26.480 INFO:teuthology.orchestra.run.smithi115.stdout: flexiblas-netlib x86_64 3.0.4-8.el9 CentOS-AppStream 3.0 M 2024-03-25T16:37:26.480 INFO:teuthology.orchestra.run.smithi115.stdout: flexiblas-openblas-openmp x86_64 3.0.4-8.el9 CentOS-AppStream 17 k 2024-03-25T16:37:26.480 INFO:teuthology.orchestra.run.smithi115.stdout: libgfortran x86_64 11.4.1-3.el9 baseos 803 k 2024-03-25T16:37:26.480 INFO:teuthology.orchestra.run.smithi115.stdout: libquadmath x86_64 11.4.1-3.el9 baseos 195 k 2024-03-25T16:37:26.481 INFO:teuthology.orchestra.run.smithi115.stdout: openblas x86_64 0.3.21-2.el9 CentOS-AppStream 34 k 2024-03-25T16:37:26.481 INFO:teuthology.orchestra.run.smithi115.stdout: openblas-openmp x86_64 0.3.21-2.el9 CentOS-AppStream 4.7 M 2024-03-25T16:37:26.481 INFO:teuthology.orchestra.run.smithi115.stdout: python3-devel x86_64 3.9.17-2.el9 appstream 245 k 2024-03-25T16:37:26.481 INFO:teuthology.orchestra.run.smithi115.stdout: python3-numpy x86_64 1:1.20.1-5.el9 CentOS-AppStream 4.8 M 2024-03-25T16:37:26.481 INFO:teuthology.orchestra.run.smithi115.stdout: python3-numpy-f2py x86_64 1:1.20.1-5.el9 CentOS-AppStream 217 k 2024-03-25T16:37:26.481 INFO:teuthology.orchestra.run.smithi115.stdout: python3-scipy x86_64 1.6.2-8.el9 CentOS-AppStream 16 M 2024-03-25T16:37:26.481 INFO:teuthology.orchestra.run.smithi115.stdout: 2024-03-25T16:37:26.481 INFO:teuthology.orchestra.run.smithi115.stdout:Transaction Summary 2024-03-25T16:37:26.481 INFO:teuthology.orchestra.run.smithi115.stdout:============================================================================================ 2024-03-25T16:37:26.481 INFO:teuthology.orchestra.run.smithi115.stdout:Install 12 Packages 2024-03-25T16:37:26.481 INFO:teuthology.orchestra.run.smithi115.stdout: 2024-03-25T16:37:26.482 INFO:teuthology.orchestra.run.smithi115.stdout:Total download size: 37 M 2024-03-25T16:37:26.482 INFO:teuthology.orchestra.run.smithi115.stdout:Installed size: 209 M 2024-03-25T16:37:26.482 INFO:teuthology.orchestra.run.smithi115.stdout:Downloading Packages: 2024-03-25T16:37:26.488 INFO:teuthology.orchestra.run.smithi001.stdout: Installing : python3-routes-2.5.1-5.el9.noarch 4/12 2024-03-25T16:37:26.598 INFO:teuthology.orchestra.run.smithi001.stdout: Installing : grpc-data-1.46.7-10.el9.noarch 5/12 2024-03-25T16:37:26.737 INFO:teuthology.orchestra.run.smithi001.stdout: Installing : abseil-cpp-20211102.0-3.el9.x86_64 6/12 2024-03-25T16:37:26.836 INFO:teuthology.orchestra.run.smithi001.stdout: Installing : python3-grpcio-1.46.7-10.el9.x86_64 7/12 2024-03-25T16:37:26.886 INFO:teuthology.orchestra.run.smithi001.stdout: Installing : python3-protobuf-3.14.0-13.el9.noarch 8/12 2024-03-25T16:37:26.913 INFO:teuthology.orchestra.run.smithi001.stdout: Installing : python3-grpcio-tools-1.46.7-10.el9.x86_64 9/12 2024-03-25T16:37:26.945 INFO:teuthology.orchestra.run.smithi001.stdout: Installing : ceph-prometheus-alerts-2:19.0.0-1439.g5dcb190d.el9 10/12 2024-03-25T16:37:27.030 INFO:teuthology.orchestra.run.smithi115.stdout:(1/12): flexiblas-3.0.4-8.el9.x86_64.rpm 233 kB/s | 32 kB 00:00 2024-03-25T16:37:27.081 INFO:teuthology.orchestra.run.smithi115.stdout:(2/12): flexiblas-openblas-openmp-3.0.4-8.el9.x 344 kB/s | 17 kB 00:00 2024-03-25T16:37:27.114 INFO:teuthology.orchestra.run.smithi115.stdout:(3/12): openblas-0.3.21-2.el9.x86_64.rpm 1.0 MB/s | 34 kB 00:00 2024-03-25T16:37:27.209 INFO:teuthology.orchestra.run.smithi001.stdout: Installing : ceph-grafana-dashboards-2:19.0.0-1439.g5dcb190d.el 11/12 2024-03-25T16:37:27.245 INFO:teuthology.orchestra.run.smithi001.stdout: Installing : ceph-mgr-dashboard-2:19.0.0-1439.g5dcb190d.el9.noa 12/12 2024-03-25T16:37:27.331 INFO:teuthology.orchestra.run.smithi115.stdout:(4/12): openblas-openmp-0.3.21-2.el9.x86_64.rpm 22 MB/s | 4.7 MB 00:00 2024-03-25T16:37:27.456 INFO:teuthology.orchestra.run.smithi115.stdout:(5/12): python3-numpy-1.20.1-5.el9.x86_64.rpm 38 MB/s | 4.8 MB 00:00 2024-03-25T16:37:27.498 INFO:teuthology.orchestra.run.smithi115.stdout:(6/12): python3-numpy-f2py-1.20.1-5.el9.x86_64. 5.2 MB/s | 217 kB 00:00 2024-03-25T16:37:27.557 INFO:teuthology.orchestra.run.smithi115.stdout:(7/12): flexiblas-netlib-3.0.4-8.el9.x86_64.rpm 4.5 MB/s | 3.0 MB 00:00 2024-03-25T16:37:27.924 INFO:teuthology.orchestra.run.smithi115.stdout:(8/12): python3-scipy-1.6.2-8.el9.x86_64.rpm 37 MB/s | 16 MB 00:00 2024-03-25T16:37:28.013 INFO:teuthology.orchestra.run.smithi001.stdout: Running scriptlet: ceph-mgr-dashboard-2:19.0.0-1439.g5dcb190d.el9.noa 12/12 2024-03-25T16:37:28.013 INFO:teuthology.orchestra.run.smithi001.stdout: Verifying : ceph-grafana-dashboards-2:19.0.0-1439.g5dcb190d.el 1/12 2024-03-25T16:37:28.013 INFO:teuthology.orchestra.run.smithi001.stdout: Verifying : ceph-mgr-dashboard-2:19.0.0-1439.g5dcb190d.el9.noa 2/12 2024-03-25T16:37:28.013 INFO:teuthology.orchestra.run.smithi001.stdout: Verifying : ceph-prometheus-alerts-2:19.0.0-1439.g5dcb190d.el9 3/12 2024-03-25T16:37:28.013 INFO:teuthology.orchestra.run.smithi001.stdout: Verifying : protobuf-3.14.0-13.el9.x86_64 4/12 2024-03-25T16:37:28.014 INFO:teuthology.orchestra.run.smithi001.stdout: Verifying : python3-protobuf-3.14.0-13.el9.noarch 5/12 2024-03-25T16:37:28.014 INFO:teuthology.orchestra.run.smithi001.stdout: Verifying : protobuf-compiler-3.14.0-13.el9.x86_64 6/12 2024-03-25T16:37:28.014 INFO:teuthology.orchestra.run.smithi001.stdout: Verifying : abseil-cpp-20211102.0-3.el9.x86_64 7/12 2024-03-25T16:37:28.014 INFO:teuthology.orchestra.run.smithi001.stdout: Verifying : grpc-data-1.46.7-10.el9.noarch 8/12 2024-03-25T16:37:28.014 INFO:teuthology.orchestra.run.smithi001.stdout: Verifying : python3-grpcio-1.46.7-10.el9.x86_64 9/12 2024-03-25T16:37:28.014 INFO:teuthology.orchestra.run.smithi001.stdout: Verifying : python3-grpcio-tools-1.46.7-10.el9.x86_64 10/12 2024-03-25T16:37:28.014 INFO:teuthology.orchestra.run.smithi001.stdout: Verifying : python3-repoze-lru-0.7-16.el9.noarch 11/12 2024-03-25T16:37:28.365 INFO:teuthology.orchestra.run.smithi001.stdout: Verifying : python3-routes-2.5.1-5.el9.noarch 12/12 2024-03-25T16:37:28.365 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-25T16:37:28.366 INFO:teuthology.orchestra.run.smithi001.stdout:Installed: 2024-03-25T16:37:28.366 INFO:teuthology.orchestra.run.smithi001.stdout: abseil-cpp-20211102.0-3.el9.x86_64 2024-03-25T16:37:28.366 INFO:teuthology.orchestra.run.smithi001.stdout: ceph-grafana-dashboards-2:19.0.0-1439.g5dcb190d.el9.noarch 2024-03-25T16:37:28.366 INFO:teuthology.orchestra.run.smithi001.stdout: ceph-mgr-dashboard-2:19.0.0-1439.g5dcb190d.el9.noarch 2024-03-25T16:37:28.366 INFO:teuthology.orchestra.run.smithi001.stdout: ceph-prometheus-alerts-2:19.0.0-1439.g5dcb190d.el9.noarch 2024-03-25T16:37:28.366 INFO:teuthology.orchestra.run.smithi001.stdout: grpc-data-1.46.7-10.el9.noarch 2024-03-25T16:37:28.366 INFO:teuthology.orchestra.run.smithi001.stdout: protobuf-3.14.0-13.el9.x86_64 2024-03-25T16:37:28.366 INFO:teuthology.orchestra.run.smithi001.stdout: protobuf-compiler-3.14.0-13.el9.x86_64 2024-03-25T16:37:28.366 INFO:teuthology.orchestra.run.smithi001.stdout: python3-grpcio-1.46.7-10.el9.x86_64 2024-03-25T16:37:28.366 INFO:teuthology.orchestra.run.smithi001.stdout: python3-grpcio-tools-1.46.7-10.el9.x86_64 2024-03-25T16:37:28.366 INFO:teuthology.orchestra.run.smithi001.stdout: python3-protobuf-3.14.0-13.el9.noarch 2024-03-25T16:37:28.366 INFO:teuthology.orchestra.run.smithi001.stdout: python3-repoze-lru-0.7-16.el9.noarch 2024-03-25T16:37:28.366 INFO:teuthology.orchestra.run.smithi001.stdout: python3-routes-2.5.1-5.el9.noarch 2024-03-25T16:37:28.366 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-25T16:37:28.367 INFO:teuthology.orchestra.run.smithi001.stdout:Complete! 2024-03-25T16:37:28.629 DEBUG:teuthology.orchestra.run.smithi001:> sudo yum -y install ceph-mgr-diskprediction-local 2024-03-25T16:37:28.799 INFO:teuthology.orchestra.run.smithi115.stdout:(9/12): libquadmath-11.4.1-3.el9.x86_64.rpm 229 kB/s | 195 kB 00:00 2024-03-25T16:37:29.204 INFO:teuthology.orchestra.run.smithi001.stdout:Last metadata expiration check: 0:04:40 ago on Mon 25 Mar 2024 04:32:49 PM UTC. 2024-03-25T16:37:29.358 INFO:teuthology.orchestra.run.smithi115.stdout:(10/12): libgfortran-11.4.1-3.el9.x86_64.rpm 446 kB/s | 803 kB 00:01 2024-03-25T16:37:29.362 INFO:teuthology.orchestra.run.smithi001.stdout:Dependencies resolved. 2024-03-25T16:37:29.363 INFO:teuthology.orchestra.run.smithi001.stdout:============================================================================================ 2024-03-25T16:37:29.363 INFO:teuthology.orchestra.run.smithi001.stdout: Package Arch Version Repository Size 2024-03-25T16:37:29.363 INFO:teuthology.orchestra.run.smithi001.stdout:============================================================================================ 2024-03-25T16:37:29.364 INFO:teuthology.orchestra.run.smithi001.stdout:Installing: 2024-03-25T16:37:29.364 INFO:teuthology.orchestra.run.smithi001.stdout: ceph-mgr-diskprediction-local noarch 2:19.0.0-1439.g5dcb190d.el9 ceph-noarch 7.4 M 2024-03-25T16:37:29.364 INFO:teuthology.orchestra.run.smithi001.stdout:Installing dependencies: 2024-03-25T16:37:29.364 INFO:teuthology.orchestra.run.smithi001.stdout: flexiblas x86_64 3.0.4-8.el9 CentOS-AppStream 32 k 2024-03-25T16:37:29.364 INFO:teuthology.orchestra.run.smithi001.stdout: flexiblas-netlib x86_64 3.0.4-8.el9 CentOS-AppStream 3.0 M 2024-03-25T16:37:29.364 INFO:teuthology.orchestra.run.smithi001.stdout: flexiblas-openblas-openmp x86_64 3.0.4-8.el9 CentOS-AppStream 17 k 2024-03-25T16:37:29.364 INFO:teuthology.orchestra.run.smithi001.stdout: libgfortran x86_64 11.4.1-3.el9 baseos 803 k 2024-03-25T16:37:29.364 INFO:teuthology.orchestra.run.smithi001.stdout: libquadmath x86_64 11.4.1-3.el9 baseos 195 k 2024-03-25T16:37:29.364 INFO:teuthology.orchestra.run.smithi001.stdout: openblas x86_64 0.3.21-2.el9 CentOS-AppStream 34 k 2024-03-25T16:37:29.364 INFO:teuthology.orchestra.run.smithi001.stdout: openblas-openmp x86_64 0.3.21-2.el9 CentOS-AppStream 4.7 M 2024-03-25T16:37:29.365 INFO:teuthology.orchestra.run.smithi001.stdout: python3-devel x86_64 3.9.17-2.el9 appstream 245 k 2024-03-25T16:37:29.365 INFO:teuthology.orchestra.run.smithi001.stdout: python3-numpy x86_64 1:1.20.1-5.el9 CentOS-AppStream 4.8 M 2024-03-25T16:37:29.365 INFO:teuthology.orchestra.run.smithi001.stdout: python3-numpy-f2py x86_64 1:1.20.1-5.el9 CentOS-AppStream 217 k 2024-03-25T16:37:29.365 INFO:teuthology.orchestra.run.smithi001.stdout: python3-scipy x86_64 1.6.2-8.el9 CentOS-AppStream 16 M 2024-03-25T16:37:29.365 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-25T16:37:29.365 INFO:teuthology.orchestra.run.smithi001.stdout:Transaction Summary 2024-03-25T16:37:29.365 INFO:teuthology.orchestra.run.smithi001.stdout:============================================================================================ 2024-03-25T16:37:29.365 INFO:teuthology.orchestra.run.smithi001.stdout:Install 12 Packages 2024-03-25T16:37:29.365 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-25T16:37:29.366 INFO:teuthology.orchestra.run.smithi001.stdout:Total download size: 37 M 2024-03-25T16:37:29.366 INFO:teuthology.orchestra.run.smithi001.stdout:Installed size: 209 M 2024-03-25T16:37:29.366 INFO:teuthology.orchestra.run.smithi001.stdout:Downloading Packages: 2024-03-25T16:37:29.657 INFO:teuthology.orchestra.run.smithi001.stdout:(1/12): flexiblas-3.0.4-8.el9.x86_64.rpm 189 kB/s | 32 kB 00:00 2024-03-25T16:37:29.707 INFO:teuthology.orchestra.run.smithi001.stdout:(2/12): flexiblas-openblas-openmp-3.0.4-8.el9.x 347 kB/s | 17 kB 00:00 2024-03-25T16:37:29.758 INFO:teuthology.orchestra.run.smithi001.stdout:(3/12): openblas-0.3.21-2.el9.x86_64.rpm 707 kB/s | 34 kB 00:00 2024-03-25T16:37:29.766 INFO:teuthology.orchestra.run.smithi115.stdout:(11/12): python3-devel-3.9.17-2.el9.x86_64.rpm 254 kB/s | 245 kB 00:00 2024-03-25T16:37:29.850 INFO:teuthology.orchestra.run.smithi001.stdout:(4/12): flexiblas-netlib-3.0.4-8.el9.x86_64.rpm 8.3 MB/s | 3.0 MB 00:00 2024-03-25T16:37:30.000 INFO:teuthology.orchestra.run.smithi001.stdout:(5/12): openblas-openmp-0.3.21-2.el9.x86_64.rpm 20 MB/s | 4.7 MB 00:00 2024-03-25T16:37:30.067 INFO:teuthology.orchestra.run.smithi001.stdout:(6/12): python3-numpy-1.20.1-5.el9.x86_64.rpm 22 MB/s | 4.8 MB 00:00 2024-03-25T16:37:30.092 INFO:teuthology.orchestra.run.smithi001.stdout:(7/12): python3-numpy-f2py-1.20.1-5.el9.x86_64. 2.4 MB/s | 217 kB 00:00 2024-03-25T16:37:30.409 INFO:teuthology.orchestra.run.smithi001.stdout:(8/12): python3-scipy-1.6.2-8.el9.x86_64.rpm 46 MB/s | 16 MB 00:00 2024-03-25T16:37:30.443 INFO:teuthology.orchestra.run.smithi001.stdout:(9/12): libgfortran-11.4.1-3.el9.x86_64.rpm 2.3 MB/s | 803 kB 00:00 2024-03-25T16:37:30.509 INFO:teuthology.orchestra.run.smithi001.stdout:(10/12): python3-devel-3.9.17-2.el9.x86_64.rpm 3.6 MB/s | 245 kB 00:00 2024-03-25T16:37:30.551 INFO:teuthology.orchestra.run.smithi001.stdout:(11/12): libquadmath-11.4.1-3.el9.x86_64.rpm 1.3 MB/s | 195 kB 00:00 2024-03-25T16:37:46.289 INFO:teuthology.orchestra.run.smithi115.stdout:(12/12): ceph-mgr-diskprediction-local-19.0.0-1 390 kB/s | 7.4 MB 00:19 2024-03-25T16:37:46.292 INFO:teuthology.orchestra.run.smithi115.stdout:-------------------------------------------------------------------------------- 2024-03-25T16:37:46.292 INFO:teuthology.orchestra.run.smithi115.stdout:Total 1.9 MB/s | 37 MB 00:19 2024-03-25T16:37:46.335 INFO:teuthology.orchestra.run.smithi115.stdout:Running transaction check 2024-03-25T16:37:46.360 INFO:teuthology.orchestra.run.smithi115.stdout:Transaction check succeeded. 2024-03-25T16:37:46.360 INFO:teuthology.orchestra.run.smithi115.stdout:Running transaction test 2024-03-25T16:37:46.716 INFO:teuthology.orchestra.run.smithi115.stdout:Transaction test succeeded. 2024-03-25T16:37:46.716 INFO:teuthology.orchestra.run.smithi115.stdout:Running transaction 2024-03-25T16:37:47.416 INFO:teuthology.orchestra.run.smithi115.stdout: Preparing : 1/1 2024-03-25T16:37:47.478 INFO:teuthology.orchestra.run.smithi115.stdout: Installing : libquadmath-11.4.1-3.el9.x86_64 1/12 2024-03-25T16:37:47.517 INFO:teuthology.orchestra.run.smithi115.stdout: Installing : libgfortran-11.4.1-3.el9.x86_64 2/12 2024-03-25T16:37:47.619 INFO:teuthology.orchestra.run.smithi115.stdout: Installing : flexiblas-3.0.4-8.el9.x86_64 3/12 2024-03-25T16:37:47.671 INFO:teuthology.orchestra.run.smithi115.stdout: Installing : python3-devel-3.9.17-2.el9.x86_64 4/12 2024-03-25T16:37:47.850 INFO:teuthology.orchestra.run.smithi115.stdout: Installing : openblas-0.3.21-2.el9.x86_64 5/12 2024-03-25T16:37:47.878 INFO:teuthology.orchestra.run.smithi115.stdout: Installing : openblas-openmp-0.3.21-2.el9.x86_64 6/12 2024-03-25T16:37:47.963 INFO:teuthology.orchestra.run.smithi115.stdout: Installing : flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 7/12 2024-03-25T16:37:48.619 INFO:teuthology.orchestra.run.smithi115.stdout: Installing : flexiblas-netlib-3.0.4-8.el9.x86_64 8/12 2024-03-25T16:37:48.740 INFO:teuthology.orchestra.run.smithi115.stdout: Installing : python3-numpy-1:1.20.1-5.el9.x86_64 9/12 2024-03-25T16:37:49.958 INFO:teuthology.orchestra.run.smithi115.stdout: Installing : python3-numpy-f2py-1:1.20.1-5.el9.x86_64 10/12 2024-03-25T16:37:51.280 INFO:teuthology.orchestra.run.smithi115.stdout: Installing : python3-scipy-1.6.2-8.el9.x86_64 11/12 2024-03-25T16:37:51.310 INFO:teuthology.orchestra.run.smithi115.stdout: Installing : ceph-mgr-diskprediction-local-2:19.0.0-1439.g5dcb1 12/12 2024-03-25T16:37:53.703 INFO:teuthology.orchestra.run.smithi115.stdout: Running scriptlet: ceph-mgr-diskprediction-local-2:19.0.0-1439.g5dcb1 12/12 2024-03-25T16:37:53.703 INFO:teuthology.orchestra.run.smithi115.stdout: Verifying : ceph-mgr-diskprediction-local-2:19.0.0-1439.g5dcb1 1/12 2024-03-25T16:37:53.703 INFO:teuthology.orchestra.run.smithi115.stdout: Verifying : flexiblas-3.0.4-8.el9.x86_64 2/12 2024-03-25T16:37:53.703 INFO:teuthology.orchestra.run.smithi115.stdout: Verifying : flexiblas-netlib-3.0.4-8.el9.x86_64 3/12 2024-03-25T16:37:53.704 INFO:teuthology.orchestra.run.smithi115.stdout: Verifying : flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 4/12 2024-03-25T16:37:53.704 INFO:teuthology.orchestra.run.smithi115.stdout: Verifying : openblas-0.3.21-2.el9.x86_64 5/12 2024-03-25T16:37:53.704 INFO:teuthology.orchestra.run.smithi115.stdout: Verifying : openblas-openmp-0.3.21-2.el9.x86_64 6/12 2024-03-25T16:37:53.704 INFO:teuthology.orchestra.run.smithi115.stdout: Verifying : python3-numpy-1:1.20.1-5.el9.x86_64 7/12 2024-03-25T16:37:53.704 INFO:teuthology.orchestra.run.smithi115.stdout: Verifying : python3-numpy-f2py-1:1.20.1-5.el9.x86_64 8/12 2024-03-25T16:37:53.704 INFO:teuthology.orchestra.run.smithi115.stdout: Verifying : python3-scipy-1.6.2-8.el9.x86_64 9/12 2024-03-25T16:37:53.704 INFO:teuthology.orchestra.run.smithi115.stdout: Verifying : libgfortran-11.4.1-3.el9.x86_64 10/12 2024-03-25T16:37:53.704 INFO:teuthology.orchestra.run.smithi115.stdout: Verifying : libquadmath-11.4.1-3.el9.x86_64 11/12 2024-03-25T16:37:54.116 INFO:teuthology.orchestra.run.smithi115.stdout: Verifying : python3-devel-3.9.17-2.el9.x86_64 12/12 2024-03-25T16:37:54.116 INFO:teuthology.orchestra.run.smithi115.stdout: 2024-03-25T16:37:54.116 INFO:teuthology.orchestra.run.smithi115.stdout:Installed: 2024-03-25T16:37:54.116 INFO:teuthology.orchestra.run.smithi115.stdout: ceph-mgr-diskprediction-local-2:19.0.0-1439.g5dcb190d.el9.noarch 2024-03-25T16:37:54.116 INFO:teuthology.orchestra.run.smithi115.stdout: flexiblas-3.0.4-8.el9.x86_64 2024-03-25T16:37:54.116 INFO:teuthology.orchestra.run.smithi115.stdout: flexiblas-netlib-3.0.4-8.el9.x86_64 2024-03-25T16:37:54.116 INFO:teuthology.orchestra.run.smithi115.stdout: flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 2024-03-25T16:37:54.116 INFO:teuthology.orchestra.run.smithi115.stdout: libgfortran-11.4.1-3.el9.x86_64 2024-03-25T16:37:54.116 INFO:teuthology.orchestra.run.smithi115.stdout: libquadmath-11.4.1-3.el9.x86_64 2024-03-25T16:37:54.116 INFO:teuthology.orchestra.run.smithi115.stdout: openblas-0.3.21-2.el9.x86_64 2024-03-25T16:37:54.116 INFO:teuthology.orchestra.run.smithi115.stdout: openblas-openmp-0.3.21-2.el9.x86_64 2024-03-25T16:37:54.116 INFO:teuthology.orchestra.run.smithi115.stdout: python3-devel-3.9.17-2.el9.x86_64 2024-03-25T16:37:54.116 INFO:teuthology.orchestra.run.smithi115.stdout: python3-numpy-1:1.20.1-5.el9.x86_64 2024-03-25T16:37:54.117 INFO:teuthology.orchestra.run.smithi115.stdout: python3-numpy-f2py-1:1.20.1-5.el9.x86_64 2024-03-25T16:37:54.117 INFO:teuthology.orchestra.run.smithi115.stdout: python3-scipy-1.6.2-8.el9.x86_64 2024-03-25T16:37:54.117 INFO:teuthology.orchestra.run.smithi115.stdout: 2024-03-25T16:37:54.117 INFO:teuthology.orchestra.run.smithi115.stdout:Complete! 2024-03-25T16:37:54.545 DEBUG:teuthology.orchestra.run.smithi115:> sudo yum -y install ceph-mgr-rook 2024-03-25T16:37:55.120 INFO:teuthology.orchestra.run.smithi115.stdout:Last metadata expiration check: 0:05:10 ago on Mon 25 Mar 2024 04:32:45 PM UTC. 2024-03-25T16:37:55.271 INFO:teuthology.orchestra.run.smithi115.stdout:Dependencies resolved. 2024-03-25T16:37:55.272 INFO:teuthology.orchestra.run.smithi115.stdout:======================================================================================== 2024-03-25T16:37:55.272 INFO:teuthology.orchestra.run.smithi115.stdout: Package Arch Version Repository Size 2024-03-25T16:37:55.273 INFO:teuthology.orchestra.run.smithi115.stdout:======================================================================================== 2024-03-25T16:37:55.273 INFO:teuthology.orchestra.run.smithi115.stdout:Installing: 2024-03-25T16:37:55.273 INFO:teuthology.orchestra.run.smithi115.stdout: ceph-mgr-rook noarch 2:19.0.0-1439.g5dcb190d.el9 ceph-noarch 52 k 2024-03-25T16:37:55.273 INFO:teuthology.orchestra.run.smithi115.stdout:Installing dependencies: 2024-03-25T16:37:55.273 INFO:teuthology.orchestra.run.smithi115.stdout: python3-cachetools noarch 4.2.4-1.el9 epel 32 k 2024-03-25T16:37:55.273 INFO:teuthology.orchestra.run.smithi115.stdout: python3-certifi noarch 2021.10.8-2.el9 epel 15 k 2024-03-25T16:37:55.273 INFO:teuthology.orchestra.run.smithi115.stdout: python3-google-auth noarch 1:2.28.2-1.el9 epel 203 k 2024-03-25T16:37:55.273 INFO:teuthology.orchestra.run.smithi115.stdout: python3-jsonpatch noarch 1.21-16.el9 CentOS-AppStream 24 k 2024-03-25T16:37:55.273 INFO:teuthology.orchestra.run.smithi115.stdout: python3-jsonpointer noarch 2.0-4.el9 CentOS-AppStream 18 k 2024-03-25T16:37:55.273 INFO:teuthology.orchestra.run.smithi115.stdout: python3-kubernetes noarch 1:26.1.0-2.el9 epel 1.0 M 2024-03-25T16:37:55.273 INFO:teuthology.orchestra.run.smithi115.stdout: python3-oauthlib noarch 3.1.1-5.el9 CentOS-AppStream 174 k 2024-03-25T16:37:55.273 INFO:teuthology.orchestra.run.smithi115.stdout: python3-pyasn1 noarch 0.4.8-6.el9 CentOS-AppStream 132 k 2024-03-25T16:37:55.273 INFO:teuthology.orchestra.run.smithi115.stdout: python3-pyasn1-modules noarch 0.4.8-6.el9 CentOS-AppStream 211 k 2024-03-25T16:37:55.274 INFO:teuthology.orchestra.run.smithi115.stdout: python3-requests-oauthlib noarch 1.3.0-12.el9 CentOS-AppStream 43 k 2024-03-25T16:37:55.274 INFO:teuthology.orchestra.run.smithi115.stdout: python3-rsa noarch 4.9-2.el9 epel 59 k 2024-03-25T16:37:55.274 INFO:teuthology.orchestra.run.smithi115.stdout: python3-websocket-client noarch 1.2.3-2.el9 epel 90 k 2024-03-25T16:37:55.274 INFO:teuthology.orchestra.run.smithi115.stdout: 2024-03-25T16:37:55.274 INFO:teuthology.orchestra.run.smithi115.stdout:Transaction Summary 2024-03-25T16:37:55.274 INFO:teuthology.orchestra.run.smithi115.stdout:======================================================================================== 2024-03-25T16:37:55.274 INFO:teuthology.orchestra.run.smithi115.stdout:Install 13 Packages 2024-03-25T16:37:55.274 INFO:teuthology.orchestra.run.smithi115.stdout: 2024-03-25T16:37:55.275 INFO:teuthology.orchestra.run.smithi115.stdout:Total download size: 2.0 M 2024-03-25T16:37:55.275 INFO:teuthology.orchestra.run.smithi115.stdout:Installed size: 26 M 2024-03-25T16:37:55.275 INFO:teuthology.orchestra.run.smithi115.stdout:Downloading Packages: 2024-03-25T16:37:55.727 INFO:teuthology.orchestra.run.smithi115.stdout:(1/13): python3-jsonpatch-1.21-16.el9.noarch.rp 261 kB/s | 24 kB 00:00 2024-03-25T16:37:55.752 INFO:teuthology.orchestra.run.smithi115.stdout:(2/13): python3-jsonpointer-2.0-4.el9.noarch.rp 153 kB/s | 18 kB 00:00 2024-03-25T16:37:55.827 INFO:teuthology.orchestra.run.smithi115.stdout:(3/13): python3-oauthlib-3.1.1-5.el9.noarch.rpm 1.7 MB/s | 174 kB 00:00 2024-03-25T16:37:55.853 INFO:teuthology.orchestra.run.smithi115.stdout:(4/13): python3-pyasn1-0.4.8-6.el9.noarch.rpm 1.7 MB/s | 132 kB 00:00 2024-03-25T16:37:55.886 INFO:teuthology.orchestra.run.smithi115.stdout:(5/13): python3-pyasn1-modules-0.4.8-6.el9.noar 3.5 MB/s | 211 kB 00:00 2024-03-25T16:37:55.911 INFO:teuthology.orchestra.run.smithi115.stdout:(6/13): python3-requests-oauthlib-1.3.0-12.el9. 734 kB/s | 43 kB 00:00 2024-03-25T16:37:56.253 INFO:teuthology.orchestra.run.smithi115.stdout:(7/13): python3-certifi-2021.10.8-2.el9.noarch. 44 kB/s | 15 kB 00:00 2024-03-25T16:37:56.320 INFO:teuthology.orchestra.run.smithi115.stdout:(8/13): python3-cachetools-4.2.4-1.el9.noarch.r 74 kB/s | 32 kB 00:00 2024-03-25T16:37:56.487 INFO:teuthology.orchestra.run.smithi115.stdout:(9/13): ceph-mgr-rook-19.0.0-1439.g5dcb190d.el9 61 kB/s | 52 kB 00:00 2024-03-25T16:37:56.588 INFO:teuthology.orchestra.run.smithi115.stdout:(10/13): python3-google-auth-2.28.2-1.el9.noarc 607 kB/s | 203 kB 00:00 2024-03-25T16:37:56.738 INFO:teuthology.orchestra.run.smithi115.stdout:(11/13): python3-websocket-client-1.2.3-2.el9.n 717 kB/s | 90 kB 00:00 2024-03-25T16:37:56.796 INFO:teuthology.orchestra.run.smithi115.stdout:(12/13): python3-kubernetes-26.1.0-2.el9.noarch 2.1 MB/s | 1.0 MB 00:00 2024-03-25T16:37:56.946 INFO:teuthology.orchestra.run.smithi115.stdout:(13/13): python3-rsa-4.9-2.el9.noarch.rpm 129 kB/s | 59 kB 00:00 2024-03-25T16:37:56.951 INFO:teuthology.orchestra.run.smithi115.stdout:-------------------------------------------------------------------------------- 2024-03-25T16:37:56.951 INFO:teuthology.orchestra.run.smithi115.stdout:Total 1.2 MB/s | 2.0 MB 00:01 2024-03-25T16:37:56.952 INFO:teuthology.orchestra.run.smithi115.stdout:Running transaction check 2024-03-25T16:37:56.965 INFO:teuthology.orchestra.run.smithi115.stdout:Transaction check succeeded. 2024-03-25T16:37:56.965 INFO:teuthology.orchestra.run.smithi115.stdout:Running transaction test 2024-03-25T16:37:57.057 INFO:teuthology.orchestra.run.smithi115.stdout:Transaction test succeeded. 2024-03-25T16:37:57.058 INFO:teuthology.orchestra.run.smithi115.stdout:Running transaction 2024-03-25T16:37:57.560 INFO:teuthology.orchestra.run.smithi115.stdout: Preparing : 1/1 2024-03-25T16:37:57.735 INFO:teuthology.orchestra.run.smithi115.stdout: Installing : python3-pyasn1-0.4.8-6.el9.noarch 1/13 2024-03-25T16:37:57.806 INFO:teuthology.orchestra.run.smithi115.stdout: Installing : python3-pyasn1-modules-0.4.8-6.el9.noarch 2/13 2024-03-25T16:37:57.861 INFO:teuthology.orchestra.run.smithi115.stdout: Installing : python3-rsa-4.9-2.el9.noarch 3/13 2024-03-25T16:37:57.899 INFO:teuthology.orchestra.run.smithi115.stdout: Installing : python3-websocket-client-1.2.3-2.el9.noarch 4/13 2024-03-25T16:37:57.940 INFO:teuthology.orchestra.run.smithi115.stdout: Installing : python3-certifi-2021.10.8-2.el9.noarch 5/13 2024-03-25T16:37:58.034 INFO:teuthology.orchestra.run.smithi115.stdout: Installing : python3-cachetools-4.2.4-1.el9.noarch 6/13 2024-03-25T16:37:58.152 INFO:teuthology.orchestra.run.smithi115.stdout: Installing : python3-google-auth-1:2.28.2-1.el9.noarch 7/13 2024-03-25T16:37:58.225 INFO:teuthology.orchestra.run.smithi115.stdout: Installing : python3-oauthlib-3.1.1-5.el9.noarch 8/13 2024-03-25T16:37:58.950 INFO:teuthology.orchestra.run.smithi115.stdout: Installing : python3-requests-oauthlib-1.3.0-12.el9.noarch 9/13 2024-03-25T16:37:58.997 INFO:teuthology.orchestra.run.smithi115.stdout: Installing : python3-kubernetes-1:26.1.0-2.el9.noarch 10/13 2024-03-25T16:37:59.039 INFO:teuthology.orchestra.run.smithi115.stdout: Installing : python3-jsonpointer-2.0-4.el9.noarch 11/13 2024-03-25T16:37:59.048 INFO:teuthology.orchestra.run.smithi001.stdout:(12/12): ceph-mgr-diskprediction-local-19.0.0-1 256 kB/s | 7.4 MB 00:29 2024-03-25T16:37:59.050 INFO:teuthology.orchestra.run.smithi001.stdout:-------------------------------------------------------------------------------- 2024-03-25T16:37:59.051 INFO:teuthology.orchestra.run.smithi001.stdout:Total 1.2 MB/s | 37 MB 00:29 2024-03-25T16:37:59.091 INFO:teuthology.orchestra.run.smithi001.stdout:Running transaction check 2024-03-25T16:37:59.096 INFO:teuthology.orchestra.run.smithi115.stdout: Installing : python3-jsonpatch-1.21-16.el9.noarch 12/13 2024-03-25T16:37:59.117 INFO:teuthology.orchestra.run.smithi001.stdout:Transaction check succeeded. 2024-03-25T16:37:59.117 INFO:teuthology.orchestra.run.smithi001.stdout:Running transaction test 2024-03-25T16:37:59.118 INFO:teuthology.orchestra.run.smithi115.stdout: Installing : ceph-mgr-rook-2:19.0.0-1439.g5dcb190d.el9.noarch 13/13 2024-03-25T16:37:59.495 INFO:teuthology.orchestra.run.smithi001.stdout:Transaction test succeeded. 2024-03-25T16:37:59.495 INFO:teuthology.orchestra.run.smithi001.stdout:Running transaction 2024-03-25T16:37:59.927 INFO:teuthology.orchestra.run.smithi115.stdout: Running scriptlet: ceph-mgr-rook-2:19.0.0-1439.g5dcb190d.el9.noarch 13/13 2024-03-25T16:37:59.927 INFO:teuthology.orchestra.run.smithi115.stdout: Verifying : ceph-mgr-rook-2:19.0.0-1439.g5dcb190d.el9.noarch 1/13 2024-03-25T16:37:59.927 INFO:teuthology.orchestra.run.smithi115.stdout: Verifying : python3-jsonpatch-1.21-16.el9.noarch 2/13 2024-03-25T16:37:59.927 INFO:teuthology.orchestra.run.smithi115.stdout: Verifying : python3-jsonpointer-2.0-4.el9.noarch 3/13 2024-03-25T16:37:59.927 INFO:teuthology.orchestra.run.smithi115.stdout: Verifying : python3-oauthlib-3.1.1-5.el9.noarch 4/13 2024-03-25T16:37:59.927 INFO:teuthology.orchestra.run.smithi115.stdout: Verifying : python3-pyasn1-0.4.8-6.el9.noarch 5/13 2024-03-25T16:37:59.927 INFO:teuthology.orchestra.run.smithi115.stdout: Verifying : python3-pyasn1-modules-0.4.8-6.el9.noarch 6/13 2024-03-25T16:37:59.927 INFO:teuthology.orchestra.run.smithi115.stdout: Verifying : python3-requests-oauthlib-1.3.0-12.el9.noarch 7/13 2024-03-25T16:37:59.927 INFO:teuthology.orchestra.run.smithi115.stdout: Verifying : python3-cachetools-4.2.4-1.el9.noarch 8/13 2024-03-25T16:37:59.927 INFO:teuthology.orchestra.run.smithi115.stdout: Verifying : python3-certifi-2021.10.8-2.el9.noarch 9/13 2024-03-25T16:37:59.927 INFO:teuthology.orchestra.run.smithi115.stdout: Verifying : python3-google-auth-1:2.28.2-1.el9.noarch 10/13 2024-03-25T16:37:59.927 INFO:teuthology.orchestra.run.smithi115.stdout: Verifying : python3-kubernetes-1:26.1.0-2.el9.noarch 11/13 2024-03-25T16:37:59.927 INFO:teuthology.orchestra.run.smithi115.stdout: Verifying : python3-rsa-4.9-2.el9.noarch 12/13 2024-03-25T16:38:00.256 INFO:teuthology.orchestra.run.smithi001.stdout: Preparing : 1/1 2024-03-25T16:38:00.301 INFO:teuthology.orchestra.run.smithi115.stdout: Verifying : python3-websocket-client-1.2.3-2.el9.noarch 13/13 2024-03-25T16:38:00.301 INFO:teuthology.orchestra.run.smithi115.stdout: 2024-03-25T16:38:00.301 INFO:teuthology.orchestra.run.smithi115.stdout:Installed: 2024-03-25T16:38:00.301 INFO:teuthology.orchestra.run.smithi115.stdout: ceph-mgr-rook-2:19.0.0-1439.g5dcb190d.el9.noarch 2024-03-25T16:38:00.301 INFO:teuthology.orchestra.run.smithi115.stdout: python3-cachetools-4.2.4-1.el9.noarch 2024-03-25T16:38:00.301 INFO:teuthology.orchestra.run.smithi115.stdout: python3-certifi-2021.10.8-2.el9.noarch 2024-03-25T16:38:00.301 INFO:teuthology.orchestra.run.smithi115.stdout: python3-google-auth-1:2.28.2-1.el9.noarch 2024-03-25T16:38:00.301 INFO:teuthology.orchestra.run.smithi115.stdout: python3-jsonpatch-1.21-16.el9.noarch 2024-03-25T16:38:00.301 INFO:teuthology.orchestra.run.smithi115.stdout: python3-jsonpointer-2.0-4.el9.noarch 2024-03-25T16:38:00.301 INFO:teuthology.orchestra.run.smithi115.stdout: python3-kubernetes-1:26.1.0-2.el9.noarch 2024-03-25T16:38:00.302 INFO:teuthology.orchestra.run.smithi115.stdout: python3-oauthlib-3.1.1-5.el9.noarch 2024-03-25T16:38:00.302 INFO:teuthology.orchestra.run.smithi115.stdout: python3-pyasn1-0.4.8-6.el9.noarch 2024-03-25T16:38:00.302 INFO:teuthology.orchestra.run.smithi115.stdout: python3-pyasn1-modules-0.4.8-6.el9.noarch 2024-03-25T16:38:00.302 INFO:teuthology.orchestra.run.smithi115.stdout: python3-requests-oauthlib-1.3.0-12.el9.noarch 2024-03-25T16:38:00.302 INFO:teuthology.orchestra.run.smithi115.stdout: python3-rsa-4.9-2.el9.noarch 2024-03-25T16:38:00.302 INFO:teuthology.orchestra.run.smithi115.stdout: python3-websocket-client-1.2.3-2.el9.noarch 2024-03-25T16:38:00.302 INFO:teuthology.orchestra.run.smithi115.stdout: 2024-03-25T16:38:00.302 INFO:teuthology.orchestra.run.smithi115.stdout:Complete! 2024-03-25T16:38:00.322 INFO:teuthology.orchestra.run.smithi001.stdout: Installing : libquadmath-11.4.1-3.el9.x86_64 1/12 2024-03-25T16:38:00.361 INFO:teuthology.orchestra.run.smithi001.stdout: Installing : libgfortran-11.4.1-3.el9.x86_64 2/12 2024-03-25T16:38:00.466 INFO:teuthology.orchestra.run.smithi001.stdout: Installing : flexiblas-3.0.4-8.el9.x86_64 3/12 2024-03-25T16:38:00.515 INFO:teuthology.orchestra.run.smithi001.stdout: Installing : python3-devel-3.9.17-2.el9.x86_64 4/12 2024-03-25T16:38:00.554 DEBUG:teuthology.orchestra.run.smithi115:> sudo yum -y install ceph-mgr-cephadm 2024-03-25T16:38:00.701 INFO:teuthology.orchestra.run.smithi001.stdout: Installing : openblas-0.3.21-2.el9.x86_64 5/12 2024-03-25T16:38:00.739 INFO:teuthology.orchestra.run.smithi001.stdout: Installing : openblas-openmp-0.3.21-2.el9.x86_64 6/12 2024-03-25T16:38:00.823 INFO:teuthology.orchestra.run.smithi001.stdout: Installing : flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 7/12 2024-03-25T16:38:01.152 INFO:teuthology.orchestra.run.smithi115.stdout:Last metadata expiration check: 0:05:16 ago on Mon 25 Mar 2024 04:32:45 PM UTC. 2024-03-25T16:38:01.309 INFO:teuthology.orchestra.run.smithi115.stdout:Dependencies resolved. 2024-03-25T16:38:01.310 INFO:teuthology.orchestra.run.smithi115.stdout:======================================================================================== 2024-03-25T16:38:01.310 INFO:teuthology.orchestra.run.smithi115.stdout: Package Arch Version Repository Size 2024-03-25T16:38:01.310 INFO:teuthology.orchestra.run.smithi115.stdout:======================================================================================== 2024-03-25T16:38:01.310 INFO:teuthology.orchestra.run.smithi115.stdout:Installing: 2024-03-25T16:38:01.310 INFO:teuthology.orchestra.run.smithi115.stdout: ceph-mgr-cephadm noarch 2:19.0.0-1439.g5dcb190d.el9 ceph-noarch 138 k 2024-03-25T16:38:01.311 INFO:teuthology.orchestra.run.smithi115.stdout:Installing dependencies: 2024-03-25T16:38:01.311 INFO:teuthology.orchestra.run.smithi115.stdout: python3-asyncssh noarch 2.13.2-5.el9 epel 548 k 2024-03-25T16:38:01.311 INFO:teuthology.orchestra.run.smithi115.stdout: python3-babel noarch 2.9.1-2.el9 CentOS-AppStream 5.8 M 2024-03-25T16:38:01.311 INFO:teuthology.orchestra.run.smithi115.stdout: python3-jinja2 noarch 2.11.3-5.el9 appstream 248 k 2024-03-25T16:38:01.311 INFO:teuthology.orchestra.run.smithi115.stdout: python3-natsort noarch 7.1.1-5.el9 epel 58 k 2024-03-25T16:38:01.311 INFO:teuthology.orchestra.run.smithi115.stdout: python3-typing-extensions noarch 4.4.0-2.el9 epel 51 k 2024-03-25T16:38:01.311 INFO:teuthology.orchestra.run.smithi115.stdout: 2024-03-25T16:38:01.311 INFO:teuthology.orchestra.run.smithi115.stdout:Transaction Summary 2024-03-25T16:38:01.311 INFO:teuthology.orchestra.run.smithi115.stdout:======================================================================================== 2024-03-25T16:38:01.311 INFO:teuthology.orchestra.run.smithi115.stdout:Install 6 Packages 2024-03-25T16:38:01.311 INFO:teuthology.orchestra.run.smithi115.stdout: 2024-03-25T16:38:01.312 INFO:teuthology.orchestra.run.smithi115.stdout:Total download size: 6.8 M 2024-03-25T16:38:01.312 INFO:teuthology.orchestra.run.smithi115.stdout:Installed size: 33 M 2024-03-25T16:38:01.312 INFO:teuthology.orchestra.run.smithi115.stdout:Downloading Packages: 2024-03-25T16:38:01.512 INFO:teuthology.orchestra.run.smithi001.stdout: Installing : flexiblas-netlib-3.0.4-8.el9.x86_64 8/12 2024-03-25T16:38:01.637 INFO:teuthology.orchestra.run.smithi001.stdout: Installing : python3-numpy-1:1.20.1-5.el9.x86_64 9/12 2024-03-25T16:38:01.809 INFO:teuthology.orchestra.run.smithi115.stdout:(1/6): python3-jinja2-2.11.3-5.el9.noarch.rpm 1.2 MB/s | 248 kB 00:00 2024-03-25T16:38:01.916 INFO:teuthology.orchestra.run.smithi115.stdout:(2/6): python3-babel-2.9.1-2.el9.noarch.rpm 19 MB/s | 5.8 MB 00:00 2024-03-25T16:38:02.283 INFO:teuthology.orchestra.run.smithi115.stdout:(3/6): python3-natsort-7.1.1-5.el9.noarch.rpm 168 kB/s | 58 kB 00:00 2024-03-25T16:38:02.392 INFO:teuthology.orchestra.run.smithi115.stdout:(4/6): python3-typing-extensions-4.4.0-2.el9.no 470 kB/s | 51 kB 00:00 2024-03-25T16:38:02.417 INFO:teuthology.orchestra.run.smithi115.stdout:(5/6): python3-asyncssh-2.13.2-5.el9.noarch.rpm 903 kB/s | 548 kB 00:00 2024-03-25T16:38:02.917 INFO:teuthology.orchestra.run.smithi115.stdout:(6/6): ceph-mgr-cephadm-19.0.0-1439.g5dcb190d.e 106 kB/s | 138 kB 00:01 2024-03-25T16:38:02.920 INFO:teuthology.orchestra.run.smithi115.stdout:-------------------------------------------------------------------------------- 2024-03-25T16:38:02.920 INFO:teuthology.orchestra.run.smithi115.stdout:Total 4.2 MB/s | 6.8 MB 00:01 2024-03-25T16:38:02.934 INFO:teuthology.orchestra.run.smithi115.stdout:Running transaction check 2024-03-25T16:38:02.935 INFO:teuthology.orchestra.run.smithi001.stdout: Installing : python3-numpy-f2py-1:1.20.1-5.el9.x86_64 10/12 2024-03-25T16:38:02.944 INFO:teuthology.orchestra.run.smithi115.stdout:Transaction check succeeded. 2024-03-25T16:38:02.944 INFO:teuthology.orchestra.run.smithi115.stdout:Running transaction test 2024-03-25T16:38:03.034 INFO:teuthology.orchestra.run.smithi115.stdout:Transaction test succeeded. 2024-03-25T16:38:03.035 INFO:teuthology.orchestra.run.smithi115.stdout:Running transaction 2024-03-25T16:38:03.338 INFO:teuthology.orchestra.run.smithi115.stdout: Preparing : 1/1 2024-03-25T16:38:03.503 INFO:teuthology.orchestra.run.smithi115.stdout: Installing : python3-typing-extensions-4.4.0-2.el9.noarch 1/6 2024-03-25T16:38:03.560 INFO:teuthology.orchestra.run.smithi115.stdout: Installing : python3-asyncssh-2.13.2-5.el9.noarch 2/6 2024-03-25T16:38:04.023 INFO:teuthology.orchestra.run.smithi001.stdout: Installing : python3-scipy-1.6.2-8.el9.x86_64 11/12 2024-03-25T16:38:04.046 INFO:teuthology.orchestra.run.smithi001.stdout: Installing : ceph-mgr-diskprediction-local-2:19.0.0-1439.g5dcb1 12/12 2024-03-25T16:38:04.100 INFO:teuthology.orchestra.run.smithi115.stdout: Installing : python3-natsort-7.1.1-5.el9.noarch 3/6 2024-03-25T16:38:04.179 INFO:teuthology.orchestra.run.smithi115.stdout: Installing : python3-babel-2.9.1-2.el9.noarch 4/6 2024-03-25T16:38:04.241 INFO:teuthology.orchestra.run.smithi115.stdout: Installing : python3-jinja2-2.11.3-5.el9.noarch 5/6 2024-03-25T16:38:04.264 INFO:teuthology.orchestra.run.smithi115.stdout: Installing : ceph-mgr-cephadm-2:19.0.0-1439.g5dcb190d.el9.noarch 6/6 2024-03-25T16:38:05.036 INFO:teuthology.orchestra.run.smithi115.stdout: Running scriptlet: ceph-mgr-cephadm-2:19.0.0-1439.g5dcb190d.el9.noarch 6/6 2024-03-25T16:38:05.036 INFO:teuthology.orchestra.run.smithi115.stdout: Verifying : ceph-mgr-cephadm-2:19.0.0-1439.g5dcb190d.el9.noarch 1/6 2024-03-25T16:38:05.037 INFO:teuthology.orchestra.run.smithi115.stdout: Verifying : python3-babel-2.9.1-2.el9.noarch 2/6 2024-03-25T16:38:05.037 INFO:teuthology.orchestra.run.smithi115.stdout: Verifying : python3-jinja2-2.11.3-5.el9.noarch 3/6 2024-03-25T16:38:05.037 INFO:teuthology.orchestra.run.smithi115.stdout: Verifying : python3-asyncssh-2.13.2-5.el9.noarch 4/6 2024-03-25T16:38:05.037 INFO:teuthology.orchestra.run.smithi115.stdout: Verifying : python3-natsort-7.1.1-5.el9.noarch 5/6 2024-03-25T16:38:05.328 INFO:teuthology.orchestra.run.smithi115.stdout: Verifying : python3-typing-extensions-4.4.0-2.el9.noarch 6/6 2024-03-25T16:38:05.328 INFO:teuthology.orchestra.run.smithi115.stdout: 2024-03-25T16:38:05.328 INFO:teuthology.orchestra.run.smithi115.stdout:Installed: 2024-03-25T16:38:05.328 INFO:teuthology.orchestra.run.smithi115.stdout: ceph-mgr-cephadm-2:19.0.0-1439.g5dcb190d.el9.noarch 2024-03-25T16:38:05.328 INFO:teuthology.orchestra.run.smithi115.stdout: python3-asyncssh-2.13.2-5.el9.noarch 2024-03-25T16:38:05.328 INFO:teuthology.orchestra.run.smithi115.stdout: python3-babel-2.9.1-2.el9.noarch 2024-03-25T16:38:05.328 INFO:teuthology.orchestra.run.smithi115.stdout: python3-jinja2-2.11.3-5.el9.noarch 2024-03-25T16:38:05.328 INFO:teuthology.orchestra.run.smithi115.stdout: python3-natsort-7.1.1-5.el9.noarch 2024-03-25T16:38:05.329 INFO:teuthology.orchestra.run.smithi115.stdout: python3-typing-extensions-4.4.0-2.el9.noarch 2024-03-25T16:38:05.329 INFO:teuthology.orchestra.run.smithi115.stdout: 2024-03-25T16:38:05.329 INFO:teuthology.orchestra.run.smithi115.stdout:Complete! 2024-03-25T16:38:05.570 DEBUG:teuthology.orchestra.run.smithi115:> sudo yum -y install ceph-fuse 2024-03-25T16:38:06.134 INFO:teuthology.orchestra.run.smithi115.stdout:Last metadata expiration check: 0:05:21 ago on Mon 25 Mar 2024 04:32:45 PM UTC. 2024-03-25T16:38:06.285 INFO:teuthology.orchestra.run.smithi115.stdout:Dependencies resolved. 2024-03-25T16:38:06.286 INFO:teuthology.orchestra.run.smithi115.stdout:================================================================================ 2024-03-25T16:38:06.286 INFO:teuthology.orchestra.run.smithi115.stdout: Package Arch Version Repository Size 2024-03-25T16:38:06.286 INFO:teuthology.orchestra.run.smithi115.stdout:================================================================================ 2024-03-25T16:38:06.286 INFO:teuthology.orchestra.run.smithi115.stdout:Installing: 2024-03-25T16:38:06.286 INFO:teuthology.orchestra.run.smithi115.stdout: ceph-fuse x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 885 k 2024-03-25T16:38:06.287 INFO:teuthology.orchestra.run.smithi115.stdout:Installing dependencies: 2024-03-25T16:38:06.287 INFO:teuthology.orchestra.run.smithi115.stdout: fuse x86_64 2.9.9-15.el9 CentOS-BaseOS 79 k 2024-03-25T16:38:06.287 INFO:teuthology.orchestra.run.smithi115.stdout: 2024-03-25T16:38:06.287 INFO:teuthology.orchestra.run.smithi115.stdout:Transaction Summary 2024-03-25T16:38:06.287 INFO:teuthology.orchestra.run.smithi115.stdout:================================================================================ 2024-03-25T16:38:06.287 INFO:teuthology.orchestra.run.smithi115.stdout:Install 2 Packages 2024-03-25T16:38:06.287 INFO:teuthology.orchestra.run.smithi115.stdout: 2024-03-25T16:38:06.287 INFO:teuthology.orchestra.run.smithi115.stdout:Total download size: 964 k 2024-03-25T16:38:06.287 INFO:teuthology.orchestra.run.smithi115.stdout:Installed size: 2.8 M 2024-03-25T16:38:06.287 INFO:teuthology.orchestra.run.smithi115.stdout:Downloading Packages: 2024-03-25T16:38:06.401 INFO:teuthology.orchestra.run.smithi115.stdout:(1/2): fuse-2.9.9-15.el9.x86_64.rpm 690 kB/s | 79 kB 00:00 2024-03-25T16:38:06.522 INFO:teuthology.orchestra.run.smithi001.stdout: Running scriptlet: ceph-mgr-diskprediction-local-2:19.0.0-1439.g5dcb1 12/12 2024-03-25T16:38:06.522 INFO:teuthology.orchestra.run.smithi001.stdout: Verifying : ceph-mgr-diskprediction-local-2:19.0.0-1439.g5dcb1 1/12 2024-03-25T16:38:06.522 INFO:teuthology.orchestra.run.smithi001.stdout: Verifying : flexiblas-3.0.4-8.el9.x86_64 2/12 2024-03-25T16:38:06.522 INFO:teuthology.orchestra.run.smithi001.stdout: Verifying : flexiblas-netlib-3.0.4-8.el9.x86_64 3/12 2024-03-25T16:38:06.522 INFO:teuthology.orchestra.run.smithi001.stdout: Verifying : flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 4/12 2024-03-25T16:38:06.523 INFO:teuthology.orchestra.run.smithi001.stdout: Verifying : openblas-0.3.21-2.el9.x86_64 5/12 2024-03-25T16:38:06.523 INFO:teuthology.orchestra.run.smithi001.stdout: Verifying : openblas-openmp-0.3.21-2.el9.x86_64 6/12 2024-03-25T16:38:06.523 INFO:teuthology.orchestra.run.smithi001.stdout: Verifying : python3-numpy-1:1.20.1-5.el9.x86_64 7/12 2024-03-25T16:38:06.523 INFO:teuthology.orchestra.run.smithi001.stdout: Verifying : python3-numpy-f2py-1:1.20.1-5.el9.x86_64 8/12 2024-03-25T16:38:06.523 INFO:teuthology.orchestra.run.smithi001.stdout: Verifying : python3-scipy-1.6.2-8.el9.x86_64 9/12 2024-03-25T16:38:06.523 INFO:teuthology.orchestra.run.smithi001.stdout: Verifying : libgfortran-11.4.1-3.el9.x86_64 10/12 2024-03-25T16:38:06.523 INFO:teuthology.orchestra.run.smithi001.stdout: Verifying : libquadmath-11.4.1-3.el9.x86_64 11/12 2024-03-25T16:38:07.121 INFO:teuthology.orchestra.run.smithi001.stdout: Verifying : python3-devel-3.9.17-2.el9.x86_64 12/12 2024-03-25T16:38:07.121 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-25T16:38:07.121 INFO:teuthology.orchestra.run.smithi001.stdout:Installed: 2024-03-25T16:38:07.121 INFO:teuthology.orchestra.run.smithi001.stdout: ceph-mgr-diskprediction-local-2:19.0.0-1439.g5dcb190d.el9.noarch 2024-03-25T16:38:07.121 INFO:teuthology.orchestra.run.smithi001.stdout: flexiblas-3.0.4-8.el9.x86_64 2024-03-25T16:38:07.121 INFO:teuthology.orchestra.run.smithi001.stdout: flexiblas-netlib-3.0.4-8.el9.x86_64 2024-03-25T16:38:07.121 INFO:teuthology.orchestra.run.smithi001.stdout: flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 2024-03-25T16:38:07.121 INFO:teuthology.orchestra.run.smithi001.stdout: libgfortran-11.4.1-3.el9.x86_64 2024-03-25T16:38:07.121 INFO:teuthology.orchestra.run.smithi001.stdout: libquadmath-11.4.1-3.el9.x86_64 2024-03-25T16:38:07.121 INFO:teuthology.orchestra.run.smithi001.stdout: openblas-0.3.21-2.el9.x86_64 2024-03-25T16:38:07.121 INFO:teuthology.orchestra.run.smithi001.stdout: openblas-openmp-0.3.21-2.el9.x86_64 2024-03-25T16:38:07.121 INFO:teuthology.orchestra.run.smithi001.stdout: python3-devel-3.9.17-2.el9.x86_64 2024-03-25T16:38:07.121 INFO:teuthology.orchestra.run.smithi001.stdout: python3-numpy-1:1.20.1-5.el9.x86_64 2024-03-25T16:38:07.122 INFO:teuthology.orchestra.run.smithi001.stdout: python3-numpy-f2py-1:1.20.1-5.el9.x86_64 2024-03-25T16:38:07.122 INFO:teuthology.orchestra.run.smithi001.stdout: python3-scipy-1.6.2-8.el9.x86_64 2024-03-25T16:38:07.122 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-25T16:38:07.122 INFO:teuthology.orchestra.run.smithi001.stdout:Complete! 2024-03-25T16:38:07.537 DEBUG:teuthology.orchestra.run.smithi001:> sudo yum -y install ceph-mgr-rook 2024-03-25T16:38:08.114 INFO:teuthology.orchestra.run.smithi001.stdout:Last metadata expiration check: 0:05:19 ago on Mon 25 Mar 2024 04:32:49 PM UTC. 2024-03-25T16:38:08.269 INFO:teuthology.orchestra.run.smithi001.stdout:Dependencies resolved. 2024-03-25T16:38:08.270 INFO:teuthology.orchestra.run.smithi001.stdout:======================================================================================== 2024-03-25T16:38:08.270 INFO:teuthology.orchestra.run.smithi001.stdout: Package Arch Version Repository Size 2024-03-25T16:38:08.270 INFO:teuthology.orchestra.run.smithi001.stdout:======================================================================================== 2024-03-25T16:38:08.270 INFO:teuthology.orchestra.run.smithi001.stdout:Installing: 2024-03-25T16:38:08.270 INFO:teuthology.orchestra.run.smithi001.stdout: ceph-mgr-rook noarch 2:19.0.0-1439.g5dcb190d.el9 ceph-noarch 52 k 2024-03-25T16:38:08.270 INFO:teuthology.orchestra.run.smithi001.stdout:Installing dependencies: 2024-03-25T16:38:08.270 INFO:teuthology.orchestra.run.smithi001.stdout: python3-cachetools noarch 4.2.4-1.el9 epel 32 k 2024-03-25T16:38:08.270 INFO:teuthology.orchestra.run.smithi001.stdout: python3-certifi noarch 2021.10.8-2.el9 epel 15 k 2024-03-25T16:38:08.271 INFO:teuthology.orchestra.run.smithi001.stdout: python3-google-auth noarch 1:2.28.2-1.el9 epel 203 k 2024-03-25T16:38:08.271 INFO:teuthology.orchestra.run.smithi001.stdout: python3-jsonpatch noarch 1.21-16.el9 CentOS-AppStream 24 k 2024-03-25T16:38:08.271 INFO:teuthology.orchestra.run.smithi001.stdout: python3-jsonpointer noarch 2.0-4.el9 CentOS-AppStream 18 k 2024-03-25T16:38:08.271 INFO:teuthology.orchestra.run.smithi001.stdout: python3-kubernetes noarch 1:26.1.0-2.el9 epel 1.0 M 2024-03-25T16:38:08.271 INFO:teuthology.orchestra.run.smithi001.stdout: python3-oauthlib noarch 3.1.1-5.el9 CentOS-AppStream 174 k 2024-03-25T16:38:08.271 INFO:teuthology.orchestra.run.smithi001.stdout: python3-pyasn1 noarch 0.4.8-6.el9 CentOS-AppStream 132 k 2024-03-25T16:38:08.271 INFO:teuthology.orchestra.run.smithi001.stdout: python3-pyasn1-modules noarch 0.4.8-6.el9 CentOS-AppStream 211 k 2024-03-25T16:38:08.271 INFO:teuthology.orchestra.run.smithi001.stdout: python3-requests-oauthlib noarch 1.3.0-12.el9 CentOS-AppStream 43 k 2024-03-25T16:38:08.271 INFO:teuthology.orchestra.run.smithi001.stdout: python3-rsa noarch 4.9-2.el9 epel 59 k 2024-03-25T16:38:08.271 INFO:teuthology.orchestra.run.smithi001.stdout: python3-websocket-client noarch 1.2.3-2.el9 epel 90 k 2024-03-25T16:38:08.271 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-25T16:38:08.271 INFO:teuthology.orchestra.run.smithi001.stdout:Transaction Summary 2024-03-25T16:38:08.271 INFO:teuthology.orchestra.run.smithi001.stdout:======================================================================================== 2024-03-25T16:38:08.271 INFO:teuthology.orchestra.run.smithi001.stdout:Install 13 Packages 2024-03-25T16:38:08.271 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-25T16:38:08.272 INFO:teuthology.orchestra.run.smithi001.stdout:Total download size: 2.0 M 2024-03-25T16:38:08.272 INFO:teuthology.orchestra.run.smithi001.stdout:Installed size: 26 M 2024-03-25T16:38:08.272 INFO:teuthology.orchestra.run.smithi001.stdout:Downloading Packages: 2024-03-25T16:38:08.564 INFO:teuthology.orchestra.run.smithi001.stdout:(1/13): python3-jsonpointer-2.0-4.el9.noarch.rp 195 kB/s | 18 kB 00:00 2024-03-25T16:38:08.589 INFO:teuthology.orchestra.run.smithi001.stdout:(2/13): python3-jsonpatch-1.21-16.el9.noarch.rp 207 kB/s | 24 kB 00:00 2024-03-25T16:38:08.665 INFO:teuthology.orchestra.run.smithi001.stdout:(3/13): python3-oauthlib-3.1.1-5.el9.noarch.rpm 1.7 MB/s | 174 kB 00:00 2024-03-25T16:38:08.690 INFO:teuthology.orchestra.run.smithi001.stdout:(4/13): python3-pyasn1-0.4.8-6.el9.noarch.rpm 1.7 MB/s | 132 kB 00:00 2024-03-25T16:38:08.723 INFO:teuthology.orchestra.run.smithi001.stdout:(5/13): python3-pyasn1-modules-0.4.8-6.el9.noar 3.5 MB/s | 211 kB 00:00 2024-03-25T16:38:08.749 INFO:teuthology.orchestra.run.smithi001.stdout:(6/13): python3-requests-oauthlib-1.3.0-12.el9. 732 kB/s | 43 kB 00:00 2024-03-25T16:38:08.807 INFO:teuthology.orchestra.run.smithi001.stdout:(7/13): python3-certifi-2021.10.8-2.el9.noarch. 256 kB/s | 15 kB 00:00 2024-03-25T16:38:08.833 INFO:teuthology.orchestra.run.smithi001.stdout:(8/13): python3-cachetools-4.2.4-1.el9.noarch.r 298 kB/s | 32 kB 00:00 2024-03-25T16:38:08.883 INFO:teuthology.orchestra.run.smithi001.stdout:(9/13): python3-google-auth-2.28.2-1.el9.noarch 2.6 MB/s | 203 kB 00:00 2024-03-25T16:38:08.917 INFO:teuthology.orchestra.run.smithi001.stdout:(10/13): python3-rsa-4.9-2.el9.noarch.rpm 1.7 MB/s | 59 kB 00:00 2024-03-25T16:38:08.950 INFO:teuthology.orchestra.run.smithi001.stdout:(11/13): python3-kubernetes-26.1.0-2.el9.noarch 8.7 MB/s | 1.0 MB 00:00 2024-03-25T16:38:08.967 INFO:teuthology.orchestra.run.smithi001.stdout:(12/13): python3-websocket-client-1.2.3-2.el9.n 1.8 MB/s | 90 kB 00:00 2024-03-25T16:38:09.142 INFO:teuthology.orchestra.run.smithi001.stdout:(13/13): ceph-mgr-rook-19.0.0-1439.g5dcb190d.el 78 kB/s | 52 kB 00:00 2024-03-25T16:38:09.144 INFO:teuthology.orchestra.run.smithi001.stdout:-------------------------------------------------------------------------------- 2024-03-25T16:38:09.144 INFO:teuthology.orchestra.run.smithi001.stdout:Total 2.3 MB/s | 2.0 MB 00:00 2024-03-25T16:38:09.145 INFO:teuthology.orchestra.run.smithi001.stdout:Running transaction check 2024-03-25T16:38:09.159 INFO:teuthology.orchestra.run.smithi001.stdout:Transaction check succeeded. 2024-03-25T16:38:09.159 INFO:teuthology.orchestra.run.smithi001.stdout:Running transaction test 2024-03-25T16:38:09.255 INFO:teuthology.orchestra.run.smithi001.stdout:Transaction test succeeded. 2024-03-25T16:38:09.255 INFO:teuthology.orchestra.run.smithi001.stdout:Running transaction 2024-03-25T16:38:09.401 INFO:teuthology.orchestra.run.smithi115.stdout:(2/2): ceph-fuse-19.0.0-1439.g5dcb190d.el9.x86_ 284 kB/s | 885 kB 00:03 2024-03-25T16:38:09.437 INFO:teuthology.orchestra.run.smithi115.stdout:-------------------------------------------------------------------------------- 2024-03-25T16:38:09.437 INFO:teuthology.orchestra.run.smithi115.stdout:Total 309 kB/s | 964 kB 00:03 2024-03-25T16:38:09.437 INFO:teuthology.orchestra.run.smithi115.stdout:Running transaction check 2024-03-25T16:38:09.437 INFO:teuthology.orchestra.run.smithi115.stdout:Transaction check succeeded. 2024-03-25T16:38:09.437 INFO:teuthology.orchestra.run.smithi115.stdout:Running transaction test 2024-03-25T16:38:09.502 INFO:teuthology.orchestra.run.smithi115.stdout:Transaction test succeeded. 2024-03-25T16:38:09.503 INFO:teuthology.orchestra.run.smithi115.stdout:Running transaction 2024-03-25T16:38:09.684 INFO:teuthology.orchestra.run.smithi115.stdout: Preparing : 1/1 2024-03-25T16:38:09.827 INFO:teuthology.orchestra.run.smithi115.stdout: Installing : fuse-2.9.9-15.el9.x86_64 1/2 2024-03-25T16:38:09.859 INFO:teuthology.orchestra.run.smithi001.stdout: Preparing : 1/1 2024-03-25T16:38:09.910 INFO:teuthology.orchestra.run.smithi115.stdout: Installing : ceph-fuse-2:19.0.0-1439.g5dcb190d.el9.x86_64 2/2 2024-03-25T16:38:10.038 INFO:teuthology.orchestra.run.smithi001.stdout: Installing : python3-pyasn1-0.4.8-6.el9.noarch 1/13 2024-03-25T16:38:10.087 INFO:teuthology.orchestra.run.smithi001.stdout: Installing : python3-pyasn1-modules-0.4.8-6.el9.noarch 2/13 2024-03-25T16:38:10.143 INFO:teuthology.orchestra.run.smithi001.stdout: Installing : python3-rsa-4.9-2.el9.noarch 3/13 2024-03-25T16:38:10.187 INFO:teuthology.orchestra.run.smithi001.stdout: Installing : python3-websocket-client-1.2.3-2.el9.noarch 4/13 2024-03-25T16:38:10.228 INFO:teuthology.orchestra.run.smithi001.stdout: Installing : python3-certifi-2021.10.8-2.el9.noarch 5/13 2024-03-25T16:38:10.325 INFO:teuthology.orchestra.run.smithi001.stdout: Installing : python3-cachetools-4.2.4-1.el9.noarch 6/13 2024-03-25T16:38:10.434 INFO:teuthology.orchestra.run.smithi001.stdout: Installing : python3-google-auth-1:2.28.2-1.el9.noarch 7/13 2024-03-25T16:38:10.488 INFO:teuthology.orchestra.run.smithi001.stdout: Installing : python3-oauthlib-3.1.1-5.el9.noarch 8/13 2024-03-25T16:38:10.556 INFO:teuthology.orchestra.run.smithi115.stdout: Running scriptlet: ceph-fuse-2:19.0.0-1439.g5dcb190d.el9.x86_64 2/2 2024-03-25T16:38:10.556 INFO:teuthology.orchestra.run.smithi115.stdout: Verifying : ceph-fuse-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/2 2024-03-25T16:38:10.843 INFO:teuthology.orchestra.run.smithi115.stdout: Verifying : fuse-2.9.9-15.el9.x86_64 2/2 2024-03-25T16:38:10.941 INFO:teuthology.orchestra.run.smithi115.stdout: 2024-03-25T16:38:10.941 INFO:teuthology.orchestra.run.smithi115.stdout:Installed: 2024-03-25T16:38:10.942 INFO:teuthology.orchestra.run.smithi115.stdout: ceph-fuse-2:19.0.0-1439.g5dcb190d.el9.x86_64 fuse-2.9.9-15.el9.x86_64 2024-03-25T16:38:10.942 INFO:teuthology.orchestra.run.smithi115.stdout: 2024-03-25T16:38:10.942 INFO:teuthology.orchestra.run.smithi115.stdout:Complete! 2024-03-25T16:38:11.105 DEBUG:teuthology.orchestra.run.smithi115:> sudo yum -y install ceph-volume 2024-03-25T16:38:11.245 INFO:teuthology.orchestra.run.smithi001.stdout: Installing : python3-requests-oauthlib-1.3.0-12.el9.noarch 9/13 2024-03-25T16:38:11.293 INFO:teuthology.orchestra.run.smithi001.stdout: Installing : python3-kubernetes-1:26.1.0-2.el9.noarch 10/13 2024-03-25T16:38:11.326 INFO:teuthology.orchestra.run.smithi001.stdout: Installing : python3-jsonpointer-2.0-4.el9.noarch 11/13 2024-03-25T16:38:11.367 INFO:teuthology.orchestra.run.smithi001.stdout: Installing : python3-jsonpatch-1.21-16.el9.noarch 12/13 2024-03-25T16:38:11.388 INFO:teuthology.orchestra.run.smithi001.stdout: Installing : ceph-mgr-rook-2:19.0.0-1439.g5dcb190d.el9.noarch 13/13 2024-03-25T16:38:11.675 INFO:teuthology.orchestra.run.smithi115.stdout:Last metadata expiration check: 0:05:26 ago on Mon 25 Mar 2024 04:32:45 PM UTC. 2024-03-25T16:38:11.831 INFO:teuthology.orchestra.run.smithi115.stdout:Dependencies resolved. 2024-03-25T16:38:11.832 INFO:teuthology.orchestra.run.smithi115.stdout:================================================================================ 2024-03-25T16:38:11.832 INFO:teuthology.orchestra.run.smithi115.stdout: Package Arch Version Repository Size 2024-03-25T16:38:11.832 INFO:teuthology.orchestra.run.smithi115.stdout:================================================================================ 2024-03-25T16:38:11.832 INFO:teuthology.orchestra.run.smithi115.stdout:Installing: 2024-03-25T16:38:11.832 INFO:teuthology.orchestra.run.smithi115.stdout: ceph-volume noarch 2:19.0.0-1439.g5dcb190d.el9 ceph-noarch 264 k 2024-03-25T16:38:11.832 INFO:teuthology.orchestra.run.smithi115.stdout:Installing dependencies: 2024-03-25T16:38:11.832 INFO:teuthology.orchestra.run.smithi115.stdout: python3-packaging noarch 20.9-5.el9 CentOS-AppStream 69 k 2024-03-25T16:38:11.832 INFO:teuthology.orchestra.run.smithi115.stdout: 2024-03-25T16:38:11.832 INFO:teuthology.orchestra.run.smithi115.stdout:Transaction Summary 2024-03-25T16:38:11.832 INFO:teuthology.orchestra.run.smithi115.stdout:================================================================================ 2024-03-25T16:38:11.833 INFO:teuthology.orchestra.run.smithi115.stdout:Install 2 Packages 2024-03-25T16:38:11.833 INFO:teuthology.orchestra.run.smithi115.stdout: 2024-03-25T16:38:11.833 INFO:teuthology.orchestra.run.smithi115.stdout:Total download size: 332 k 2024-03-25T16:38:11.833 INFO:teuthology.orchestra.run.smithi115.stdout:Installed size: 1.5 M 2024-03-25T16:38:11.833 INFO:teuthology.orchestra.run.smithi115.stdout:Downloading Packages: 2024-03-25T16:38:11.948 INFO:teuthology.orchestra.run.smithi115.stdout:(1/2): python3-packaging-20.9-5.el9.noarch.rpm 601 kB/s | 69 kB 00:00 2024-03-25T16:38:12.098 INFO:teuthology.orchestra.run.smithi001.stdout: Running scriptlet: ceph-mgr-rook-2:19.0.0-1439.g5dcb190d.el9.noarch 13/13 2024-03-25T16:38:12.098 INFO:teuthology.orchestra.run.smithi001.stdout: Verifying : ceph-mgr-rook-2:19.0.0-1439.g5dcb190d.el9.noarch 1/13 2024-03-25T16:38:12.098 INFO:teuthology.orchestra.run.smithi001.stdout: Verifying : python3-jsonpatch-1.21-16.el9.noarch 2/13 2024-03-25T16:38:12.098 INFO:teuthology.orchestra.run.smithi001.stdout: Verifying : python3-jsonpointer-2.0-4.el9.noarch 3/13 2024-03-25T16:38:12.098 INFO:teuthology.orchestra.run.smithi001.stdout: Verifying : python3-oauthlib-3.1.1-5.el9.noarch 4/13 2024-03-25T16:38:12.098 INFO:teuthology.orchestra.run.smithi001.stdout: Verifying : python3-pyasn1-0.4.8-6.el9.noarch 5/13 2024-03-25T16:38:12.098 INFO:teuthology.orchestra.run.smithi001.stdout: Verifying : python3-pyasn1-modules-0.4.8-6.el9.noarch 6/13 2024-03-25T16:38:12.098 INFO:teuthology.orchestra.run.smithi001.stdout: Verifying : python3-requests-oauthlib-1.3.0-12.el9.noarch 7/13 2024-03-25T16:38:12.098 INFO:teuthology.orchestra.run.smithi001.stdout: Verifying : python3-cachetools-4.2.4-1.el9.noarch 8/13 2024-03-25T16:38:12.098 INFO:teuthology.orchestra.run.smithi001.stdout: Verifying : python3-certifi-2021.10.8-2.el9.noarch 9/13 2024-03-25T16:38:12.098 INFO:teuthology.orchestra.run.smithi001.stdout: Verifying : python3-google-auth-1:2.28.2-1.el9.noarch 10/13 2024-03-25T16:38:12.099 INFO:teuthology.orchestra.run.smithi001.stdout: Verifying : python3-kubernetes-1:26.1.0-2.el9.noarch 11/13 2024-03-25T16:38:12.099 INFO:teuthology.orchestra.run.smithi001.stdout: Verifying : python3-rsa-4.9-2.el9.noarch 12/13 2024-03-25T16:38:12.461 INFO:teuthology.orchestra.run.smithi001.stdout: Verifying : python3-websocket-client-1.2.3-2.el9.noarch 13/13 2024-03-25T16:38:12.461 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-25T16:38:12.461 INFO:teuthology.orchestra.run.smithi001.stdout:Installed: 2024-03-25T16:38:12.461 INFO:teuthology.orchestra.run.smithi001.stdout: ceph-mgr-rook-2:19.0.0-1439.g5dcb190d.el9.noarch 2024-03-25T16:38:12.461 INFO:teuthology.orchestra.run.smithi001.stdout: python3-cachetools-4.2.4-1.el9.noarch 2024-03-25T16:38:12.462 INFO:teuthology.orchestra.run.smithi001.stdout: python3-certifi-2021.10.8-2.el9.noarch 2024-03-25T16:38:12.462 INFO:teuthology.orchestra.run.smithi001.stdout: python3-google-auth-1:2.28.2-1.el9.noarch 2024-03-25T16:38:12.462 INFO:teuthology.orchestra.run.smithi001.stdout: python3-jsonpatch-1.21-16.el9.noarch 2024-03-25T16:38:12.462 INFO:teuthology.orchestra.run.smithi001.stdout: python3-jsonpointer-2.0-4.el9.noarch 2024-03-25T16:38:12.462 INFO:teuthology.orchestra.run.smithi001.stdout: python3-kubernetes-1:26.1.0-2.el9.noarch 2024-03-25T16:38:12.462 INFO:teuthology.orchestra.run.smithi001.stdout: python3-oauthlib-3.1.1-5.el9.noarch 2024-03-25T16:38:12.462 INFO:teuthology.orchestra.run.smithi001.stdout: python3-pyasn1-0.4.8-6.el9.noarch 2024-03-25T16:38:12.462 INFO:teuthology.orchestra.run.smithi001.stdout: python3-pyasn1-modules-0.4.8-6.el9.noarch 2024-03-25T16:38:12.462 INFO:teuthology.orchestra.run.smithi001.stdout: python3-requests-oauthlib-1.3.0-12.el9.noarch 2024-03-25T16:38:12.462 INFO:teuthology.orchestra.run.smithi001.stdout: python3-rsa-4.9-2.el9.noarch 2024-03-25T16:38:12.462 INFO:teuthology.orchestra.run.smithi001.stdout: python3-websocket-client-1.2.3-2.el9.noarch 2024-03-25T16:38:12.462 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-25T16:38:12.462 INFO:teuthology.orchestra.run.smithi001.stdout:Complete! 2024-03-25T16:38:12.741 DEBUG:teuthology.orchestra.run.smithi001:> sudo yum -y install ceph-mgr-cephadm 2024-03-25T16:38:13.320 INFO:teuthology.orchestra.run.smithi001.stdout:Last metadata expiration check: 0:05:24 ago on Mon 25 Mar 2024 04:32:49 PM UTC. 2024-03-25T16:38:13.476 INFO:teuthology.orchestra.run.smithi001.stdout:Dependencies resolved. 2024-03-25T16:38:13.477 INFO:teuthology.orchestra.run.smithi001.stdout:======================================================================================== 2024-03-25T16:38:13.477 INFO:teuthology.orchestra.run.smithi001.stdout: Package Arch Version Repository Size 2024-03-25T16:38:13.477 INFO:teuthology.orchestra.run.smithi001.stdout:======================================================================================== 2024-03-25T16:38:13.477 INFO:teuthology.orchestra.run.smithi001.stdout:Installing: 2024-03-25T16:38:13.478 INFO:teuthology.orchestra.run.smithi001.stdout: ceph-mgr-cephadm noarch 2:19.0.0-1439.g5dcb190d.el9 ceph-noarch 138 k 2024-03-25T16:38:13.478 INFO:teuthology.orchestra.run.smithi001.stdout:Installing dependencies: 2024-03-25T16:38:13.478 INFO:teuthology.orchestra.run.smithi001.stdout: python3-asyncssh noarch 2.13.2-5.el9 epel 548 k 2024-03-25T16:38:13.478 INFO:teuthology.orchestra.run.smithi001.stdout: python3-babel noarch 2.9.1-2.el9 CentOS-AppStream 5.8 M 2024-03-25T16:38:13.478 INFO:teuthology.orchestra.run.smithi001.stdout: python3-jinja2 noarch 2.11.3-5.el9 appstream 248 k 2024-03-25T16:38:13.478 INFO:teuthology.orchestra.run.smithi001.stdout: python3-natsort noarch 7.1.1-5.el9 epel 58 k 2024-03-25T16:38:13.478 INFO:teuthology.orchestra.run.smithi001.stdout: python3-typing-extensions noarch 4.4.0-2.el9 epel 51 k 2024-03-25T16:38:13.478 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-25T16:38:13.478 INFO:teuthology.orchestra.run.smithi001.stdout:Transaction Summary 2024-03-25T16:38:13.478 INFO:teuthology.orchestra.run.smithi001.stdout:======================================================================================== 2024-03-25T16:38:13.478 INFO:teuthology.orchestra.run.smithi001.stdout:Install 6 Packages 2024-03-25T16:38:13.478 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-25T16:38:13.479 INFO:teuthology.orchestra.run.smithi001.stdout:Total download size: 6.8 M 2024-03-25T16:38:13.479 INFO:teuthology.orchestra.run.smithi001.stdout:Installed size: 33 M 2024-03-25T16:38:13.479 INFO:teuthology.orchestra.run.smithi001.stdout:Downloading Packages: 2024-03-25T16:38:13.715 INFO:teuthology.orchestra.run.smithi115.stdout:(2/2): ceph-volume-19.0.0-1439.g5dcb190d.el9.no 140 kB/s | 264 kB 00:01 2024-03-25T16:38:13.716 INFO:teuthology.orchestra.run.smithi115.stdout:-------------------------------------------------------------------------------- 2024-03-25T16:38:13.717 INFO:teuthology.orchestra.run.smithi115.stdout:Total 176 kB/s | 332 kB 00:01 2024-03-25T16:38:13.717 INFO:teuthology.orchestra.run.smithi115.stdout:Running transaction check 2024-03-25T16:38:13.726 INFO:teuthology.orchestra.run.smithi115.stdout:Transaction check succeeded. 2024-03-25T16:38:13.727 INFO:teuthology.orchestra.run.smithi115.stdout:Running transaction test 2024-03-25T16:38:13.768 INFO:teuthology.orchestra.run.smithi115.stdout:Transaction test succeeded. 2024-03-25T16:38:13.769 INFO:teuthology.orchestra.run.smithi115.stdout:Running transaction 2024-03-25T16:38:13.923 INFO:teuthology.orchestra.run.smithi115.stdout: Preparing : 1/1 2024-03-25T16:38:13.985 INFO:teuthology.orchestra.run.smithi001.stdout:(1/6): python3-jinja2-2.11.3-5.el9.noarch.rpm 1.0 MB/s | 248 kB 00:00 2024-03-25T16:38:14.085 INFO:teuthology.orchestra.run.smithi001.stdout:(2/6): python3-babel-2.9.1-2.el9.noarch.rpm 17 MB/s | 5.8 MB 00:00 2024-03-25T16:38:14.103 INFO:teuthology.orchestra.run.smithi115.stdout: Installing : python3-packaging-20.9-5.el9.noarch 1/2 2024-03-25T16:38:14.134 INFO:teuthology.orchestra.run.smithi115.stdout: Installing : ceph-volume-2:19.0.0-1439.g5dcb190d.el9.noarch 2/2 2024-03-25T16:38:14.149 INFO:teuthology.orchestra.run.smithi115.stdout: Running scriptlet: ceph-volume-2:19.0.0-1439.g5dcb190d.el9.noarch 2/2 2024-03-25T16:38:14.149 INFO:teuthology.orchestra.run.smithi115.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-25T16:38:14.150 INFO:teuthology.orchestra.run.smithi115.stdout:Invalid unit name "ceph-volume@*.service" escaped as "ceph-volume@\x2a.service". 2024-03-25T16:38:14.150 INFO:teuthology.orchestra.run.smithi115.stdout: 2024-03-25T16:38:14.444 INFO:teuthology.orchestra.run.smithi001.stdout:(3/6): python3-natsort-7.1.1-5.el9.noarch.rpm 161 kB/s | 58 kB 00:00 2024-03-25T16:38:14.552 INFO:teuthology.orchestra.run.smithi001.stdout:(4/6): python3-typing-extensions-4.4.0-2.el9.no 471 kB/s | 51 kB 00:00 2024-03-25T16:38:14.603 INFO:teuthology.orchestra.run.smithi001.stdout:(5/6): python3-asyncssh-2.13.2-5.el9.noarch.rpm 888 kB/s | 548 kB 00:00 2024-03-25T16:38:14.870 INFO:teuthology.orchestra.run.smithi115.stdout: Verifying : ceph-volume-2:19.0.0-1439.g5dcb190d.el9.noarch 1/2 2024-03-25T16:38:14.937 INFO:teuthology.orchestra.run.smithi001.stdout:(6/6): ceph-mgr-cephadm-19.0.0-1439.g5dcb190d.e 116 kB/s | 138 kB 00:01 2024-03-25T16:38:14.940 INFO:teuthology.orchestra.run.smithi001.stdout:-------------------------------------------------------------------------------- 2024-03-25T16:38:14.940 INFO:teuthology.orchestra.run.smithi001.stdout:Total 4.6 MB/s | 6.8 MB 00:01 2024-03-25T16:38:14.954 INFO:teuthology.orchestra.run.smithi001.stdout:Running transaction check 2024-03-25T16:38:14.965 INFO:teuthology.orchestra.run.smithi001.stdout:Transaction check succeeded. 2024-03-25T16:38:14.965 INFO:teuthology.orchestra.run.smithi001.stdout:Running transaction test 2024-03-25T16:38:15.057 INFO:teuthology.orchestra.run.smithi001.stdout:Transaction test succeeded. 2024-03-25T16:38:15.058 INFO:teuthology.orchestra.run.smithi001.stdout:Running transaction 2024-03-25T16:38:15.192 INFO:teuthology.orchestra.run.smithi115.stdout: Verifying : python3-packaging-20.9-5.el9.noarch 2/2 2024-03-25T16:38:15.192 INFO:teuthology.orchestra.run.smithi115.stdout: 2024-03-25T16:38:15.192 INFO:teuthology.orchestra.run.smithi115.stdout:Installed: 2024-03-25T16:38:15.193 INFO:teuthology.orchestra.run.smithi115.stdout: ceph-volume-2:19.0.0-1439.g5dcb190d.el9.noarch 2024-03-25T16:38:15.193 INFO:teuthology.orchestra.run.smithi115.stdout: python3-packaging-20.9-5.el9.noarch 2024-03-25T16:38:15.193 INFO:teuthology.orchestra.run.smithi115.stdout: 2024-03-25T16:38:15.193 INFO:teuthology.orchestra.run.smithi115.stdout:Complete! 2024-03-25T16:38:15.361 INFO:teuthology.orchestra.run.smithi001.stdout: Preparing : 1/1 2024-03-25T16:38:15.528 DEBUG:teuthology.orchestra.run.smithi115:> sudo yum -y install librados-devel 2024-03-25T16:38:15.547 INFO:teuthology.orchestra.run.smithi001.stdout: Installing : python3-typing-extensions-4.4.0-2.el9.noarch 1/6 2024-03-25T16:38:15.597 INFO:teuthology.orchestra.run.smithi001.stdout: Installing : python3-asyncssh-2.13.2-5.el9.noarch 2/6 2024-03-25T16:38:16.107 INFO:teuthology.orchestra.run.smithi115.stdout:Last metadata expiration check: 0:05:31 ago on Mon 25 Mar 2024 04:32:45 PM UTC. 2024-03-25T16:38:16.149 INFO:teuthology.orchestra.run.smithi001.stdout: Installing : python3-natsort-7.1.1-5.el9.noarch 3/6 2024-03-25T16:38:16.235 INFO:teuthology.orchestra.run.smithi001.stdout: Installing : python3-babel-2.9.1-2.el9.noarch 4/6 2024-03-25T16:38:16.256 INFO:teuthology.orchestra.run.smithi115.stdout:Dependencies resolved. 2024-03-25T16:38:16.257 INFO:teuthology.orchestra.run.smithi115.stdout:================================================================================ 2024-03-25T16:38:16.257 INFO:teuthology.orchestra.run.smithi115.stdout: Package Arch Version Repo Size 2024-03-25T16:38:16.257 INFO:teuthology.orchestra.run.smithi115.stdout:================================================================================ 2024-03-25T16:38:16.257 INFO:teuthology.orchestra.run.smithi115.stdout:Installing: 2024-03-25T16:38:16.258 INFO:teuthology.orchestra.run.smithi115.stdout: librados-devel x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 127 k 2024-03-25T16:38:16.258 INFO:teuthology.orchestra.run.smithi115.stdout: 2024-03-25T16:38:16.258 INFO:teuthology.orchestra.run.smithi115.stdout:Transaction Summary 2024-03-25T16:38:16.258 INFO:teuthology.orchestra.run.smithi115.stdout:================================================================================ 2024-03-25T16:38:16.258 INFO:teuthology.orchestra.run.smithi115.stdout:Install 1 Package 2024-03-25T16:38:16.258 INFO:teuthology.orchestra.run.smithi115.stdout: 2024-03-25T16:38:16.258 INFO:teuthology.orchestra.run.smithi115.stdout:Total download size: 127 k 2024-03-25T16:38:16.258 INFO:teuthology.orchestra.run.smithi115.stdout:Installed size: 456 k 2024-03-25T16:38:16.258 INFO:teuthology.orchestra.run.smithi115.stdout:Downloading Packages: 2024-03-25T16:38:16.304 INFO:teuthology.orchestra.run.smithi001.stdout: Installing : python3-jinja2-2.11.3-5.el9.noarch 5/6 2024-03-25T16:38:16.334 INFO:teuthology.orchestra.run.smithi001.stdout: Installing : ceph-mgr-cephadm-2:19.0.0-1439.g5dcb190d.el9.noarch 6/6 2024-03-25T16:38:17.187 INFO:teuthology.orchestra.run.smithi001.stdout: Running scriptlet: ceph-mgr-cephadm-2:19.0.0-1439.g5dcb190d.el9.noarch 6/6 2024-03-25T16:38:17.187 INFO:teuthology.orchestra.run.smithi001.stdout: Verifying : ceph-mgr-cephadm-2:19.0.0-1439.g5dcb190d.el9.noarch 1/6 2024-03-25T16:38:17.187 INFO:teuthology.orchestra.run.smithi001.stdout: Verifying : python3-babel-2.9.1-2.el9.noarch 2/6 2024-03-25T16:38:17.187 INFO:teuthology.orchestra.run.smithi001.stdout: Verifying : python3-jinja2-2.11.3-5.el9.noarch 3/6 2024-03-25T16:38:17.187 INFO:teuthology.orchestra.run.smithi001.stdout: Verifying : python3-asyncssh-2.13.2-5.el9.noarch 4/6 2024-03-25T16:38:17.187 INFO:teuthology.orchestra.run.smithi001.stdout: Verifying : python3-natsort-7.1.1-5.el9.noarch 5/6 2024-03-25T16:38:17.330 INFO:teuthology.orchestra.run.smithi115.stdout:librados-devel-19.0.0-1439.g5dcb190d.el9.x86_64 118 kB/s | 127 kB 00:01 2024-03-25T16:38:17.331 INFO:teuthology.orchestra.run.smithi115.stdout:-------------------------------------------------------------------------------- 2024-03-25T16:38:17.331 INFO:teuthology.orchestra.run.smithi115.stdout:Total 118 kB/s | 127 kB 00:01 2024-03-25T16:38:17.331 INFO:teuthology.orchestra.run.smithi115.stdout:Running transaction check 2024-03-25T16:38:17.338 INFO:teuthology.orchestra.run.smithi115.stdout:Transaction check succeeded. 2024-03-25T16:38:17.338 INFO:teuthology.orchestra.run.smithi115.stdout:Running transaction test 2024-03-25T16:38:17.398 INFO:teuthology.orchestra.run.smithi115.stdout:Transaction test succeeded. 2024-03-25T16:38:17.398 INFO:teuthology.orchestra.run.smithi115.stdout:Running transaction 2024-03-25T16:38:17.493 INFO:teuthology.orchestra.run.smithi001.stdout: Verifying : python3-typing-extensions-4.4.0-2.el9.noarch 6/6 2024-03-25T16:38:17.493 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-25T16:38:17.493 INFO:teuthology.orchestra.run.smithi001.stdout:Installed: 2024-03-25T16:38:17.493 INFO:teuthology.orchestra.run.smithi001.stdout: ceph-mgr-cephadm-2:19.0.0-1439.g5dcb190d.el9.noarch 2024-03-25T16:38:17.493 INFO:teuthology.orchestra.run.smithi001.stdout: python3-asyncssh-2.13.2-5.el9.noarch 2024-03-25T16:38:17.493 INFO:teuthology.orchestra.run.smithi001.stdout: python3-babel-2.9.1-2.el9.noarch 2024-03-25T16:38:17.493 INFO:teuthology.orchestra.run.smithi001.stdout: python3-jinja2-2.11.3-5.el9.noarch 2024-03-25T16:38:17.493 INFO:teuthology.orchestra.run.smithi001.stdout: python3-natsort-7.1.1-5.el9.noarch 2024-03-25T16:38:17.493 INFO:teuthology.orchestra.run.smithi001.stdout: python3-typing-extensions-4.4.0-2.el9.noarch 2024-03-25T16:38:17.493 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-25T16:38:17.493 INFO:teuthology.orchestra.run.smithi001.stdout:Complete! 2024-03-25T16:38:17.552 INFO:teuthology.orchestra.run.smithi115.stdout: Preparing : 1/1 2024-03-25T16:38:17.646 INFO:teuthology.orchestra.run.smithi115.stdout: Installing : librados-devel-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/1 2024-03-25T16:38:17.778 DEBUG:teuthology.orchestra.run.smithi001:> sudo yum -y install ceph-fuse 2024-03-25T16:38:18.053 INFO:teuthology.orchestra.run.smithi115.stdout: Running scriptlet: librados-devel-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/1 2024-03-25T16:38:18.333 INFO:teuthology.orchestra.run.smithi115.stdout: Verifying : librados-devel-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/1 2024-03-25T16:38:18.333 INFO:teuthology.orchestra.run.smithi115.stdout: 2024-03-25T16:38:18.333 INFO:teuthology.orchestra.run.smithi115.stdout:Installed: 2024-03-25T16:38:18.334 INFO:teuthology.orchestra.run.smithi115.stdout: librados-devel-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T16:38:18.334 INFO:teuthology.orchestra.run.smithi115.stdout: 2024-03-25T16:38:18.334 INFO:teuthology.orchestra.run.smithi115.stdout:Complete! 2024-03-25T16:38:18.354 INFO:teuthology.orchestra.run.smithi001.stdout:Last metadata expiration check: 0:05:29 ago on Mon 25 Mar 2024 04:32:49 PM UTC. 2024-03-25T16:38:18.510 INFO:teuthology.orchestra.run.smithi001.stdout:Dependencies resolved. 2024-03-25T16:38:18.511 INFO:teuthology.orchestra.run.smithi001.stdout:================================================================================ 2024-03-25T16:38:18.511 INFO:teuthology.orchestra.run.smithi001.stdout: Package Arch Version Repository Size 2024-03-25T16:38:18.511 INFO:teuthology.orchestra.run.smithi001.stdout:================================================================================ 2024-03-25T16:38:18.511 INFO:teuthology.orchestra.run.smithi001.stdout:Installing: 2024-03-25T16:38:18.511 INFO:teuthology.orchestra.run.smithi001.stdout: ceph-fuse x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 885 k 2024-03-25T16:38:18.511 INFO:teuthology.orchestra.run.smithi001.stdout:Installing dependencies: 2024-03-25T16:38:18.511 INFO:teuthology.orchestra.run.smithi001.stdout: fuse x86_64 2.9.9-15.el9 CentOS-BaseOS 79 k 2024-03-25T16:38:18.511 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-25T16:38:18.511 INFO:teuthology.orchestra.run.smithi001.stdout:Transaction Summary 2024-03-25T16:38:18.511 INFO:teuthology.orchestra.run.smithi001.stdout:================================================================================ 2024-03-25T16:38:18.511 INFO:teuthology.orchestra.run.smithi001.stdout:Install 2 Packages 2024-03-25T16:38:18.511 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-25T16:38:18.512 INFO:teuthology.orchestra.run.smithi001.stdout:Total download size: 964 k 2024-03-25T16:38:18.512 INFO:teuthology.orchestra.run.smithi001.stdout:Installed size: 2.8 M 2024-03-25T16:38:18.512 INFO:teuthology.orchestra.run.smithi001.stdout:Downloading Packages: 2024-03-25T16:38:18.581 DEBUG:teuthology.orchestra.run.smithi115:> sudo yum -y install libcephfs2 2024-03-25T16:38:18.621 INFO:teuthology.orchestra.run.smithi001.stdout:(1/2): fuse-2.9.9-15.el9.x86_64.rpm 721 kB/s | 79 kB 00:00 2024-03-25T16:38:19.161 INFO:teuthology.orchestra.run.smithi115.stdout:Last metadata expiration check: 0:05:34 ago on Mon 25 Mar 2024 04:32:45 PM UTC. 2024-03-25T16:38:19.262 INFO:teuthology.orchestra.run.smithi115.stdout:Package libcephfs2-2:19.0.0-1439.g5dcb190d.el9.x86_64 is already installed. 2024-03-25T16:38:19.318 INFO:teuthology.orchestra.run.smithi115.stdout:Dependencies resolved. 2024-03-25T16:38:19.319 INFO:teuthology.orchestra.run.smithi115.stdout:Nothing to do. 2024-03-25T16:38:19.319 INFO:teuthology.orchestra.run.smithi115.stdout:Complete! 2024-03-25T16:38:19.381 DEBUG:teuthology.orchestra.run.smithi115:> sudo yum -y install libcephfs-devel 2024-03-25T16:38:19.945 INFO:teuthology.orchestra.run.smithi115.stdout:Last metadata expiration check: 0:05:34 ago on Mon 25 Mar 2024 04:32:45 PM UTC. 2024-03-25T16:38:20.094 INFO:teuthology.orchestra.run.smithi115.stdout:Dependencies resolved. 2024-03-25T16:38:20.095 INFO:teuthology.orchestra.run.smithi115.stdout:================================================================================ 2024-03-25T16:38:20.095 INFO:teuthology.orchestra.run.smithi115.stdout: Package Arch Version Repo Size 2024-03-25T16:38:20.095 INFO:teuthology.orchestra.run.smithi115.stdout:================================================================================ 2024-03-25T16:38:20.095 INFO:teuthology.orchestra.run.smithi115.stdout:Installing: 2024-03-25T16:38:20.096 INFO:teuthology.orchestra.run.smithi115.stdout: libcephfs-devel x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 31 k 2024-03-25T16:38:20.096 INFO:teuthology.orchestra.run.smithi115.stdout: 2024-03-25T16:38:20.096 INFO:teuthology.orchestra.run.smithi115.stdout:Transaction Summary 2024-03-25T16:38:20.096 INFO:teuthology.orchestra.run.smithi115.stdout:================================================================================ 2024-03-25T16:38:20.096 INFO:teuthology.orchestra.run.smithi115.stdout:Install 1 Package 2024-03-25T16:38:20.096 INFO:teuthology.orchestra.run.smithi115.stdout: 2024-03-25T16:38:20.096 INFO:teuthology.orchestra.run.smithi115.stdout:Total download size: 31 k 2024-03-25T16:38:20.096 INFO:teuthology.orchestra.run.smithi115.stdout:Installed size: 140 k 2024-03-25T16:38:20.096 INFO:teuthology.orchestra.run.smithi115.stdout:Downloading Packages: 2024-03-25T16:38:20.742 INFO:teuthology.orchestra.run.smithi115.stdout:libcephfs-devel-19.0.0-1439.g5dcb190d.el9.x86_6 48 kB/s | 31 kB 00:00 2024-03-25T16:38:20.743 INFO:teuthology.orchestra.run.smithi115.stdout:-------------------------------------------------------------------------------- 2024-03-25T16:38:20.743 INFO:teuthology.orchestra.run.smithi115.stdout:Total 48 kB/s | 31 kB 00:00 2024-03-25T16:38:20.743 INFO:teuthology.orchestra.run.smithi115.stdout:Running transaction check 2024-03-25T16:38:20.746 INFO:teuthology.orchestra.run.smithi115.stdout:Transaction check succeeded. 2024-03-25T16:38:20.746 INFO:teuthology.orchestra.run.smithi115.stdout:Running transaction test 2024-03-25T16:38:20.780 INFO:teuthology.orchestra.run.smithi115.stdout:Transaction test succeeded. 2024-03-25T16:38:20.781 INFO:teuthology.orchestra.run.smithi115.stdout:Running transaction 2024-03-25T16:38:20.912 INFO:teuthology.orchestra.run.smithi115.stdout: Preparing : 1/1 2024-03-25T16:38:21.007 INFO:teuthology.orchestra.run.smithi115.stdout: Installing : libcephfs-devel-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/1 2024-03-25T16:38:21.407 INFO:teuthology.orchestra.run.smithi115.stdout: Running scriptlet: libcephfs-devel-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/1 2024-03-25T16:38:21.677 INFO:teuthology.orchestra.run.smithi115.stdout: Verifying : libcephfs-devel-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/1 2024-03-25T16:38:21.678 INFO:teuthology.orchestra.run.smithi115.stdout: 2024-03-25T16:38:21.678 INFO:teuthology.orchestra.run.smithi115.stdout:Installed: 2024-03-25T16:38:21.678 INFO:teuthology.orchestra.run.smithi115.stdout: libcephfs-devel-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T16:38:21.678 INFO:teuthology.orchestra.run.smithi115.stdout: 2024-03-25T16:38:21.678 INFO:teuthology.orchestra.run.smithi115.stdout:Complete! 2024-03-25T16:38:21.813 INFO:teuthology.orchestra.run.smithi001.stdout:(2/2): ceph-fuse-19.0.0-1439.g5dcb190d.el9.x86_ 268 kB/s | 885 kB 00:03 2024-03-25T16:38:21.814 INFO:teuthology.orchestra.run.smithi001.stdout:-------------------------------------------------------------------------------- 2024-03-25T16:38:21.815 INFO:teuthology.orchestra.run.smithi001.stdout:Total 292 kB/s | 964 kB 00:03 2024-03-25T16:38:21.815 INFO:teuthology.orchestra.run.smithi001.stdout:Running transaction check 2024-03-25T16:38:21.839 INFO:teuthology.orchestra.run.smithi001.stdout:Transaction check succeeded. 2024-03-25T16:38:21.839 INFO:teuthology.orchestra.run.smithi001.stdout:Running transaction test 2024-03-25T16:38:21.843 DEBUG:teuthology.orchestra.run.smithi115:> sudo yum -y install librados2 2024-03-25T16:38:21.918 INFO:teuthology.orchestra.run.smithi001.stdout:Transaction test succeeded. 2024-03-25T16:38:21.918 INFO:teuthology.orchestra.run.smithi001.stdout:Running transaction 2024-03-25T16:38:22.106 INFO:teuthology.orchestra.run.smithi001.stdout: Preparing : 1/1 2024-03-25T16:38:22.235 INFO:teuthology.orchestra.run.smithi001.stdout: Installing : fuse-2.9.9-15.el9.x86_64 1/2 2024-03-25T16:38:22.307 INFO:teuthology.orchestra.run.smithi001.stdout: Installing : ceph-fuse-2:19.0.0-1439.g5dcb190d.el9.x86_64 2/2 2024-03-25T16:38:22.407 INFO:teuthology.orchestra.run.smithi115.stdout:Last metadata expiration check: 0:05:37 ago on Mon 25 Mar 2024 04:32:45 PM UTC. 2024-03-25T16:38:22.505 INFO:teuthology.orchestra.run.smithi115.stdout:Package librados2-2:19.0.0-1439.g5dcb190d.el9.x86_64 is already installed. 2024-03-25T16:38:22.556 INFO:teuthology.orchestra.run.smithi115.stdout:Dependencies resolved. 2024-03-25T16:38:22.557 INFO:teuthology.orchestra.run.smithi115.stdout:Nothing to do. 2024-03-25T16:38:22.557 INFO:teuthology.orchestra.run.smithi115.stdout:Complete! 2024-03-25T16:38:22.610 DEBUG:teuthology.orchestra.run.smithi115:> sudo yum -y install librbd1 2024-03-25T16:38:23.057 INFO:teuthology.orchestra.run.smithi001.stdout: Running scriptlet: ceph-fuse-2:19.0.0-1439.g5dcb190d.el9.x86_64 2/2 2024-03-25T16:38:23.057 INFO:teuthology.orchestra.run.smithi001.stdout: Verifying : ceph-fuse-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/2 2024-03-25T16:38:23.173 INFO:teuthology.orchestra.run.smithi115.stdout:Last metadata expiration check: 0:05:38 ago on Mon 25 Mar 2024 04:32:45 PM UTC. 2024-03-25T16:38:23.271 INFO:teuthology.orchestra.run.smithi115.stdout:Package librbd1-2:19.0.0-1439.g5dcb190d.el9.x86_64 is already installed. 2024-03-25T16:38:23.322 INFO:teuthology.orchestra.run.smithi115.stdout:Dependencies resolved. 2024-03-25T16:38:23.323 INFO:teuthology.orchestra.run.smithi115.stdout:Nothing to do. 2024-03-25T16:38:23.323 INFO:teuthology.orchestra.run.smithi115.stdout:Complete! 2024-03-25T16:38:23.367 INFO:teuthology.orchestra.run.smithi001.stdout: Verifying : fuse-2.9.9-15.el9.x86_64 2/2 2024-03-25T16:38:23.367 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-25T16:38:23.367 INFO:teuthology.orchestra.run.smithi001.stdout:Installed: 2024-03-25T16:38:23.367 INFO:teuthology.orchestra.run.smithi001.stdout: ceph-fuse-2:19.0.0-1439.g5dcb190d.el9.x86_64 fuse-2.9.9-15.el9.x86_64 2024-03-25T16:38:23.367 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-25T16:38:23.367 INFO:teuthology.orchestra.run.smithi001.stdout:Complete! 2024-03-25T16:38:23.382 DEBUG:teuthology.orchestra.run.smithi115:> sudo yum -y install python3-rados 2024-03-25T16:38:23.653 DEBUG:teuthology.orchestra.run.smithi001:> sudo yum -y install ceph-volume 2024-03-25T16:38:23.945 INFO:teuthology.orchestra.run.smithi115.stdout:Last metadata expiration check: 0:05:38 ago on Mon 25 Mar 2024 04:32:45 PM UTC. 2024-03-25T16:38:24.042 INFO:teuthology.orchestra.run.smithi115.stdout:Package python3-rados-2:19.0.0-1439.g5dcb190d.el9.x86_64 is already installed. 2024-03-25T16:38:24.093 INFO:teuthology.orchestra.run.smithi115.stdout:Dependencies resolved. 2024-03-25T16:38:24.094 INFO:teuthology.orchestra.run.smithi115.stdout:Nothing to do. 2024-03-25T16:38:24.094 INFO:teuthology.orchestra.run.smithi115.stdout:Complete! 2024-03-25T16:38:24.146 DEBUG:teuthology.orchestra.run.smithi115:> sudo yum -y install python3-rgw 2024-03-25T16:38:24.237 INFO:teuthology.orchestra.run.smithi001.stdout:Last metadata expiration check: 0:05:35 ago on Mon 25 Mar 2024 04:32:49 PM UTC. 2024-03-25T16:38:24.393 INFO:teuthology.orchestra.run.smithi001.stdout:Dependencies resolved. 2024-03-25T16:38:24.393 INFO:teuthology.orchestra.run.smithi001.stdout:================================================================================ 2024-03-25T16:38:24.393 INFO:teuthology.orchestra.run.smithi001.stdout: Package Arch Version Repository Size 2024-03-25T16:38:24.393 INFO:teuthology.orchestra.run.smithi001.stdout:================================================================================ 2024-03-25T16:38:24.393 INFO:teuthology.orchestra.run.smithi001.stdout:Installing: 2024-03-25T16:38:24.394 INFO:teuthology.orchestra.run.smithi001.stdout: ceph-volume noarch 2:19.0.0-1439.g5dcb190d.el9 ceph-noarch 264 k 2024-03-25T16:38:24.394 INFO:teuthology.orchestra.run.smithi001.stdout:Installing dependencies: 2024-03-25T16:38:24.394 INFO:teuthology.orchestra.run.smithi001.stdout: python3-packaging noarch 20.9-5.el9 CentOS-AppStream 69 k 2024-03-25T16:38:24.394 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-25T16:38:24.394 INFO:teuthology.orchestra.run.smithi001.stdout:Transaction Summary 2024-03-25T16:38:24.394 INFO:teuthology.orchestra.run.smithi001.stdout:================================================================================ 2024-03-25T16:38:24.394 INFO:teuthology.orchestra.run.smithi001.stdout:Install 2 Packages 2024-03-25T16:38:24.394 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-25T16:38:24.395 INFO:teuthology.orchestra.run.smithi001.stdout:Total download size: 332 k 2024-03-25T16:38:24.395 INFO:teuthology.orchestra.run.smithi001.stdout:Installed size: 1.5 M 2024-03-25T16:38:24.395 INFO:teuthology.orchestra.run.smithi001.stdout:Downloading Packages: 2024-03-25T16:38:24.509 INFO:teuthology.orchestra.run.smithi001.stdout:(1/2): python3-packaging-20.9-5.el9.noarch.rpm 604 kB/s | 69 kB 00:00 2024-03-25T16:38:24.706 INFO:teuthology.orchestra.run.smithi115.stdout:Last metadata expiration check: 0:05:39 ago on Mon 25 Mar 2024 04:32:45 PM UTC. 2024-03-25T16:38:24.802 INFO:teuthology.orchestra.run.smithi115.stdout:Package python3-rgw-2:19.0.0-1439.g5dcb190d.el9.x86_64 is already installed. 2024-03-25T16:38:24.853 INFO:teuthology.orchestra.run.smithi115.stdout:Dependencies resolved. 2024-03-25T16:38:24.854 INFO:teuthology.orchestra.run.smithi115.stdout:Nothing to do. 2024-03-25T16:38:24.854 INFO:teuthology.orchestra.run.smithi115.stdout:Complete! 2024-03-25T16:38:24.914 DEBUG:teuthology.orchestra.run.smithi115:> sudo yum -y install python3-cephfs 2024-03-25T16:38:25.474 INFO:teuthology.orchestra.run.smithi115.stdout:Last metadata expiration check: 0:05:40 ago on Mon 25 Mar 2024 04:32:45 PM UTC. 2024-03-25T16:38:25.569 INFO:teuthology.orchestra.run.smithi115.stdout:Package python3-cephfs-2:19.0.0-1439.g5dcb190d.el9.x86_64 is already installed. 2024-03-25T16:38:25.620 INFO:teuthology.orchestra.run.smithi115.stdout:Dependencies resolved. 2024-03-25T16:38:25.621 INFO:teuthology.orchestra.run.smithi115.stdout:Nothing to do. 2024-03-25T16:38:25.621 INFO:teuthology.orchestra.run.smithi115.stdout:Complete! 2024-03-25T16:38:25.678 DEBUG:teuthology.orchestra.run.smithi115:> sudo yum -y install python3-rbd 2024-03-25T16:38:25.876 INFO:teuthology.orchestra.run.smithi001.stdout:(2/2): ceph-volume-19.0.0-1439.g5dcb190d.el9.no 178 kB/s | 264 kB 00:01 2024-03-25T16:38:25.877 INFO:teuthology.orchestra.run.smithi001.stdout:-------------------------------------------------------------------------------- 2024-03-25T16:38:25.877 INFO:teuthology.orchestra.run.smithi001.stdout:Total 224 kB/s | 332 kB 00:01 2024-03-25T16:38:25.878 INFO:teuthology.orchestra.run.smithi001.stdout:Running transaction check 2024-03-25T16:38:25.888 INFO:teuthology.orchestra.run.smithi001.stdout:Transaction check succeeded. 2024-03-25T16:38:25.888 INFO:teuthology.orchestra.run.smithi001.stdout:Running transaction test 2024-03-25T16:38:25.932 INFO:teuthology.orchestra.run.smithi001.stdout:Transaction test succeeded. 2024-03-25T16:38:25.932 INFO:teuthology.orchestra.run.smithi001.stdout:Running transaction 2024-03-25T16:38:26.085 INFO:teuthology.orchestra.run.smithi001.stdout: Preparing : 1/1 2024-03-25T16:38:26.240 INFO:teuthology.orchestra.run.smithi115.stdout:Last metadata expiration check: 0:05:41 ago on Mon 25 Mar 2024 04:32:45 PM UTC. 2024-03-25T16:38:26.285 INFO:teuthology.orchestra.run.smithi001.stdout: Installing : python3-packaging-20.9-5.el9.noarch 1/2 2024-03-25T16:38:26.322 INFO:teuthology.orchestra.run.smithi001.stdout: Installing : ceph-volume-2:19.0.0-1439.g5dcb190d.el9.noarch 2/2 2024-03-25T16:38:26.336 INFO:teuthology.orchestra.run.smithi115.stdout:Package python3-rbd-2:19.0.0-1439.g5dcb190d.el9.x86_64 is already installed. 2024-03-25T16:38:26.337 INFO:teuthology.orchestra.run.smithi001.stdout: Running scriptlet: ceph-volume-2:19.0.0-1439.g5dcb190d.el9.noarch 2/2 2024-03-25T16:38:26.337 INFO:teuthology.orchestra.run.smithi001.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-25T16:38:26.337 INFO:teuthology.orchestra.run.smithi001.stdout:Invalid unit name "ceph-volume@*.service" escaped as "ceph-volume@\x2a.service". 2024-03-25T16:38:26.337 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-25T16:38:26.387 INFO:teuthology.orchestra.run.smithi115.stdout:Dependencies resolved. 2024-03-25T16:38:26.388 INFO:teuthology.orchestra.run.smithi115.stdout:Nothing to do. 2024-03-25T16:38:26.388 INFO:teuthology.orchestra.run.smithi115.stdout:Complete! 2024-03-25T16:38:26.446 DEBUG:teuthology.orchestra.run.smithi115:> sudo yum -y install rbd-fuse 2024-03-25T16:38:27.010 INFO:teuthology.orchestra.run.smithi115.stdout:Last metadata expiration check: 0:05:42 ago on Mon 25 Mar 2024 04:32:45 PM UTC. 2024-03-25T16:38:27.023 INFO:teuthology.orchestra.run.smithi001.stdout: Verifying : ceph-volume-2:19.0.0-1439.g5dcb190d.el9.noarch 1/2 2024-03-25T16:38:27.158 INFO:teuthology.orchestra.run.smithi115.stdout:Dependencies resolved. 2024-03-25T16:38:27.159 INFO:teuthology.orchestra.run.smithi115.stdout:================================================================================ 2024-03-25T16:38:27.159 INFO:teuthology.orchestra.run.smithi115.stdout: Package Architecture Version Repository Size 2024-03-25T16:38:27.159 INFO:teuthology.orchestra.run.smithi115.stdout:================================================================================ 2024-03-25T16:38:27.159 INFO:teuthology.orchestra.run.smithi115.stdout:Installing: 2024-03-25T16:38:27.159 INFO:teuthology.orchestra.run.smithi115.stdout: rbd-fuse x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 85 k 2024-03-25T16:38:27.159 INFO:teuthology.orchestra.run.smithi115.stdout: 2024-03-25T16:38:27.159 INFO:teuthology.orchestra.run.smithi115.stdout:Transaction Summary 2024-03-25T16:38:27.159 INFO:teuthology.orchestra.run.smithi115.stdout:================================================================================ 2024-03-25T16:38:27.159 INFO:teuthology.orchestra.run.smithi115.stdout:Install 1 Package 2024-03-25T16:38:27.160 INFO:teuthology.orchestra.run.smithi115.stdout: 2024-03-25T16:38:27.160 INFO:teuthology.orchestra.run.smithi115.stdout:Total download size: 85 k 2024-03-25T16:38:27.160 INFO:teuthology.orchestra.run.smithi115.stdout:Installed size: 231 k 2024-03-25T16:38:27.160 INFO:teuthology.orchestra.run.smithi115.stdout:Downloading Packages: 2024-03-25T16:38:27.373 INFO:teuthology.orchestra.run.smithi001.stdout: Verifying : python3-packaging-20.9-5.el9.noarch 2/2 2024-03-25T16:38:27.373 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-25T16:38:27.373 INFO:teuthology.orchestra.run.smithi001.stdout:Installed: 2024-03-25T16:38:27.373 INFO:teuthology.orchestra.run.smithi001.stdout: ceph-volume-2:19.0.0-1439.g5dcb190d.el9.noarch 2024-03-25T16:38:27.373 INFO:teuthology.orchestra.run.smithi001.stdout: python3-packaging-20.9-5.el9.noarch 2024-03-25T16:38:27.374 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-25T16:38:27.374 INFO:teuthology.orchestra.run.smithi001.stdout:Complete! 2024-03-25T16:38:27.764 DEBUG:teuthology.orchestra.run.smithi001:> sudo yum -y install librados-devel 2024-03-25T16:38:28.061 INFO:teuthology.orchestra.run.smithi115.stdout:rbd-fuse-19.0.0-1439.g5dcb190d.el9.x86_64.rpm 95 kB/s | 85 kB 00:00 2024-03-25T16:38:28.061 INFO:teuthology.orchestra.run.smithi115.stdout:-------------------------------------------------------------------------------- 2024-03-25T16:38:28.062 INFO:teuthology.orchestra.run.smithi115.stdout:Total 94 kB/s | 85 kB 00:00 2024-03-25T16:38:28.062 INFO:teuthology.orchestra.run.smithi115.stdout:Running transaction check 2024-03-25T16:38:28.070 INFO:teuthology.orchestra.run.smithi115.stdout:Transaction check succeeded. 2024-03-25T16:38:28.070 INFO:teuthology.orchestra.run.smithi115.stdout:Running transaction test 2024-03-25T16:38:28.128 INFO:teuthology.orchestra.run.smithi115.stdout:Transaction test succeeded. 2024-03-25T16:38:28.129 INFO:teuthology.orchestra.run.smithi115.stdout:Running transaction 2024-03-25T16:38:28.263 INFO:teuthology.orchestra.run.smithi115.stdout: Preparing : 1/1 2024-03-25T16:38:28.347 INFO:teuthology.orchestra.run.smithi001.stdout:Last metadata expiration check: 0:05:39 ago on Mon 25 Mar 2024 04:32:49 PM UTC. 2024-03-25T16:38:28.350 INFO:teuthology.orchestra.run.smithi115.stdout: Installing : rbd-fuse-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/1 2024-03-25T16:38:28.505 INFO:teuthology.orchestra.run.smithi001.stdout:Dependencies resolved. 2024-03-25T16:38:28.506 INFO:teuthology.orchestra.run.smithi001.stdout:================================================================================ 2024-03-25T16:38:28.506 INFO:teuthology.orchestra.run.smithi001.stdout: Package Arch Version Repo Size 2024-03-25T16:38:28.506 INFO:teuthology.orchestra.run.smithi001.stdout:================================================================================ 2024-03-25T16:38:28.506 INFO:teuthology.orchestra.run.smithi001.stdout:Installing: 2024-03-25T16:38:28.506 INFO:teuthology.orchestra.run.smithi001.stdout: librados-devel x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 127 k 2024-03-25T16:38:28.507 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-25T16:38:28.507 INFO:teuthology.orchestra.run.smithi001.stdout:Transaction Summary 2024-03-25T16:38:28.507 INFO:teuthology.orchestra.run.smithi001.stdout:================================================================================ 2024-03-25T16:38:28.507 INFO:teuthology.orchestra.run.smithi001.stdout:Install 1 Package 2024-03-25T16:38:28.507 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-25T16:38:28.507 INFO:teuthology.orchestra.run.smithi001.stdout:Total download size: 127 k 2024-03-25T16:38:28.507 INFO:teuthology.orchestra.run.smithi001.stdout:Installed size: 456 k 2024-03-25T16:38:28.507 INFO:teuthology.orchestra.run.smithi001.stdout:Downloading Packages: 2024-03-25T16:38:28.732 INFO:teuthology.orchestra.run.smithi115.stdout: Running scriptlet: rbd-fuse-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/1 2024-03-25T16:38:29.015 INFO:teuthology.orchestra.run.smithi115.stdout: Verifying : rbd-fuse-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/1 2024-03-25T16:38:29.015 INFO:teuthology.orchestra.run.smithi115.stdout: 2024-03-25T16:38:29.015 INFO:teuthology.orchestra.run.smithi115.stdout:Installed: 2024-03-25T16:38:29.015 INFO:teuthology.orchestra.run.smithi115.stdout: rbd-fuse-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T16:38:29.015 INFO:teuthology.orchestra.run.smithi115.stdout: 2024-03-25T16:38:29.015 INFO:teuthology.orchestra.run.smithi115.stdout:Complete! 2024-03-25T16:38:29.261 DEBUG:teuthology.orchestra.run.smithi115:> sudo yum -y install rbd-mirror 2024-03-25T16:38:29.349 INFO:teuthology.orchestra.run.smithi001.stdout:librados-devel-19.0.0-1439.g5dcb190d.el9.x86_64 150 kB/s | 127 kB 00:00 2024-03-25T16:38:29.350 INFO:teuthology.orchestra.run.smithi001.stdout:-------------------------------------------------------------------------------- 2024-03-25T16:38:29.350 INFO:teuthology.orchestra.run.smithi001.stdout:Total 150 kB/s | 127 kB 00:00 2024-03-25T16:38:29.350 INFO:teuthology.orchestra.run.smithi001.stdout:Running transaction check 2024-03-25T16:38:29.357 INFO:teuthology.orchestra.run.smithi001.stdout:Transaction check succeeded. 2024-03-25T16:38:29.357 INFO:teuthology.orchestra.run.smithi001.stdout:Running transaction test 2024-03-25T16:38:29.419 INFO:teuthology.orchestra.run.smithi001.stdout:Transaction test succeeded. 2024-03-25T16:38:29.419 INFO:teuthology.orchestra.run.smithi001.stdout:Running transaction 2024-03-25T16:38:29.572 INFO:teuthology.orchestra.run.smithi001.stdout: Preparing : 1/1 2024-03-25T16:38:29.692 INFO:teuthology.orchestra.run.smithi001.stdout: Installing : librados-devel-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/1 2024-03-25T16:38:29.840 INFO:teuthology.orchestra.run.smithi115.stdout:Last metadata expiration check: 0:05:44 ago on Mon 25 Mar 2024 04:32:45 PM UTC. 2024-03-25T16:38:29.997 INFO:teuthology.orchestra.run.smithi115.stdout:Dependencies resolved. 2024-03-25T16:38:29.998 INFO:teuthology.orchestra.run.smithi115.stdout:================================================================================ 2024-03-25T16:38:29.998 INFO:teuthology.orchestra.run.smithi115.stdout: Package Arch Version Repo Size 2024-03-25T16:38:29.998 INFO:teuthology.orchestra.run.smithi115.stdout:================================================================================ 2024-03-25T16:38:29.998 INFO:teuthology.orchestra.run.smithi115.stdout:Installing: 2024-03-25T16:38:29.998 INFO:teuthology.orchestra.run.smithi115.stdout: rbd-mirror x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 3.1 M 2024-03-25T16:38:29.998 INFO:teuthology.orchestra.run.smithi115.stdout: 2024-03-25T16:38:29.998 INFO:teuthology.orchestra.run.smithi115.stdout:Transaction Summary 2024-03-25T16:38:29.998 INFO:teuthology.orchestra.run.smithi115.stdout:================================================================================ 2024-03-25T16:38:29.998 INFO:teuthology.orchestra.run.smithi115.stdout:Install 1 Package 2024-03-25T16:38:29.998 INFO:teuthology.orchestra.run.smithi115.stdout: 2024-03-25T16:38:29.998 INFO:teuthology.orchestra.run.smithi115.stdout:Total download size: 3.1 M 2024-03-25T16:38:29.999 INFO:teuthology.orchestra.run.smithi115.stdout:Installed size: 13 M 2024-03-25T16:38:29.999 INFO:teuthology.orchestra.run.smithi115.stdout:Downloading Packages: 2024-03-25T16:38:30.110 INFO:teuthology.orchestra.run.smithi001.stdout: Running scriptlet: librados-devel-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/1 2024-03-25T16:38:30.382 INFO:teuthology.orchestra.run.smithi001.stdout: Verifying : librados-devel-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/1 2024-03-25T16:38:30.382 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-25T16:38:30.382 INFO:teuthology.orchestra.run.smithi001.stdout:Installed: 2024-03-25T16:38:30.382 INFO:teuthology.orchestra.run.smithi001.stdout: librados-devel-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T16:38:30.382 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-25T16:38:30.382 INFO:teuthology.orchestra.run.smithi001.stdout:Complete! 2024-03-25T16:38:30.606 DEBUG:teuthology.orchestra.run.smithi001:> sudo yum -y install libcephfs2 2024-03-25T16:38:31.190 INFO:teuthology.orchestra.run.smithi001.stdout:Last metadata expiration check: 0:05:42 ago on Mon 25 Mar 2024 04:32:49 PM UTC. 2024-03-25T16:38:31.290 INFO:teuthology.orchestra.run.smithi001.stdout:Package libcephfs2-2:19.0.0-1439.g5dcb190d.el9.x86_64 is already installed. 2024-03-25T16:38:31.347 INFO:teuthology.orchestra.run.smithi001.stdout:Dependencies resolved. 2024-03-25T16:38:31.347 INFO:teuthology.orchestra.run.smithi001.stdout:Nothing to do. 2024-03-25T16:38:31.348 INFO:teuthology.orchestra.run.smithi001.stdout:Complete! 2024-03-25T16:38:31.416 DEBUG:teuthology.orchestra.run.smithi001:> sudo yum -y install libcephfs-devel 2024-03-25T16:38:31.998 INFO:teuthology.orchestra.run.smithi001.stdout:Last metadata expiration check: 0:05:42 ago on Mon 25 Mar 2024 04:32:49 PM UTC. 2024-03-25T16:38:32.155 INFO:teuthology.orchestra.run.smithi001.stdout:Dependencies resolved. 2024-03-25T16:38:32.156 INFO:teuthology.orchestra.run.smithi001.stdout:================================================================================ 2024-03-25T16:38:32.156 INFO:teuthology.orchestra.run.smithi001.stdout: Package Arch Version Repo Size 2024-03-25T16:38:32.156 INFO:teuthology.orchestra.run.smithi001.stdout:================================================================================ 2024-03-25T16:38:32.156 INFO:teuthology.orchestra.run.smithi001.stdout:Installing: 2024-03-25T16:38:32.156 INFO:teuthology.orchestra.run.smithi001.stdout: libcephfs-devel x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 31 k 2024-03-25T16:38:32.156 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-25T16:38:32.156 INFO:teuthology.orchestra.run.smithi001.stdout:Transaction Summary 2024-03-25T16:38:32.156 INFO:teuthology.orchestra.run.smithi001.stdout:================================================================================ 2024-03-25T16:38:32.157 INFO:teuthology.orchestra.run.smithi001.stdout:Install 1 Package 2024-03-25T16:38:32.157 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-25T16:38:32.157 INFO:teuthology.orchestra.run.smithi001.stdout:Total download size: 31 k 2024-03-25T16:38:32.157 INFO:teuthology.orchestra.run.smithi001.stdout:Installed size: 140 k 2024-03-25T16:38:32.157 INFO:teuthology.orchestra.run.smithi001.stdout:Downloading Packages: 2024-03-25T16:38:32.619 INFO:teuthology.orchestra.run.smithi001.stdout:libcephfs-devel-19.0.0-1439.g5dcb190d.el9.x86_6 67 kB/s | 31 kB 00:00 2024-03-25T16:38:32.620 INFO:teuthology.orchestra.run.smithi001.stdout:-------------------------------------------------------------------------------- 2024-03-25T16:38:32.620 INFO:teuthology.orchestra.run.smithi001.stdout:Total 67 kB/s | 31 kB 00:00 2024-03-25T16:38:32.620 INFO:teuthology.orchestra.run.smithi001.stdout:Running transaction check 2024-03-25T16:38:32.624 INFO:teuthology.orchestra.run.smithi001.stdout:Transaction check succeeded. 2024-03-25T16:38:32.624 INFO:teuthology.orchestra.run.smithi001.stdout:Running transaction test 2024-03-25T16:38:32.653 INFO:teuthology.orchestra.run.smithi001.stdout:Transaction test succeeded. 2024-03-25T16:38:32.654 INFO:teuthology.orchestra.run.smithi001.stdout:Running transaction 2024-03-25T16:38:32.782 INFO:teuthology.orchestra.run.smithi001.stdout: Preparing : 1/1 2024-03-25T16:38:32.896 INFO:teuthology.orchestra.run.smithi001.stdout: Installing : libcephfs-devel-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/1 2024-03-25T16:38:33.281 INFO:teuthology.orchestra.run.smithi001.stdout: Running scriptlet: libcephfs-devel-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/1 2024-03-25T16:38:33.546 INFO:teuthology.orchestra.run.smithi001.stdout: Verifying : libcephfs-devel-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/1 2024-03-25T16:38:33.546 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-25T16:38:33.546 INFO:teuthology.orchestra.run.smithi001.stdout:Installed: 2024-03-25T16:38:33.546 INFO:teuthology.orchestra.run.smithi001.stdout: libcephfs-devel-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T16:38:33.546 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-25T16:38:33.546 INFO:teuthology.orchestra.run.smithi001.stdout:Complete! 2024-03-25T16:38:33.730 DEBUG:teuthology.orchestra.run.smithi001:> sudo yum -y install librados2 2024-03-25T16:38:34.312 INFO:teuthology.orchestra.run.smithi001.stdout:Last metadata expiration check: 0:05:45 ago on Mon 25 Mar 2024 04:32:49 PM UTC. 2024-03-25T16:38:34.411 INFO:teuthology.orchestra.run.smithi001.stdout:Package librados2-2:19.0.0-1439.g5dcb190d.el9.x86_64 is already installed. 2024-03-25T16:38:34.466 INFO:teuthology.orchestra.run.smithi001.stdout:Dependencies resolved. 2024-03-25T16:38:34.467 INFO:teuthology.orchestra.run.smithi001.stdout:Nothing to do. 2024-03-25T16:38:34.467 INFO:teuthology.orchestra.run.smithi001.stdout:Complete! 2024-03-25T16:38:34.525 DEBUG:teuthology.orchestra.run.smithi001:> sudo yum -y install librbd1 2024-03-25T16:38:35.102 INFO:teuthology.orchestra.run.smithi001.stdout:Last metadata expiration check: 0:05:46 ago on Mon 25 Mar 2024 04:32:49 PM UTC. 2024-03-25T16:38:35.206 INFO:teuthology.orchestra.run.smithi001.stdout:Package librbd1-2:19.0.0-1439.g5dcb190d.el9.x86_64 is already installed. 2024-03-25T16:38:35.262 INFO:teuthology.orchestra.run.smithi001.stdout:Dependencies resolved. 2024-03-25T16:38:35.263 INFO:teuthology.orchestra.run.smithi001.stdout:Nothing to do. 2024-03-25T16:38:35.263 INFO:teuthology.orchestra.run.smithi001.stdout:Complete! 2024-03-25T16:38:35.313 DEBUG:teuthology.orchestra.run.smithi001:> sudo yum -y install python3-rados 2024-03-25T16:38:35.914 INFO:teuthology.orchestra.run.smithi001.stdout:Last metadata expiration check: 0:05:46 ago on Mon 25 Mar 2024 04:32:49 PM UTC. 2024-03-25T16:38:36.014 INFO:teuthology.orchestra.run.smithi001.stdout:Package python3-rados-2:19.0.0-1439.g5dcb190d.el9.x86_64 is already installed. 2024-03-25T16:38:36.068 INFO:teuthology.orchestra.run.smithi001.stdout:Dependencies resolved. 2024-03-25T16:38:36.069 INFO:teuthology.orchestra.run.smithi001.stdout:Nothing to do. 2024-03-25T16:38:36.069 INFO:teuthology.orchestra.run.smithi001.stdout:Complete! 2024-03-25T16:38:36.122 DEBUG:teuthology.orchestra.run.smithi001:> sudo yum -y install python3-rgw 2024-03-25T16:38:36.622 INFO:teuthology.orchestra.run.smithi115.stdout:rbd-mirror-19.0.0-1439.g5dcb190d.el9.x86_64.rpm 477 kB/s | 3.1 MB 00:06 2024-03-25T16:38:36.623 INFO:teuthology.orchestra.run.smithi115.stdout:-------------------------------------------------------------------------------- 2024-03-25T16:38:36.623 INFO:teuthology.orchestra.run.smithi115.stdout:Total 477 kB/s | 3.1 MB 00:06 2024-03-25T16:38:36.623 INFO:teuthology.orchestra.run.smithi115.stdout:Running transaction check 2024-03-25T16:38:36.635 INFO:teuthology.orchestra.run.smithi115.stdout:Transaction check succeeded. 2024-03-25T16:38:36.635 INFO:teuthology.orchestra.run.smithi115.stdout:Running transaction test 2024-03-25T16:38:36.706 INFO:teuthology.orchestra.run.smithi001.stdout:Last metadata expiration check: 0:05:47 ago on Mon 25 Mar 2024 04:32:49 PM UTC. 2024-03-25T16:38:36.710 INFO:teuthology.orchestra.run.smithi115.stdout:Transaction test succeeded. 2024-03-25T16:38:36.710 INFO:teuthology.orchestra.run.smithi115.stdout:Running transaction 2024-03-25T16:38:36.806 INFO:teuthology.orchestra.run.smithi001.stdout:Package python3-rgw-2:19.0.0-1439.g5dcb190d.el9.x86_64 is already installed. 2024-03-25T16:38:36.861 INFO:teuthology.orchestra.run.smithi001.stdout:Dependencies resolved. 2024-03-25T16:38:36.862 INFO:teuthology.orchestra.run.smithi001.stdout:Nothing to do. 2024-03-25T16:38:36.862 INFO:teuthology.orchestra.run.smithi001.stdout:Complete! 2024-03-25T16:38:36.914 DEBUG:teuthology.orchestra.run.smithi001:> sudo yum -y install python3-cephfs 2024-03-25T16:38:37.146 INFO:teuthology.orchestra.run.smithi115.stdout: Preparing : 1/1 2024-03-25T16:38:37.222 INFO:teuthology.orchestra.run.smithi115.stdout: Installing : rbd-mirror-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/1 2024-03-25T16:38:37.248 INFO:teuthology.orchestra.run.smithi115.stdout: Running scriptlet: rbd-mirror-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/1 2024-03-25T16:38:37.248 INFO:teuthology.orchestra.run.smithi115.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-25T16:38:37.248 INFO:teuthology.orchestra.run.smithi115.stdout:Invalid unit name "ceph-rbd-mirror@*.service" escaped as "ceph-rbd-mirror@\x2a.service". 2024-03-25T16:38:37.248 INFO:teuthology.orchestra.run.smithi115.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-rbd-mirror.target → /usr/lib/systemd/system/ceph-rbd-mirror.target. 2024-03-25T16:38:37.248 INFO:teuthology.orchestra.run.smithi115.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-rbd-mirror.target → /usr/lib/systemd/system/ceph-rbd-mirror.target. 2024-03-25T16:38:37.248 INFO:teuthology.orchestra.run.smithi115.stdout: 2024-03-25T16:38:37.490 INFO:teuthology.orchestra.run.smithi001.stdout:Last metadata expiration check: 0:05:48 ago on Mon 25 Mar 2024 04:32:49 PM UTC. 2024-03-25T16:38:37.589 INFO:teuthology.orchestra.run.smithi001.stdout:Package python3-cephfs-2:19.0.0-1439.g5dcb190d.el9.x86_64 is already installed. 2024-03-25T16:38:37.643 INFO:teuthology.orchestra.run.smithi001.stdout:Dependencies resolved. 2024-03-25T16:38:37.644 INFO:teuthology.orchestra.run.smithi001.stdout:Nothing to do. 2024-03-25T16:38:37.645 INFO:teuthology.orchestra.run.smithi001.stdout:Complete! 2024-03-25T16:38:37.695 DEBUG:teuthology.orchestra.run.smithi001:> sudo yum -y install python3-rbd 2024-03-25T16:38:38.272 INFO:teuthology.orchestra.run.smithi001.stdout:Last metadata expiration check: 0:05:49 ago on Mon 25 Mar 2024 04:32:49 PM UTC. 2024-03-25T16:38:38.358 INFO:teuthology.orchestra.run.smithi115.stdout: Verifying : rbd-mirror-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/1 2024-03-25T16:38:38.358 INFO:teuthology.orchestra.run.smithi115.stdout: 2024-03-25T16:38:38.359 INFO:teuthology.orchestra.run.smithi115.stdout:Installed: 2024-03-25T16:38:38.359 INFO:teuthology.orchestra.run.smithi115.stdout: rbd-mirror-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T16:38:38.359 INFO:teuthology.orchestra.run.smithi115.stdout: 2024-03-25T16:38:38.359 INFO:teuthology.orchestra.run.smithi115.stdout:Complete! 2024-03-25T16:38:38.372 INFO:teuthology.orchestra.run.smithi001.stdout:Package python3-rbd-2:19.0.0-1439.g5dcb190d.el9.x86_64 is already installed. 2024-03-25T16:38:38.427 INFO:teuthology.orchestra.run.smithi001.stdout:Dependencies resolved. 2024-03-25T16:38:38.428 INFO:teuthology.orchestra.run.smithi001.stdout:Nothing to do. 2024-03-25T16:38:38.429 INFO:teuthology.orchestra.run.smithi001.stdout:Complete! 2024-03-25T16:38:38.482 DEBUG:teuthology.orchestra.run.smithi001:> sudo yum -y install rbd-fuse 2024-03-25T16:38:38.599 DEBUG:teuthology.orchestra.run.smithi115:> sudo yum -y install rbd-nbd 2024-03-25T16:38:39.058 INFO:teuthology.orchestra.run.smithi001.stdout:Last metadata expiration check: 0:05:50 ago on Mon 25 Mar 2024 04:32:49 PM UTC. 2024-03-25T16:38:39.180 INFO:teuthology.orchestra.run.smithi115.stdout:Last metadata expiration check: 0:05:54 ago on Mon 25 Mar 2024 04:32:45 PM UTC. 2024-03-25T16:38:39.213 INFO:teuthology.orchestra.run.smithi001.stdout:Dependencies resolved. 2024-03-25T16:38:39.213 INFO:teuthology.orchestra.run.smithi001.stdout:================================================================================ 2024-03-25T16:38:39.213 INFO:teuthology.orchestra.run.smithi001.stdout: Package Architecture Version Repository Size 2024-03-25T16:38:39.214 INFO:teuthology.orchestra.run.smithi001.stdout:================================================================================ 2024-03-25T16:38:39.214 INFO:teuthology.orchestra.run.smithi001.stdout:Installing: 2024-03-25T16:38:39.214 INFO:teuthology.orchestra.run.smithi001.stdout: rbd-fuse x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 85 k 2024-03-25T16:38:39.214 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-25T16:38:39.214 INFO:teuthology.orchestra.run.smithi001.stdout:Transaction Summary 2024-03-25T16:38:39.214 INFO:teuthology.orchestra.run.smithi001.stdout:================================================================================ 2024-03-25T16:38:39.214 INFO:teuthology.orchestra.run.smithi001.stdout:Install 1 Package 2024-03-25T16:38:39.214 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-25T16:38:39.214 INFO:teuthology.orchestra.run.smithi001.stdout:Total download size: 85 k 2024-03-25T16:38:39.214 INFO:teuthology.orchestra.run.smithi001.stdout:Installed size: 231 k 2024-03-25T16:38:39.214 INFO:teuthology.orchestra.run.smithi001.stdout:Downloading Packages: 2024-03-25T16:38:39.330 INFO:teuthology.orchestra.run.smithi115.stdout:Dependencies resolved. 2024-03-25T16:38:39.331 INFO:teuthology.orchestra.run.smithi115.stdout:================================================================================ 2024-03-25T16:38:39.331 INFO:teuthology.orchestra.run.smithi115.stdout: Package Architecture Version Repository Size 2024-03-25T16:38:39.331 INFO:teuthology.orchestra.run.smithi115.stdout:================================================================================ 2024-03-25T16:38:39.331 INFO:teuthology.orchestra.run.smithi115.stdout:Installing: 2024-03-25T16:38:39.331 INFO:teuthology.orchestra.run.smithi115.stdout: rbd-nbd x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 171 k 2024-03-25T16:38:39.331 INFO:teuthology.orchestra.run.smithi115.stdout: 2024-03-25T16:38:39.331 INFO:teuthology.orchestra.run.smithi115.stdout:Transaction Summary 2024-03-25T16:38:39.331 INFO:teuthology.orchestra.run.smithi115.stdout:================================================================================ 2024-03-25T16:38:39.331 INFO:teuthology.orchestra.run.smithi115.stdout:Install 1 Package 2024-03-25T16:38:39.331 INFO:teuthology.orchestra.run.smithi115.stdout: 2024-03-25T16:38:39.332 INFO:teuthology.orchestra.run.smithi115.stdout:Total download size: 171 k 2024-03-25T16:38:39.332 INFO:teuthology.orchestra.run.smithi115.stdout:Installed size: 494 k 2024-03-25T16:38:39.332 INFO:teuthology.orchestra.run.smithi115.stdout:Downloading Packages: 2024-03-25T16:38:39.932 INFO:teuthology.orchestra.run.smithi001.stdout:rbd-fuse-19.0.0-1439.g5dcb190d.el9.x86_64.rpm 119 kB/s | 85 kB 00:00 2024-03-25T16:38:39.932 INFO:teuthology.orchestra.run.smithi001.stdout:-------------------------------------------------------------------------------- 2024-03-25T16:38:39.933 INFO:teuthology.orchestra.run.smithi001.stdout:Total 119 kB/s | 85 kB 00:00 2024-03-25T16:38:39.933 INFO:teuthology.orchestra.run.smithi001.stdout:Running transaction check 2024-03-25T16:38:39.941 INFO:teuthology.orchestra.run.smithi001.stdout:Transaction check succeeded. 2024-03-25T16:38:39.941 INFO:teuthology.orchestra.run.smithi001.stdout:Running transaction test 2024-03-25T16:38:40.002 INFO:teuthology.orchestra.run.smithi001.stdout:Transaction test succeeded. 2024-03-25T16:38:40.002 INFO:teuthology.orchestra.run.smithi001.stdout:Running transaction 2024-03-25T16:38:40.148 INFO:teuthology.orchestra.run.smithi001.stdout: Preparing : 1/1 2024-03-25T16:38:40.258 INFO:teuthology.orchestra.run.smithi001.stdout: Installing : rbd-fuse-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/1 2024-03-25T16:38:40.691 INFO:teuthology.orchestra.run.smithi001.stdout: Running scriptlet: rbd-fuse-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/1 2024-03-25T16:38:40.961 INFO:teuthology.orchestra.run.smithi001.stdout: Verifying : rbd-fuse-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/1 2024-03-25T16:38:40.962 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-25T16:38:40.962 INFO:teuthology.orchestra.run.smithi001.stdout:Installed: 2024-03-25T16:38:40.962 INFO:teuthology.orchestra.run.smithi001.stdout: rbd-fuse-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T16:38:40.962 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-25T16:38:40.962 INFO:teuthology.orchestra.run.smithi001.stdout:Complete! 2024-03-25T16:38:41.091 INFO:teuthology.orchestra.run.smithi115.stdout:rbd-nbd-19.0.0-1439.g5dcb190d.el9.x86_64.rpm 97 kB/s | 171 kB 00:01 2024-03-25T16:38:41.092 INFO:teuthology.orchestra.run.smithi115.stdout:-------------------------------------------------------------------------------- 2024-03-25T16:38:41.092 INFO:teuthology.orchestra.run.smithi115.stdout:Total 97 kB/s | 171 kB 00:01 2024-03-25T16:38:41.093 INFO:teuthology.orchestra.run.smithi115.stdout:Running transaction check 2024-03-25T16:38:41.101 INFO:teuthology.orchestra.run.smithi115.stdout:Transaction check succeeded. 2024-03-25T16:38:41.101 INFO:teuthology.orchestra.run.smithi115.stdout:Running transaction test 2024-03-25T16:38:41.158 INFO:teuthology.orchestra.run.smithi115.stdout:Transaction test succeeded. 2024-03-25T16:38:41.158 INFO:teuthology.orchestra.run.smithi115.stdout:Running transaction 2024-03-25T16:38:41.191 DEBUG:teuthology.orchestra.run.smithi001:> sudo yum -y install rbd-mirror 2024-03-25T16:38:41.310 INFO:teuthology.orchestra.run.smithi115.stdout: Preparing : 1/1 2024-03-25T16:38:41.397 INFO:teuthology.orchestra.run.smithi115.stdout: Installing : rbd-nbd-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/1 2024-03-25T16:38:41.768 INFO:teuthology.orchestra.run.smithi001.stdout:Last metadata expiration check: 0:05:52 ago on Mon 25 Mar 2024 04:32:49 PM UTC. 2024-03-25T16:38:41.817 INFO:teuthology.orchestra.run.smithi115.stdout: Running scriptlet: rbd-nbd-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/1 2024-03-25T16:38:41.923 INFO:teuthology.orchestra.run.smithi001.stdout:Dependencies resolved. 2024-03-25T16:38:41.923 INFO:teuthology.orchestra.run.smithi001.stdout:================================================================================ 2024-03-25T16:38:41.924 INFO:teuthology.orchestra.run.smithi001.stdout: Package Arch Version Repo Size 2024-03-25T16:38:41.924 INFO:teuthology.orchestra.run.smithi001.stdout:================================================================================ 2024-03-25T16:38:41.924 INFO:teuthology.orchestra.run.smithi001.stdout:Installing: 2024-03-25T16:38:41.924 INFO:teuthology.orchestra.run.smithi001.stdout: rbd-mirror x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 3.1 M 2024-03-25T16:38:41.924 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-25T16:38:41.924 INFO:teuthology.orchestra.run.smithi001.stdout:Transaction Summary 2024-03-25T16:38:41.924 INFO:teuthology.orchestra.run.smithi001.stdout:================================================================================ 2024-03-25T16:38:41.924 INFO:teuthology.orchestra.run.smithi001.stdout:Install 1 Package 2024-03-25T16:38:41.924 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-25T16:38:41.924 INFO:teuthology.orchestra.run.smithi001.stdout:Total download size: 3.1 M 2024-03-25T16:38:41.924 INFO:teuthology.orchestra.run.smithi001.stdout:Installed size: 13 M 2024-03-25T16:38:41.924 INFO:teuthology.orchestra.run.smithi001.stdout:Downloading Packages: 2024-03-25T16:38:42.125 INFO:teuthology.orchestra.run.smithi115.stdout: Verifying : rbd-nbd-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/1 2024-03-25T16:38:42.125 INFO:teuthology.orchestra.run.smithi115.stdout: 2024-03-25T16:38:42.125 INFO:teuthology.orchestra.run.smithi115.stdout:Installed: 2024-03-25T16:38:42.125 INFO:teuthology.orchestra.run.smithi115.stdout: rbd-nbd-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T16:38:42.125 INFO:teuthology.orchestra.run.smithi115.stdout: 2024-03-25T16:38:42.125 INFO:teuthology.orchestra.run.smithi115.stdout:Complete! 2024-03-25T16:38:42.368 DEBUG:teuthology.orchestra.run.smithi115:> sudo yum -y install ceph-base-debuginfo 2024-03-25T16:38:42.943 INFO:teuthology.orchestra.run.smithi115.stdout:Last metadata expiration check: 0:05:57 ago on Mon 25 Mar 2024 04:32:45 PM UTC. 2024-03-25T16:38:43.092 INFO:teuthology.orchestra.run.smithi115.stdout:Dependencies resolved. 2024-03-25T16:38:43.093 INFO:teuthology.orchestra.run.smithi115.stdout:================================================================================ 2024-03-25T16:38:43.093 INFO:teuthology.orchestra.run.smithi115.stdout: Package Arch Version Repo Size 2024-03-25T16:38:43.093 INFO:teuthology.orchestra.run.smithi115.stdout:================================================================================ 2024-03-25T16:38:43.093 INFO:teuthology.orchestra.run.smithi115.stdout:Installing: 2024-03-25T16:38:43.093 INFO:teuthology.orchestra.run.smithi115.stdout: ceph-base-debuginfo x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 68 M 2024-03-25T16:38:43.093 INFO:teuthology.orchestra.run.smithi115.stdout:Installing dependencies: 2024-03-25T16:38:43.093 INFO:teuthology.orchestra.run.smithi115.stdout: ceph-debuginfo x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 22 M 2024-03-25T16:38:43.093 INFO:teuthology.orchestra.run.smithi115.stdout: 2024-03-25T16:38:43.093 INFO:teuthology.orchestra.run.smithi115.stdout:Transaction Summary 2024-03-25T16:38:43.093 INFO:teuthology.orchestra.run.smithi115.stdout:================================================================================ 2024-03-25T16:38:43.093 INFO:teuthology.orchestra.run.smithi115.stdout:Install 2 Packages 2024-03-25T16:38:43.093 INFO:teuthology.orchestra.run.smithi115.stdout: 2024-03-25T16:38:43.094 INFO:teuthology.orchestra.run.smithi115.stdout:Total download size: 91 M 2024-03-25T16:38:43.095 INFO:teuthology.orchestra.run.smithi115.stdout:Installed size: 498 M 2024-03-25T16:38:43.095 INFO:teuthology.orchestra.run.smithi115.stdout:Downloading Packages: 2024-03-25T16:38:55.258 INFO:teuthology.orchestra.run.smithi001.stdout:rbd-mirror-19.0.0-1439.g5dcb190d.el9.x86_64.rpm 237 kB/s | 3.1 MB 00:13 2024-03-25T16:38:55.259 INFO:teuthology.orchestra.run.smithi001.stdout:-------------------------------------------------------------------------------- 2024-03-25T16:38:55.260 INFO:teuthology.orchestra.run.smithi001.stdout:Total 237 kB/s | 3.1 MB 00:13 2024-03-25T16:38:55.260 INFO:teuthology.orchestra.run.smithi001.stdout:Running transaction check 2024-03-25T16:38:55.272 INFO:teuthology.orchestra.run.smithi001.stdout:Transaction check succeeded. 2024-03-25T16:38:55.272 INFO:teuthology.orchestra.run.smithi001.stdout:Running transaction test 2024-03-25T16:38:55.351 INFO:teuthology.orchestra.run.smithi001.stdout:Transaction test succeeded. 2024-03-25T16:38:55.351 INFO:teuthology.orchestra.run.smithi001.stdout:Running transaction 2024-03-25T16:38:55.928 INFO:teuthology.orchestra.run.smithi001.stdout: Preparing : 1/1 2024-03-25T16:38:56.084 INFO:teuthology.orchestra.run.smithi001.stdout: Installing : rbd-mirror-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/1 2024-03-25T16:38:56.108 INFO:teuthology.orchestra.run.smithi001.stdout: Running scriptlet: rbd-mirror-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/1 2024-03-25T16:38:56.108 INFO:teuthology.orchestra.run.smithi001.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-25T16:38:56.108 INFO:teuthology.orchestra.run.smithi001.stdout:Invalid unit name "ceph-rbd-mirror@*.service" escaped as "ceph-rbd-mirror@\x2a.service". 2024-03-25T16:38:56.108 INFO:teuthology.orchestra.run.smithi001.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-rbd-mirror.target → /usr/lib/systemd/system/ceph-rbd-mirror.target. 2024-03-25T16:38:56.109 INFO:teuthology.orchestra.run.smithi001.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-rbd-mirror.target → /usr/lib/systemd/system/ceph-rbd-mirror.target. 2024-03-25T16:38:56.109 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-25T16:38:57.193 INFO:teuthology.orchestra.run.smithi001.stdout: Verifying : rbd-mirror-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/1 2024-03-25T16:38:57.193 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-25T16:38:57.193 INFO:teuthology.orchestra.run.smithi001.stdout:Installed: 2024-03-25T16:38:57.193 INFO:teuthology.orchestra.run.smithi001.stdout: rbd-mirror-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T16:38:57.193 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-25T16:38:57.194 INFO:teuthology.orchestra.run.smithi001.stdout:Complete! 2024-03-25T16:38:57.429 DEBUG:teuthology.orchestra.run.smithi001:> sudo yum -y install rbd-nbd 2024-03-25T16:38:58.007 INFO:teuthology.orchestra.run.smithi001.stdout:Last metadata expiration check: 0:06:09 ago on Mon 25 Mar 2024 04:32:49 PM UTC. 2024-03-25T16:38:58.161 INFO:teuthology.orchestra.run.smithi001.stdout:Dependencies resolved. 2024-03-25T16:38:58.162 INFO:teuthology.orchestra.run.smithi001.stdout:================================================================================ 2024-03-25T16:38:58.162 INFO:teuthology.orchestra.run.smithi001.stdout: Package Architecture Version Repository Size 2024-03-25T16:38:58.162 INFO:teuthology.orchestra.run.smithi001.stdout:================================================================================ 2024-03-25T16:38:58.163 INFO:teuthology.orchestra.run.smithi001.stdout:Installing: 2024-03-25T16:38:58.163 INFO:teuthology.orchestra.run.smithi001.stdout: rbd-nbd x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 171 k 2024-03-25T16:38:58.163 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-25T16:38:58.163 INFO:teuthology.orchestra.run.smithi001.stdout:Transaction Summary 2024-03-25T16:38:58.163 INFO:teuthology.orchestra.run.smithi001.stdout:================================================================================ 2024-03-25T16:38:58.163 INFO:teuthology.orchestra.run.smithi001.stdout:Install 1 Package 2024-03-25T16:38:58.163 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-25T16:38:58.163 INFO:teuthology.orchestra.run.smithi001.stdout:Total download size: 171 k 2024-03-25T16:38:58.163 INFO:teuthology.orchestra.run.smithi001.stdout:Installed size: 494 k 2024-03-25T16:38:58.163 INFO:teuthology.orchestra.run.smithi001.stdout:Downloading Packages: 2024-03-25T16:38:59.712 INFO:teuthology.orchestra.run.smithi001.stdout:rbd-nbd-19.0.0-1439.g5dcb190d.el9.x86_64.rpm 111 kB/s | 171 kB 00:01 2024-03-25T16:38:59.712 INFO:teuthology.orchestra.run.smithi001.stdout:-------------------------------------------------------------------------------- 2024-03-25T16:38:59.712 INFO:teuthology.orchestra.run.smithi001.stdout:Total 111 kB/s | 171 kB 00:01 2024-03-25T16:38:59.713 INFO:teuthology.orchestra.run.smithi001.stdout:Running transaction check 2024-03-25T16:38:59.723 INFO:teuthology.orchestra.run.smithi001.stdout:Transaction check succeeded. 2024-03-25T16:38:59.723 INFO:teuthology.orchestra.run.smithi001.stdout:Running transaction test 2024-03-25T16:38:59.786 INFO:teuthology.orchestra.run.smithi001.stdout:Transaction test succeeded. 2024-03-25T16:38:59.787 INFO:teuthology.orchestra.run.smithi001.stdout:Running transaction 2024-03-25T16:38:59.946 INFO:teuthology.orchestra.run.smithi001.stdout: Preparing : 1/1 2024-03-25T16:39:00.046 INFO:teuthology.orchestra.run.smithi001.stdout: Installing : rbd-nbd-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/1 2024-03-25T16:39:00.453 INFO:teuthology.orchestra.run.smithi001.stdout: Running scriptlet: rbd-nbd-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/1 2024-03-25T16:39:00.727 INFO:teuthology.orchestra.run.smithi001.stdout: Verifying : rbd-nbd-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/1 2024-03-25T16:39:00.727 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-25T16:39:00.727 INFO:teuthology.orchestra.run.smithi001.stdout:Installed: 2024-03-25T16:39:00.727 INFO:teuthology.orchestra.run.smithi001.stdout: rbd-nbd-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T16:39:00.727 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-25T16:39:00.727 INFO:teuthology.orchestra.run.smithi001.stdout:Complete! 2024-03-25T16:39:00.959 DEBUG:teuthology.orchestra.run.smithi001:> sudo yum -y install ceph-base-debuginfo 2024-03-25T16:39:01.534 INFO:teuthology.orchestra.run.smithi001.stdout:Last metadata expiration check: 0:06:12 ago on Mon 25 Mar 2024 04:32:49 PM UTC. 2024-03-25T16:39:01.690 INFO:teuthology.orchestra.run.smithi001.stdout:Dependencies resolved. 2024-03-25T16:39:01.691 INFO:teuthology.orchestra.run.smithi001.stdout:================================================================================ 2024-03-25T16:39:01.691 INFO:teuthology.orchestra.run.smithi001.stdout: Package Arch Version Repo Size 2024-03-25T16:39:01.692 INFO:teuthology.orchestra.run.smithi001.stdout:================================================================================ 2024-03-25T16:39:01.692 INFO:teuthology.orchestra.run.smithi001.stdout:Installing: 2024-03-25T16:39:01.692 INFO:teuthology.orchestra.run.smithi001.stdout: ceph-base-debuginfo x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 68 M 2024-03-25T16:39:01.692 INFO:teuthology.orchestra.run.smithi001.stdout:Installing dependencies: 2024-03-25T16:39:01.692 INFO:teuthology.orchestra.run.smithi001.stdout: ceph-debuginfo x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 22 M 2024-03-25T16:39:01.692 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-25T16:39:01.693 INFO:teuthology.orchestra.run.smithi001.stdout:Transaction Summary 2024-03-25T16:39:01.693 INFO:teuthology.orchestra.run.smithi001.stdout:================================================================================ 2024-03-25T16:39:01.693 INFO:teuthology.orchestra.run.smithi001.stdout:Install 2 Packages 2024-03-25T16:39:01.693 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-25T16:39:01.693 INFO:teuthology.orchestra.run.smithi001.stdout:Total download size: 91 M 2024-03-25T16:39:01.693 INFO:teuthology.orchestra.run.smithi001.stdout:Installed size: 498 M 2024-03-25T16:39:01.694 INFO:teuthology.orchestra.run.smithi001.stdout:Downloading Packages: 2024-03-25T16:40:17.832 INFO:teuthology.orchestra.run.smithi001.stdout:(1/2): ceph-debuginfo-19.0.0-1439.g5dcb190d.el9 302 kB/s | 22 MB 01:16 2024-03-25T16:40:34.091 INFO:teuthology.orchestra.run.smithi115.stdout:(1/2): ceph-debuginfo-19.0.0-1439.g5dcb190d.el9 207 kB/s | 22 MB 01:50 2024-03-25T16:41:22.006 INFO:teuthology.orchestra.run.smithi115.stdout:(2/2): ceph-base-debuginfo-19.0.0-1439.g5dcb190 439 kB/s | 68 MB 02:38 2024-03-25T16:41:22.007 INFO:teuthology.orchestra.run.smithi115.stdout:-------------------------------------------------------------------------------- 2024-03-25T16:41:22.007 INFO:teuthology.orchestra.run.smithi115.stdout:Total 584 kB/s | 91 MB 02:38 2024-03-25T16:41:22.007 INFO:teuthology.orchestra.run.smithi115.stdout:Running transaction check 2024-03-25T16:41:22.012 INFO:teuthology.orchestra.run.smithi115.stdout:Transaction check succeeded. 2024-03-25T16:41:22.012 INFO:teuthology.orchestra.run.smithi115.stdout:Running transaction test 2024-03-25T16:41:22.602 INFO:teuthology.orchestra.run.smithi115.stdout:Transaction test succeeded. 2024-03-25T16:41:22.602 INFO:teuthology.orchestra.run.smithi115.stdout:Running transaction 2024-03-25T16:41:25.699 INFO:teuthology.orchestra.run.smithi115.stdout: Preparing : 1/1 2024-03-25T16:41:32.796 INFO:teuthology.orchestra.run.smithi115.stdout: Installing : ceph-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/2 2024-03-25T16:41:32.873 INFO:teuthology.orchestra.run.smithi115.stdout: Installing : ceph-base-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_ 2/2 2024-03-25T16:41:36.206 INFO:teuthology.orchestra.run.smithi115.stdout: Running scriptlet: ceph-base-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_ 2/2 2024-03-25T16:41:36.207 INFO:teuthology.orchestra.run.smithi115.stdout: Verifying : ceph-base-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_ 1/2 2024-03-25T16:41:36.518 INFO:teuthology.orchestra.run.smithi115.stdout: Verifying : ceph-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_64 2/2 2024-03-25T16:41:36.518 INFO:teuthology.orchestra.run.smithi115.stdout: 2024-03-25T16:41:36.519 INFO:teuthology.orchestra.run.smithi115.stdout:Installed: 2024-03-25T16:41:36.519 INFO:teuthology.orchestra.run.smithi115.stdout: ceph-base-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T16:41:36.519 INFO:teuthology.orchestra.run.smithi115.stdout: ceph-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T16:41:36.519 INFO:teuthology.orchestra.run.smithi115.stdout: 2024-03-25T16:41:36.519 INFO:teuthology.orchestra.run.smithi115.stdout:Complete! 2024-03-25T16:41:36.739 DEBUG:teuthology.orchestra.run.smithi115:> sudo yum -y install ceph-common-debuginfo 2024-03-25T16:41:37.302 INFO:teuthology.orchestra.run.smithi115.stdout:Last metadata expiration check: 0:08:52 ago on Mon 25 Mar 2024 04:32:45 PM UTC. 2024-03-25T16:41:37.452 INFO:teuthology.orchestra.run.smithi115.stdout:Dependencies resolved. 2024-03-25T16:41:37.452 INFO:teuthology.orchestra.run.smithi115.stdout:================================================================================ 2024-03-25T16:41:37.452 INFO:teuthology.orchestra.run.smithi115.stdout: Package Arch Version Repo Size 2024-03-25T16:41:37.453 INFO:teuthology.orchestra.run.smithi115.stdout:================================================================================ 2024-03-25T16:41:37.453 INFO:teuthology.orchestra.run.smithi115.stdout:Installing: 2024-03-25T16:41:37.453 INFO:teuthology.orchestra.run.smithi115.stdout: ceph-common-debuginfo x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 416 M 2024-03-25T16:41:37.453 INFO:teuthology.orchestra.run.smithi115.stdout: 2024-03-25T16:41:37.453 INFO:teuthology.orchestra.run.smithi115.stdout:Transaction Summary 2024-03-25T16:41:37.453 INFO:teuthology.orchestra.run.smithi115.stdout:================================================================================ 2024-03-25T16:41:37.453 INFO:teuthology.orchestra.run.smithi115.stdout:Install 1 Package 2024-03-25T16:41:37.453 INFO:teuthology.orchestra.run.smithi115.stdout: 2024-03-25T16:41:37.453 INFO:teuthology.orchestra.run.smithi115.stdout:Total download size: 416 M 2024-03-25T16:41:37.453 INFO:teuthology.orchestra.run.smithi115.stdout:Installed size: 1.7 G 2024-03-25T16:41:37.453 INFO:teuthology.orchestra.run.smithi115.stdout:Downloading Packages: 2024-03-25T16:43:22.648 INFO:teuthology.orchestra.run.smithi001.stdout:(2/2): ceph-base-debuginfo-19.0.0-1439.g5dcb190 267 kB/s | 68 MB 04:20 2024-03-25T16:43:22.649 INFO:teuthology.orchestra.run.smithi001.stdout:-------------------------------------------------------------------------------- 2024-03-25T16:43:22.649 INFO:teuthology.orchestra.run.smithi001.stdout:Total 355 kB/s | 91 MB 04:20 2024-03-25T16:43:22.649 INFO:teuthology.orchestra.run.smithi001.stdout:Running transaction check 2024-03-25T16:43:22.653 INFO:teuthology.orchestra.run.smithi001.stdout:Transaction check succeeded. 2024-03-25T16:43:22.653 INFO:teuthology.orchestra.run.smithi001.stdout:Running transaction test 2024-03-25T16:43:23.267 INFO:teuthology.orchestra.run.smithi001.stdout:Transaction test succeeded. 2024-03-25T16:43:23.267 INFO:teuthology.orchestra.run.smithi001.stdout:Running transaction 2024-03-25T16:43:26.537 INFO:teuthology.orchestra.run.smithi001.stdout: Preparing : 1/1 2024-03-25T16:43:34.055 INFO:teuthology.orchestra.run.smithi001.stdout: Installing : ceph-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/2 2024-03-25T16:43:34.133 INFO:teuthology.orchestra.run.smithi001.stdout: Installing : ceph-base-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_ 2/2 2024-03-25T16:43:37.535 INFO:teuthology.orchestra.run.smithi001.stdout: Running scriptlet: ceph-base-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_ 2/2 2024-03-25T16:43:37.536 INFO:teuthology.orchestra.run.smithi001.stdout: Verifying : ceph-base-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_ 1/2 2024-03-25T16:43:37.803 INFO:teuthology.orchestra.run.smithi001.stdout: Verifying : ceph-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_64 2/2 2024-03-25T16:43:37.803 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-25T16:43:37.804 INFO:teuthology.orchestra.run.smithi001.stdout:Installed: 2024-03-25T16:43:37.804 INFO:teuthology.orchestra.run.smithi001.stdout: ceph-base-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T16:43:37.804 INFO:teuthology.orchestra.run.smithi001.stdout: ceph-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T16:43:37.804 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-25T16:43:37.804 INFO:teuthology.orchestra.run.smithi001.stdout:Complete! 2024-03-25T16:43:38.039 DEBUG:teuthology.orchestra.run.smithi001:> sudo yum -y install ceph-common-debuginfo 2024-03-25T16:43:38.613 INFO:teuthology.orchestra.run.smithi001.stdout:Last metadata expiration check: 0:10:49 ago on Mon 25 Mar 2024 04:32:49 PM UTC. 2024-03-25T16:43:38.767 INFO:teuthology.orchestra.run.smithi001.stdout:Dependencies resolved. 2024-03-25T16:43:38.768 INFO:teuthology.orchestra.run.smithi001.stdout:================================================================================ 2024-03-25T16:43:38.768 INFO:teuthology.orchestra.run.smithi001.stdout: Package Arch Version Repo Size 2024-03-25T16:43:38.768 INFO:teuthology.orchestra.run.smithi001.stdout:================================================================================ 2024-03-25T16:43:38.768 INFO:teuthology.orchestra.run.smithi001.stdout:Installing: 2024-03-25T16:43:38.769 INFO:teuthology.orchestra.run.smithi001.stdout: ceph-common-debuginfo x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 416 M 2024-03-25T16:43:38.769 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-25T16:43:38.769 INFO:teuthology.orchestra.run.smithi001.stdout:Transaction Summary 2024-03-25T16:43:38.769 INFO:teuthology.orchestra.run.smithi001.stdout:================================================================================ 2024-03-25T16:43:38.769 INFO:teuthology.orchestra.run.smithi001.stdout:Install 1 Package 2024-03-25T16:43:38.769 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-25T16:43:38.769 INFO:teuthology.orchestra.run.smithi001.stdout:Total download size: 416 M 2024-03-25T16:43:38.769 INFO:teuthology.orchestra.run.smithi001.stdout:Installed size: 1.7 G 2024-03-25T16:43:38.769 INFO:teuthology.orchestra.run.smithi001.stdout:Downloading Packages: 2024-03-25T17:00:13.627 INFO:teuthology.orchestra.run.smithi115.stdout:ceph-common-debuginfo-19.0.0-1439.g5dcb190d.el9 382 kB/s | 416 MB 18:36 2024-03-25T17:00:13.627 INFO:teuthology.orchestra.run.smithi115.stdout:-------------------------------------------------------------------------------- 2024-03-25T17:00:13.627 INFO:teuthology.orchestra.run.smithi115.stdout:Total 382 kB/s | 416 MB 18:36 2024-03-25T17:00:13.627 INFO:teuthology.orchestra.run.smithi115.stdout:Running transaction check 2024-03-25T17:00:13.630 INFO:teuthology.orchestra.run.smithi115.stdout:Transaction check succeeded. 2024-03-25T17:00:13.630 INFO:teuthology.orchestra.run.smithi115.stdout:Running transaction test 2024-03-25T17:00:16.093 INFO:teuthology.orchestra.run.smithi115.stdout:Transaction test succeeded. 2024-03-25T17:00:16.093 INFO:teuthology.orchestra.run.smithi115.stdout:Running transaction 2024-03-25T17:00:58.125 INFO:teuthology.orchestra.run.smithi115.stdout: Preparing : 1/1 2024-03-25T17:00:58.225 INFO:teuthology.orchestra.run.smithi115.stdout: Installing : ceph-common-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x8 1/1 2024-03-25T17:01:06.379 INFO:teuthology.orchestra.run.smithi115.stdout: Running scriptlet: ceph-common-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x8 1/1 2024-03-25T17:01:06.648 INFO:teuthology.orchestra.run.smithi115.stdout: Verifying : ceph-common-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x8 1/1 2024-03-25T17:01:06.649 INFO:teuthology.orchestra.run.smithi115.stdout: 2024-03-25T17:01:06.649 INFO:teuthology.orchestra.run.smithi115.stdout:Installed: 2024-03-25T17:01:06.649 INFO:teuthology.orchestra.run.smithi115.stdout: ceph-common-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T17:01:06.649 INFO:teuthology.orchestra.run.smithi115.stdout: 2024-03-25T17:01:06.649 INFO:teuthology.orchestra.run.smithi115.stdout:Complete! 2024-03-25T17:01:06.900 DEBUG:teuthology.orchestra.run.smithi115:> sudo yum -y install ceph-immutable-object-cache-debuginfo 2024-03-25T17:01:07.464 INFO:teuthology.orchestra.run.smithi115.stdout:Last metadata expiration check: 0:28:22 ago on Mon 25 Mar 2024 04:32:45 PM UTC. 2024-03-25T17:01:07.613 INFO:teuthology.orchestra.run.smithi115.stdout:Dependencies resolved. 2024-03-25T17:01:07.614 INFO:teuthology.orchestra.run.smithi115.stdout:======================================================================================== 2024-03-25T17:01:07.614 INFO:teuthology.orchestra.run.smithi115.stdout: Package Arch Version Repo Size 2024-03-25T17:01:07.614 INFO:teuthology.orchestra.run.smithi115.stdout:======================================================================================== 2024-03-25T17:01:07.614 INFO:teuthology.orchestra.run.smithi115.stdout:Installing: 2024-03-25T17:01:07.614 INFO:teuthology.orchestra.run.smithi115.stdout: ceph-immutable-object-cache-debuginfo x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 1.4 M 2024-03-25T17:01:07.614 INFO:teuthology.orchestra.run.smithi115.stdout: 2024-03-25T17:01:07.614 INFO:teuthology.orchestra.run.smithi115.stdout:Transaction Summary 2024-03-25T17:01:07.614 INFO:teuthology.orchestra.run.smithi115.stdout:======================================================================================== 2024-03-25T17:01:07.614 INFO:teuthology.orchestra.run.smithi115.stdout:Install 1 Package 2024-03-25T17:01:07.614 INFO:teuthology.orchestra.run.smithi115.stdout: 2024-03-25T17:01:07.615 INFO:teuthology.orchestra.run.smithi115.stdout:Total download size: 1.4 M 2024-03-25T17:01:07.615 INFO:teuthology.orchestra.run.smithi115.stdout:Installed size: 6.4 M 2024-03-25T17:01:07.615 INFO:teuthology.orchestra.run.smithi115.stdout:Downloading Packages: 2024-03-25T17:01:09.870 INFO:teuthology.orchestra.run.smithi115.stdout:ceph-immutable-object-cache-debuginfo-19.0.0-14 648 kB/s | 1.4 MB 00:02 2024-03-25T17:01:09.871 INFO:teuthology.orchestra.run.smithi115.stdout:-------------------------------------------------------------------------------- 2024-03-25T17:01:09.871 INFO:teuthology.orchestra.run.smithi115.stdout:Total 647 kB/s | 1.4 MB 00:02 2024-03-25T17:01:09.871 INFO:teuthology.orchestra.run.smithi115.stdout:Running transaction check 2024-03-25T17:01:09.875 INFO:teuthology.orchestra.run.smithi115.stdout:Transaction check succeeded. 2024-03-25T17:01:09.875 INFO:teuthology.orchestra.run.smithi115.stdout:Running transaction test 2024-03-25T17:01:09.920 INFO:teuthology.orchestra.run.smithi115.stdout:Transaction test succeeded. 2024-03-25T17:01:09.920 INFO:teuthology.orchestra.run.smithi115.stdout:Running transaction 2024-03-25T17:01:10.192 INFO:teuthology.orchestra.run.smithi115.stdout: Preparing : 1/1 2024-03-25T17:01:10.293 INFO:teuthology.orchestra.run.smithi115.stdout: Installing : ceph-immutable-object-cache-debuginfo-2:19.0.0-1439. 1/1 2024-03-25T17:01:10.709 INFO:teuthology.orchestra.run.smithi115.stdout: Running scriptlet: ceph-immutable-object-cache-debuginfo-2:19.0.0-1439. 1/1 2024-03-25T17:01:10.982 INFO:teuthology.orchestra.run.smithi115.stdout: Verifying : ceph-immutable-object-cache-debuginfo-2:19.0.0-1439. 1/1 2024-03-25T17:01:10.982 INFO:teuthology.orchestra.run.smithi115.stdout: 2024-03-25T17:01:10.982 INFO:teuthology.orchestra.run.smithi115.stdout:Installed: 2024-03-25T17:01:10.982 INFO:teuthology.orchestra.run.smithi115.stdout: ceph-immutable-object-cache-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T17:01:10.982 INFO:teuthology.orchestra.run.smithi115.stdout: 2024-03-25T17:01:10.983 INFO:teuthology.orchestra.run.smithi115.stdout:Complete! 2024-03-25T17:01:11.162 DEBUG:teuthology.orchestra.run.smithi115:> sudo yum -y install ceph-radosgw-debuginfo 2024-03-25T17:01:11.718 INFO:teuthology.orchestra.run.smithi115.stdout:Last metadata expiration check: 0:28:26 ago on Mon 25 Mar 2024 04:32:45 PM UTC. 2024-03-25T17:01:11.866 INFO:teuthology.orchestra.run.smithi115.stdout:Dependencies resolved. 2024-03-25T17:01:11.867 INFO:teuthology.orchestra.run.smithi115.stdout:================================================================================ 2024-03-25T17:01:11.867 INFO:teuthology.orchestra.run.smithi115.stdout: Package Arch Version Repo Size 2024-03-25T17:01:11.867 INFO:teuthology.orchestra.run.smithi115.stdout:================================================================================ 2024-03-25T17:01:11.867 INFO:teuthology.orchestra.run.smithi115.stdout:Installing: 2024-03-25T17:01:11.867 INFO:teuthology.orchestra.run.smithi115.stdout: ceph-radosgw-debuginfo x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 176 M 2024-03-25T17:01:11.867 INFO:teuthology.orchestra.run.smithi115.stdout: 2024-03-25T17:01:11.867 INFO:teuthology.orchestra.run.smithi115.stdout:Transaction Summary 2024-03-25T17:01:11.867 INFO:teuthology.orchestra.run.smithi115.stdout:================================================================================ 2024-03-25T17:01:11.867 INFO:teuthology.orchestra.run.smithi115.stdout:Install 1 Package 2024-03-25T17:01:11.867 INFO:teuthology.orchestra.run.smithi115.stdout: 2024-03-25T17:01:11.868 INFO:teuthology.orchestra.run.smithi115.stdout:Total download size: 176 M 2024-03-25T17:01:11.868 INFO:teuthology.orchestra.run.smithi115.stdout:Installed size: 753 M 2024-03-25T17:01:11.868 INFO:teuthology.orchestra.run.smithi115.stdout:Downloading Packages: 2024-03-25T17:09:07.598 INFO:teuthology.orchestra.run.smithi001.stdout:ceph-common-debuginfo-19.0.0-1439.g5dcb190d.el9 279 kB/s | 416 MB 25:28 2024-03-25T17:09:07.598 INFO:teuthology.orchestra.run.smithi001.stdout:-------------------------------------------------------------------------------- 2024-03-25T17:09:07.598 INFO:teuthology.orchestra.run.smithi001.stdout:Total 279 kB/s | 416 MB 25:28 2024-03-25T17:09:07.598 INFO:teuthology.orchestra.run.smithi001.stdout:Running transaction check 2024-03-25T17:09:07.601 INFO:teuthology.orchestra.run.smithi001.stdout:Transaction check succeeded. 2024-03-25T17:09:07.601 INFO:teuthology.orchestra.run.smithi001.stdout:Running transaction test 2024-03-25T17:09:10.188 INFO:teuthology.orchestra.run.smithi001.stdout:Transaction test succeeded. 2024-03-25T17:09:10.189 INFO:teuthology.orchestra.run.smithi001.stdout:Running transaction 2024-03-25T17:09:54.918 INFO:teuthology.orchestra.run.smithi001.stdout: Preparing : 1/1 2024-03-25T17:09:55.020 INFO:teuthology.orchestra.run.smithi001.stdout: Installing : ceph-common-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x8 1/1 2024-03-25T17:10:02.944 INFO:teuthology.orchestra.run.smithi001.stdout: Running scriptlet: ceph-common-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x8 1/1 2024-03-25T17:10:03.194 INFO:teuthology.orchestra.run.smithi001.stdout: Verifying : ceph-common-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x8 1/1 2024-03-25T17:10:03.194 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-25T17:10:03.194 INFO:teuthology.orchestra.run.smithi001.stdout:Installed: 2024-03-25T17:10:03.194 INFO:teuthology.orchestra.run.smithi001.stdout: ceph-common-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T17:10:03.194 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-25T17:10:03.194 INFO:teuthology.orchestra.run.smithi001.stdout:Complete! 2024-03-25T17:10:03.481 DEBUG:teuthology.orchestra.run.smithi001:> sudo yum -y install ceph-immutable-object-cache-debuginfo 2024-03-25T17:10:04.061 INFO:teuthology.orchestra.run.smithi001.stdout:Last metadata expiration check: 0:37:15 ago on Mon 25 Mar 2024 04:32:49 PM UTC. 2024-03-25T17:10:04.216 INFO:teuthology.orchestra.run.smithi001.stdout:Dependencies resolved. 2024-03-25T17:10:04.216 INFO:teuthology.orchestra.run.smithi001.stdout:======================================================================================== 2024-03-25T17:10:04.217 INFO:teuthology.orchestra.run.smithi001.stdout: Package Arch Version Repo Size 2024-03-25T17:10:04.217 INFO:teuthology.orchestra.run.smithi001.stdout:======================================================================================== 2024-03-25T17:10:04.217 INFO:teuthology.orchestra.run.smithi001.stdout:Installing: 2024-03-25T17:10:04.217 INFO:teuthology.orchestra.run.smithi001.stdout: ceph-immutable-object-cache-debuginfo x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 1.4 M 2024-03-25T17:10:04.217 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-25T17:10:04.217 INFO:teuthology.orchestra.run.smithi001.stdout:Transaction Summary 2024-03-25T17:10:04.217 INFO:teuthology.orchestra.run.smithi001.stdout:======================================================================================== 2024-03-25T17:10:04.217 INFO:teuthology.orchestra.run.smithi001.stdout:Install 1 Package 2024-03-25T17:10:04.217 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-25T17:10:04.217 INFO:teuthology.orchestra.run.smithi001.stdout:Total download size: 1.4 M 2024-03-25T17:10:04.217 INFO:teuthology.orchestra.run.smithi001.stdout:Installed size: 6.4 M 2024-03-25T17:10:04.218 INFO:teuthology.orchestra.run.smithi001.stdout:Downloading Packages: 2024-03-25T17:10:08.454 INFO:teuthology.orchestra.run.smithi001.stdout:ceph-immutable-object-cache-debuginfo-19.0.0-14 345 kB/s | 1.4 MB 00:04 2024-03-25T17:10:08.454 INFO:teuthology.orchestra.run.smithi001.stdout:-------------------------------------------------------------------------------- 2024-03-25T17:10:08.455 INFO:teuthology.orchestra.run.smithi001.stdout:Total 345 kB/s | 1.4 MB 00:04 2024-03-25T17:10:08.455 INFO:teuthology.orchestra.run.smithi001.stdout:Running transaction check 2024-03-25T17:10:08.459 INFO:teuthology.orchestra.run.smithi001.stdout:Transaction check succeeded. 2024-03-25T17:10:08.459 INFO:teuthology.orchestra.run.smithi001.stdout:Running transaction test 2024-03-25T17:10:08.506 INFO:teuthology.orchestra.run.smithi001.stdout:Transaction test succeeded. 2024-03-25T17:10:08.506 INFO:teuthology.orchestra.run.smithi001.stdout:Running transaction 2024-03-25T17:10:08.770 INFO:teuthology.orchestra.run.smithi001.stdout: Preparing : 1/1 2024-03-25T17:10:09.005 INFO:teuthology.orchestra.run.smithi001.stdout: Installing : ceph-immutable-object-cache-debuginfo-2:19.0.0-1439. 1/1 2024-03-25T17:10:09.420 INFO:teuthology.orchestra.run.smithi001.stdout: Running scriptlet: ceph-immutable-object-cache-debuginfo-2:19.0.0-1439. 1/1 2024-03-25T17:10:09.687 INFO:teuthology.orchestra.run.smithi001.stdout: Verifying : ceph-immutable-object-cache-debuginfo-2:19.0.0-1439. 1/1 2024-03-25T17:10:09.687 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-25T17:10:09.687 INFO:teuthology.orchestra.run.smithi001.stdout:Installed: 2024-03-25T17:10:09.687 INFO:teuthology.orchestra.run.smithi001.stdout: ceph-immutable-object-cache-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T17:10:09.688 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-25T17:10:09.688 INFO:teuthology.orchestra.run.smithi001.stdout:Complete! 2024-03-25T17:10:10.058 DEBUG:teuthology.orchestra.run.smithi001:> sudo yum -y install ceph-radosgw-debuginfo 2024-03-25T17:10:10.636 INFO:teuthology.orchestra.run.smithi001.stdout:Last metadata expiration check: 0:37:21 ago on Mon 25 Mar 2024 04:32:49 PM UTC. 2024-03-25T17:10:10.790 INFO:teuthology.orchestra.run.smithi001.stdout:Dependencies resolved. 2024-03-25T17:10:10.791 INFO:teuthology.orchestra.run.smithi001.stdout:================================================================================ 2024-03-25T17:10:10.791 INFO:teuthology.orchestra.run.smithi001.stdout: Package Arch Version Repo Size 2024-03-25T17:10:10.791 INFO:teuthology.orchestra.run.smithi001.stdout:================================================================================ 2024-03-25T17:10:10.791 INFO:teuthology.orchestra.run.smithi001.stdout:Installing: 2024-03-25T17:10:10.791 INFO:teuthology.orchestra.run.smithi001.stdout: ceph-radosgw-debuginfo x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 176 M 2024-03-25T17:10:10.791 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-25T17:10:10.791 INFO:teuthology.orchestra.run.smithi001.stdout:Transaction Summary 2024-03-25T17:10:10.791 INFO:teuthology.orchestra.run.smithi001.stdout:================================================================================ 2024-03-25T17:10:10.791 INFO:teuthology.orchestra.run.smithi001.stdout:Install 1 Package 2024-03-25T17:10:10.791 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-25T17:10:10.792 INFO:teuthology.orchestra.run.smithi001.stdout:Total download size: 176 M 2024-03-25T17:10:10.792 INFO:teuthology.orchestra.run.smithi001.stdout:Installed size: 753 M 2024-03-25T17:10:10.792 INFO:teuthology.orchestra.run.smithi001.stdout:Downloading Packages: 2024-03-25T17:11:15.559 INFO:teuthology.orchestra.run.smithi115.stdout:ceph-radosgw-debuginfo-19.0.0-1439.g5dcb190d.el 298 kB/s | 176 MB 10:03 2024-03-25T17:11:15.560 INFO:teuthology.orchestra.run.smithi115.stdout:-------------------------------------------------------------------------------- 2024-03-25T17:11:15.560 INFO:teuthology.orchestra.run.smithi115.stdout:Total 298 kB/s | 176 MB 10:03 2024-03-25T17:11:15.560 INFO:teuthology.orchestra.run.smithi115.stdout:Running transaction check 2024-03-25T17:11:15.563 INFO:teuthology.orchestra.run.smithi115.stdout:Transaction check succeeded. 2024-03-25T17:11:15.563 INFO:teuthology.orchestra.run.smithi115.stdout:Running transaction test 2024-03-25T17:11:16.623 INFO:teuthology.orchestra.run.smithi115.stdout:Transaction test succeeded. 2024-03-25T17:11:16.624 INFO:teuthology.orchestra.run.smithi115.stdout:Running transaction 2024-03-25T17:11:34.792 INFO:teuthology.orchestra.run.smithi115.stdout: Preparing : 1/1 2024-03-25T17:11:34.961 INFO:teuthology.orchestra.run.smithi115.stdout: Installing : ceph-radosgw-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x 1/1 2024-03-25T17:11:39.406 INFO:teuthology.orchestra.run.smithi115.stdout: Running scriptlet: ceph-radosgw-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x 1/1 2024-03-25T17:11:39.667 INFO:teuthology.orchestra.run.smithi115.stdout: Verifying : ceph-radosgw-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x 1/1 2024-03-25T17:11:39.667 INFO:teuthology.orchestra.run.smithi115.stdout: 2024-03-25T17:11:39.667 INFO:teuthology.orchestra.run.smithi115.stdout:Installed: 2024-03-25T17:11:39.667 INFO:teuthology.orchestra.run.smithi115.stdout: ceph-radosgw-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T17:11:39.668 INFO:teuthology.orchestra.run.smithi115.stdout: 2024-03-25T17:11:39.668 INFO:teuthology.orchestra.run.smithi115.stdout:Complete! 2024-03-25T17:11:39.878 DEBUG:teuthology.orchestra.run.smithi115:> sudo yum -y install ceph-test-debuginfo 2024-03-25T17:11:40.439 INFO:teuthology.orchestra.run.smithi115.stdout:Last metadata expiration check: 0:38:55 ago on Mon 25 Mar 2024 04:32:45 PM UTC. 2024-03-25T17:11:40.586 INFO:teuthology.orchestra.run.smithi115.stdout:Dependencies resolved. 2024-03-25T17:11:40.586 INFO:teuthology.orchestra.run.smithi115.stdout:================================================================================ 2024-03-25T17:11:40.587 INFO:teuthology.orchestra.run.smithi115.stdout: Package Arch Version Repo Size 2024-03-25T17:11:40.587 INFO:teuthology.orchestra.run.smithi115.stdout:================================================================================ 2024-03-25T17:11:40.587 INFO:teuthology.orchestra.run.smithi115.stdout:Installing: 2024-03-25T17:11:40.587 INFO:teuthology.orchestra.run.smithi115.stdout: ceph-test-debuginfo x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 672 M 2024-03-25T17:11:40.587 INFO:teuthology.orchestra.run.smithi115.stdout: 2024-03-25T17:11:40.587 INFO:teuthology.orchestra.run.smithi115.stdout:Transaction Summary 2024-03-25T17:11:40.587 INFO:teuthology.orchestra.run.smithi115.stdout:================================================================================ 2024-03-25T17:11:40.587 INFO:teuthology.orchestra.run.smithi115.stdout:Install 1 Package 2024-03-25T17:11:40.587 INFO:teuthology.orchestra.run.smithi115.stdout: 2024-03-25T17:11:40.588 INFO:teuthology.orchestra.run.smithi115.stdout:Total download size: 672 M 2024-03-25T17:11:40.588 INFO:teuthology.orchestra.run.smithi115.stdout:Installed size: 3.1 G 2024-03-25T17:11:40.588 INFO:teuthology.orchestra.run.smithi115.stdout:Downloading Packages: 2024-03-25T17:21:36.285 INFO:teuthology.orchestra.run.smithi001.stdout:ceph-radosgw-debuginfo-19.0.0-1439.g5dcb190d.el 262 kB/s | 176 MB 11:25 2024-03-25T17:21:36.302 INFO:teuthology.orchestra.run.smithi001.stdout:-------------------------------------------------------------------------------- 2024-03-25T17:21:36.302 INFO:teuthology.orchestra.run.smithi001.stdout:Total 262 kB/s | 176 MB 11:25 2024-03-25T17:21:36.302 INFO:teuthology.orchestra.run.smithi001.stdout:Running transaction check 2024-03-25T17:21:36.303 INFO:teuthology.orchestra.run.smithi001.stdout:Transaction check succeeded. 2024-03-25T17:21:36.303 INFO:teuthology.orchestra.run.smithi001.stdout:Running transaction test 2024-03-25T17:21:37.399 INFO:teuthology.orchestra.run.smithi001.stdout:Transaction test succeeded. 2024-03-25T17:21:37.399 INFO:teuthology.orchestra.run.smithi001.stdout:Running transaction 2024-03-25T17:21:56.435 INFO:teuthology.orchestra.run.smithi001.stdout: Preparing : 1/1 2024-03-25T17:21:56.533 INFO:teuthology.orchestra.run.smithi001.stdout: Installing : ceph-radosgw-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x 1/1 2024-03-25T17:22:01.104 INFO:teuthology.orchestra.run.smithi001.stdout: Running scriptlet: ceph-radosgw-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x 1/1 2024-03-25T17:22:01.359 INFO:teuthology.orchestra.run.smithi001.stdout: Verifying : ceph-radosgw-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x 1/1 2024-03-25T17:22:01.359 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-25T17:22:01.359 INFO:teuthology.orchestra.run.smithi001.stdout:Installed: 2024-03-25T17:22:01.359 INFO:teuthology.orchestra.run.smithi001.stdout: ceph-radosgw-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T17:22:01.359 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-25T17:22:01.359 INFO:teuthology.orchestra.run.smithi001.stdout:Complete! 2024-03-25T17:22:01.587 DEBUG:teuthology.orchestra.run.smithi001:> sudo yum -y install ceph-test-debuginfo 2024-03-25T17:22:02.160 INFO:teuthology.orchestra.run.smithi001.stdout:Last metadata expiration check: 0:49:13 ago on Mon 25 Mar 2024 04:32:49 PM UTC. 2024-03-25T17:22:02.315 INFO:teuthology.orchestra.run.smithi001.stdout:Dependencies resolved. 2024-03-25T17:22:02.316 INFO:teuthology.orchestra.run.smithi001.stdout:================================================================================ 2024-03-25T17:22:02.316 INFO:teuthology.orchestra.run.smithi001.stdout: Package Arch Version Repo Size 2024-03-25T17:22:02.316 INFO:teuthology.orchestra.run.smithi001.stdout:================================================================================ 2024-03-25T17:22:02.316 INFO:teuthology.orchestra.run.smithi001.stdout:Installing: 2024-03-25T17:22:02.316 INFO:teuthology.orchestra.run.smithi001.stdout: ceph-test-debuginfo x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 672 M 2024-03-25T17:22:02.316 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-25T17:22:02.316 INFO:teuthology.orchestra.run.smithi001.stdout:Transaction Summary 2024-03-25T17:22:02.316 INFO:teuthology.orchestra.run.smithi001.stdout:================================================================================ 2024-03-25T17:22:02.316 INFO:teuthology.orchestra.run.smithi001.stdout:Install 1 Package 2024-03-25T17:22:02.317 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-25T17:22:02.317 INFO:teuthology.orchestra.run.smithi001.stdout:Total download size: 672 M 2024-03-25T17:22:02.317 INFO:teuthology.orchestra.run.smithi001.stdout:Installed size: 3.1 G 2024-03-25T17:22:02.317 INFO:teuthology.orchestra.run.smithi001.stdout:Downloading Packages: 2024-03-25T17:39:03.424 INFO:teuthology.orchestra.run.smithi115.stdout:ceph-test-debuginfo-19.0.0-1439.g5dcb190d.el9.x 419 kB/s | 672 MB 27:22 2024-03-25T17:39:03.454 INFO:teuthology.orchestra.run.smithi115.stdout:-------------------------------------------------------------------------------- 2024-03-25T17:39:03.454 INFO:teuthology.orchestra.run.smithi115.stdout:Total 419 kB/s | 672 MB 27:22 2024-03-25T17:39:03.455 INFO:teuthology.orchestra.run.smithi115.stdout:Running transaction check 2024-03-25T17:39:03.455 INFO:teuthology.orchestra.run.smithi115.stdout:Transaction check succeeded. 2024-03-25T17:39:03.455 INFO:teuthology.orchestra.run.smithi115.stdout:Running transaction test 2024-03-25T17:39:07.378 INFO:teuthology.orchestra.run.smithi115.stdout:Transaction test succeeded. 2024-03-25T17:39:07.379 INFO:teuthology.orchestra.run.smithi115.stdout:Running transaction 2024-03-25T17:40:18.007 INFO:teuthology.orchestra.run.smithi115.stdout: Preparing : 1/1 2024-03-25T17:40:18.604 INFO:teuthology.orchestra.run.smithi115.stdout: Installing : ceph-test-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_ 1/1 2024-03-25T17:40:27.478 INFO:teuthology.orchestra.run.smithi115.stdout: Running scriptlet: ceph-test-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_ 1/1 2024-03-25T17:40:27.736 INFO:teuthology.orchestra.run.smithi115.stdout: Verifying : ceph-test-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_ 1/1 2024-03-25T17:40:27.737 INFO:teuthology.orchestra.run.smithi115.stdout: 2024-03-25T17:40:27.737 INFO:teuthology.orchestra.run.smithi115.stdout:Installed: 2024-03-25T17:40:27.737 INFO:teuthology.orchestra.run.smithi115.stdout: ceph-test-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T17:40:27.737 INFO:teuthology.orchestra.run.smithi115.stdout: 2024-03-25T17:40:27.737 INFO:teuthology.orchestra.run.smithi115.stdout:Complete! 2024-03-25T17:40:28.009 DEBUG:teuthology.orchestra.run.smithi115:> sudo yum -y install ceph-base-debuginfo 2024-03-25T17:40:28.570 INFO:teuthology.orchestra.run.smithi115.stdout:Last metadata expiration check: 1:07:43 ago on Mon 25 Mar 2024 04:32:45 PM UTC. 2024-03-25T17:40:28.666 INFO:teuthology.orchestra.run.smithi115.stdout:Package ceph-base-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_64 is already installed. 2024-03-25T17:40:28.718 INFO:teuthology.orchestra.run.smithi115.stdout:Dependencies resolved. 2024-03-25T17:40:28.719 INFO:teuthology.orchestra.run.smithi115.stdout:Nothing to do. 2024-03-25T17:40:28.719 INFO:teuthology.orchestra.run.smithi115.stdout:Complete! 2024-03-25T17:40:28.779 DEBUG:teuthology.orchestra.run.smithi115:> sudo yum -y install ceph-mgr-debuginfo 2024-03-25T17:40:29.340 INFO:teuthology.orchestra.run.smithi115.stdout:Last metadata expiration check: 1:07:44 ago on Mon 25 Mar 2024 04:32:45 PM UTC. 2024-03-25T17:40:29.488 INFO:teuthology.orchestra.run.smithi115.stdout:Dependencies resolved. 2024-03-25T17:40:29.489 INFO:teuthology.orchestra.run.smithi115.stdout:================================================================================ 2024-03-25T17:40:29.489 INFO:teuthology.orchestra.run.smithi115.stdout: Package Arch Version Repo Size 2024-03-25T17:40:29.489 INFO:teuthology.orchestra.run.smithi115.stdout:================================================================================ 2024-03-25T17:40:29.490 INFO:teuthology.orchestra.run.smithi115.stdout:Installing: 2024-03-25T17:40:29.490 INFO:teuthology.orchestra.run.smithi115.stdout: ceph-mgr-debuginfo x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 20 M 2024-03-25T17:40:29.490 INFO:teuthology.orchestra.run.smithi115.stdout: 2024-03-25T17:40:29.490 INFO:teuthology.orchestra.run.smithi115.stdout:Transaction Summary 2024-03-25T17:40:29.490 INFO:teuthology.orchestra.run.smithi115.stdout:================================================================================ 2024-03-25T17:40:29.490 INFO:teuthology.orchestra.run.smithi115.stdout:Install 1 Package 2024-03-25T17:40:29.490 INFO:teuthology.orchestra.run.smithi115.stdout: 2024-03-25T17:40:29.490 INFO:teuthology.orchestra.run.smithi115.stdout:Total download size: 20 M 2024-03-25T17:40:29.490 INFO:teuthology.orchestra.run.smithi115.stdout:Installed size: 103 M 2024-03-25T17:40:29.490 INFO:teuthology.orchestra.run.smithi115.stdout:Downloading Packages: 2024-03-25T17:41:06.475 INFO:teuthology.orchestra.run.smithi115.stdout:ceph-mgr-debuginfo-19.0.0-1439.g5dcb190d.el9.x8 545 kB/s | 20 MB 00:36 2024-03-25T17:41:06.475 INFO:teuthology.orchestra.run.smithi115.stdout:-------------------------------------------------------------------------------- 2024-03-25T17:41:06.476 INFO:teuthology.orchestra.run.smithi115.stdout:Total 545 kB/s | 20 MB 00:36 2024-03-25T17:41:06.476 INFO:teuthology.orchestra.run.smithi115.stdout:Running transaction check 2024-03-25T17:41:06.479 INFO:teuthology.orchestra.run.smithi115.stdout:Transaction check succeeded. 2024-03-25T17:41:06.479 INFO:teuthology.orchestra.run.smithi115.stdout:Running transaction test 2024-03-25T17:41:06.628 INFO:teuthology.orchestra.run.smithi115.stdout:Transaction test succeeded. 2024-03-25T17:41:06.629 INFO:teuthology.orchestra.run.smithi115.stdout:Running transaction 2024-03-25T17:41:08.859 INFO:teuthology.orchestra.run.smithi115.stdout: Preparing : 1/1 2024-03-25T17:41:08.973 INFO:teuthology.orchestra.run.smithi115.stdout: Installing : ceph-mgr-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_6 1/1 2024-03-25T17:41:09.916 INFO:teuthology.orchestra.run.smithi115.stdout: Running scriptlet: ceph-mgr-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_6 1/1 2024-03-25T17:41:10.175 INFO:teuthology.orchestra.run.smithi115.stdout: Verifying : ceph-mgr-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_6 1/1 2024-03-25T17:41:10.175 INFO:teuthology.orchestra.run.smithi115.stdout: 2024-03-25T17:41:10.175 INFO:teuthology.orchestra.run.smithi115.stdout:Installed: 2024-03-25T17:41:10.175 INFO:teuthology.orchestra.run.smithi115.stdout: ceph-mgr-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T17:41:10.175 INFO:teuthology.orchestra.run.smithi115.stdout: 2024-03-25T17:41:10.175 INFO:teuthology.orchestra.run.smithi115.stdout:Complete! 2024-03-25T17:41:10.340 DEBUG:teuthology.orchestra.run.smithi115:> sudo yum -y install ceph-mds-debuginfo 2024-03-25T17:41:10.906 INFO:teuthology.orchestra.run.smithi115.stdout:Last metadata expiration check: 1:08:25 ago on Mon 25 Mar 2024 04:32:45 PM UTC. 2024-03-25T17:41:11.055 INFO:teuthology.orchestra.run.smithi115.stdout:Dependencies resolved. 2024-03-25T17:41:11.056 INFO:teuthology.orchestra.run.smithi115.stdout:================================================================================ 2024-03-25T17:41:11.056 INFO:teuthology.orchestra.run.smithi115.stdout: Package Arch Version Repo Size 2024-03-25T17:41:11.056 INFO:teuthology.orchestra.run.smithi115.stdout:================================================================================ 2024-03-25T17:41:11.056 INFO:teuthology.orchestra.run.smithi115.stdout:Installing: 2024-03-25T17:41:11.056 INFO:teuthology.orchestra.run.smithi115.stdout: ceph-mds-debuginfo x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 48 M 2024-03-25T17:41:11.057 INFO:teuthology.orchestra.run.smithi115.stdout: 2024-03-25T17:41:11.057 INFO:teuthology.orchestra.run.smithi115.stdout:Transaction Summary 2024-03-25T17:41:11.057 INFO:teuthology.orchestra.run.smithi115.stdout:================================================================================ 2024-03-25T17:41:11.057 INFO:teuthology.orchestra.run.smithi115.stdout:Install 1 Package 2024-03-25T17:41:11.057 INFO:teuthology.orchestra.run.smithi115.stdout: 2024-03-25T17:41:11.057 INFO:teuthology.orchestra.run.smithi115.stdout:Total download size: 48 M 2024-03-25T17:41:11.057 INFO:teuthology.orchestra.run.smithi115.stdout:Installed size: 173 M 2024-03-25T17:41:11.057 INFO:teuthology.orchestra.run.smithi115.stdout:Downloading Packages: 2024-03-25T17:42:52.506 INFO:teuthology.orchestra.run.smithi115.stdout:ceph-mds-debuginfo-19.0.0-1439.g5dcb190d.el9.x8 482 kB/s | 48 MB 01:41 2024-03-25T17:42:52.507 INFO:teuthology.orchestra.run.smithi115.stdout:-------------------------------------------------------------------------------- 2024-03-25T17:42:52.507 INFO:teuthology.orchestra.run.smithi115.stdout:Total 482 kB/s | 48 MB 01:41 2024-03-25T17:42:52.507 INFO:teuthology.orchestra.run.smithi115.stdout:Running transaction check 2024-03-25T17:42:52.510 INFO:teuthology.orchestra.run.smithi115.stdout:Transaction check succeeded. 2024-03-25T17:42:52.510 INFO:teuthology.orchestra.run.smithi115.stdout:Running transaction test 2024-03-25T17:42:52.823 INFO:teuthology.orchestra.run.smithi115.stdout:Transaction test succeeded. 2024-03-25T17:42:52.823 INFO:teuthology.orchestra.run.smithi115.stdout:Running transaction 2024-03-25T17:42:57.554 INFO:teuthology.orchestra.run.smithi115.stdout: Preparing : 1/1 2024-03-25T17:42:57.636 INFO:teuthology.orchestra.run.smithi115.stdout: Installing : ceph-mds-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_6 1/1 2024-03-25T17:42:58.958 INFO:teuthology.orchestra.run.smithi115.stdout: Running scriptlet: ceph-mds-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_6 1/1 2024-03-25T17:42:59.207 INFO:teuthology.orchestra.run.smithi115.stdout: Verifying : ceph-mds-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_6 1/1 2024-03-25T17:42:59.207 INFO:teuthology.orchestra.run.smithi115.stdout: 2024-03-25T17:42:59.207 INFO:teuthology.orchestra.run.smithi115.stdout:Installed: 2024-03-25T17:42:59.207 INFO:teuthology.orchestra.run.smithi115.stdout: ceph-mds-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T17:42:59.207 INFO:teuthology.orchestra.run.smithi115.stdout: 2024-03-25T17:42:59.208 INFO:teuthology.orchestra.run.smithi115.stdout:Complete! 2024-03-25T17:42:59.382 DEBUG:teuthology.orchestra.run.smithi115:> sudo yum -y install ceph-mon-debuginfo 2024-03-25T17:42:59.946 INFO:teuthology.orchestra.run.smithi115.stdout:Last metadata expiration check: 1:10:14 ago on Mon 25 Mar 2024 04:32:45 PM UTC. 2024-03-25T17:43:00.093 INFO:teuthology.orchestra.run.smithi115.stdout:Dependencies resolved. 2024-03-25T17:43:00.093 INFO:teuthology.orchestra.run.smithi115.stdout:================================================================================ 2024-03-25T17:43:00.093 INFO:teuthology.orchestra.run.smithi115.stdout: Package Arch Version Repo Size 2024-03-25T17:43:00.094 INFO:teuthology.orchestra.run.smithi115.stdout:================================================================================ 2024-03-25T17:43:00.094 INFO:teuthology.orchestra.run.smithi115.stdout:Installing: 2024-03-25T17:43:00.094 INFO:teuthology.orchestra.run.smithi115.stdout: ceph-mon-debuginfo x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 99 M 2024-03-25T17:43:00.094 INFO:teuthology.orchestra.run.smithi115.stdout: 2024-03-25T17:43:00.094 INFO:teuthology.orchestra.run.smithi115.stdout:Transaction Summary 2024-03-25T17:43:00.094 INFO:teuthology.orchestra.run.smithi115.stdout:================================================================================ 2024-03-25T17:43:00.094 INFO:teuthology.orchestra.run.smithi115.stdout:Install 1 Package 2024-03-25T17:43:00.094 INFO:teuthology.orchestra.run.smithi115.stdout: 2024-03-25T17:43:00.094 INFO:teuthology.orchestra.run.smithi115.stdout:Total download size: 99 M 2024-03-25T17:43:00.094 INFO:teuthology.orchestra.run.smithi115.stdout:Installed size: 449 M 2024-03-25T17:43:00.094 INFO:teuthology.orchestra.run.smithi115.stdout:Downloading Packages: 2024-03-25T17:45:59.888 INFO:teuthology.orchestra.run.smithi115.stdout:ceph-mon-debuginfo-19.0.0-1439.g5dcb190d.el9.x8 561 kB/s | 99 MB 02:59 2024-03-25T17:45:59.889 INFO:teuthology.orchestra.run.smithi115.stdout:-------------------------------------------------------------------------------- 2024-03-25T17:45:59.889 INFO:teuthology.orchestra.run.smithi115.stdout:Total 561 kB/s | 99 MB 02:59 2024-03-25T17:45:59.889 INFO:teuthology.orchestra.run.smithi115.stdout:Running transaction check 2024-03-25T17:45:59.892 INFO:teuthology.orchestra.run.smithi115.stdout:Transaction check succeeded. 2024-03-25T17:45:59.892 INFO:teuthology.orchestra.run.smithi115.stdout:Running transaction test 2024-03-25T17:46:00.500 INFO:teuthology.orchestra.run.smithi115.stdout:Transaction test succeeded. 2024-03-25T17:46:00.500 INFO:teuthology.orchestra.run.smithi115.stdout:Running transaction 2024-03-25T17:46:10.708 INFO:teuthology.orchestra.run.smithi115.stdout: Preparing : 1/1 2024-03-25T17:46:10.801 INFO:teuthology.orchestra.run.smithi115.stdout: Installing : ceph-mon-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_6 1/1 2024-03-25T17:46:13.695 INFO:teuthology.orchestra.run.smithi115.stdout: Running scriptlet: ceph-mon-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_6 1/1 2024-03-25T17:46:13.955 INFO:teuthology.orchestra.run.smithi115.stdout: Verifying : ceph-mon-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_6 1/1 2024-03-25T17:46:13.955 INFO:teuthology.orchestra.run.smithi115.stdout: 2024-03-25T17:46:13.955 INFO:teuthology.orchestra.run.smithi115.stdout:Installed: 2024-03-25T17:46:13.955 INFO:teuthology.orchestra.run.smithi115.stdout: ceph-mon-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T17:46:13.956 INFO:teuthology.orchestra.run.smithi115.stdout: 2024-03-25T17:46:13.956 INFO:teuthology.orchestra.run.smithi115.stdout:Complete! 2024-03-25T17:46:14.124 DEBUG:teuthology.orchestra.run.smithi115:> sudo yum -y install ceph-osd-debuginfo 2024-03-25T17:46:14.689 INFO:teuthology.orchestra.run.smithi115.stdout:Last metadata expiration check: 1:13:29 ago on Mon 25 Mar 2024 04:32:45 PM UTC. 2024-03-25T17:46:14.839 INFO:teuthology.orchestra.run.smithi115.stdout:Dependencies resolved. 2024-03-25T17:46:14.840 INFO:teuthology.orchestra.run.smithi115.stdout:================================================================================ 2024-03-25T17:46:14.840 INFO:teuthology.orchestra.run.smithi115.stdout: Package Arch Version Repo Size 2024-03-25T17:46:14.840 INFO:teuthology.orchestra.run.smithi115.stdout:================================================================================ 2024-03-25T17:46:14.840 INFO:teuthology.orchestra.run.smithi115.stdout:Installing: 2024-03-25T17:46:14.840 INFO:teuthology.orchestra.run.smithi115.stdout: ceph-osd-debuginfo x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 287 M 2024-03-25T17:46:14.840 INFO:teuthology.orchestra.run.smithi115.stdout: 2024-03-25T17:46:14.840 INFO:teuthology.orchestra.run.smithi115.stdout:Transaction Summary 2024-03-25T17:46:14.840 INFO:teuthology.orchestra.run.smithi115.stdout:================================================================================ 2024-03-25T17:46:14.840 INFO:teuthology.orchestra.run.smithi115.stdout:Install 1 Package 2024-03-25T17:46:14.840 INFO:teuthology.orchestra.run.smithi115.stdout: 2024-03-25T17:46:14.841 INFO:teuthology.orchestra.run.smithi115.stdout:Total download size: 287 M 2024-03-25T17:46:14.841 INFO:teuthology.orchestra.run.smithi115.stdout:Installed size: 1.2 G 2024-03-25T17:46:14.841 INFO:teuthology.orchestra.run.smithi115.stdout:Downloading Packages: 2024-03-25T17:54:19.135 INFO:teuthology.orchestra.run.smithi001.stdout:ceph-test-debuginfo-19.0.0-1439.g5dcb190d.el9.x 355 kB/s | 672 MB 32:16 2024-03-25T17:54:19.135 INFO:teuthology.orchestra.run.smithi001.stdout:-------------------------------------------------------------------------------- 2024-03-25T17:54:19.135 INFO:teuthology.orchestra.run.smithi001.stdout:Total 355 kB/s | 672 MB 32:16 2024-03-25T17:54:19.135 INFO:teuthology.orchestra.run.smithi001.stdout:Running transaction check 2024-03-25T17:54:19.139 INFO:teuthology.orchestra.run.smithi001.stdout:Transaction check succeeded. 2024-03-25T17:54:19.140 INFO:teuthology.orchestra.run.smithi001.stdout:Running transaction test 2024-03-25T17:54:23.293 INFO:teuthology.orchestra.run.smithi001.stdout:Transaction test succeeded. 2024-03-25T17:54:23.293 INFO:teuthology.orchestra.run.smithi001.stdout:Running transaction 2024-03-25T17:55:38.148 INFO:teuthology.orchestra.run.smithi001.stdout: Preparing : 1/1 2024-03-25T17:55:38.256 INFO:teuthology.orchestra.run.smithi001.stdout: Installing : ceph-test-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_ 1/1 2024-03-25T17:55:45.789 INFO:teuthology.orchestra.run.smithi001.stdout: Running scriptlet: ceph-test-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_ 1/1 2024-03-25T17:55:46.041 INFO:teuthology.orchestra.run.smithi001.stdout: Verifying : ceph-test-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_ 1/1 2024-03-25T17:55:46.041 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-25T17:55:46.041 INFO:teuthology.orchestra.run.smithi001.stdout:Installed: 2024-03-25T17:55:46.041 INFO:teuthology.orchestra.run.smithi001.stdout: ceph-test-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T17:55:46.041 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-25T17:55:46.041 INFO:teuthology.orchestra.run.smithi001.stdout:Complete! 2024-03-25T17:55:46.371 DEBUG:teuthology.orchestra.run.smithi001:> sudo yum -y install ceph-base-debuginfo 2024-03-25T17:55:46.947 INFO:teuthology.orchestra.run.smithi001.stdout:Last metadata expiration check: 1:22:57 ago on Mon 25 Mar 2024 04:32:49 PM UTC. 2024-03-25T17:55:47.046 INFO:teuthology.orchestra.run.smithi001.stdout:Package ceph-base-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_64 is already installed. 2024-03-25T17:55:47.101 INFO:teuthology.orchestra.run.smithi001.stdout:Dependencies resolved. 2024-03-25T17:55:47.102 INFO:teuthology.orchestra.run.smithi001.stdout:Nothing to do. 2024-03-25T17:55:47.103 INFO:teuthology.orchestra.run.smithi001.stdout:Complete! 2024-03-25T17:55:47.171 DEBUG:teuthology.orchestra.run.smithi001:> sudo yum -y install ceph-mgr-debuginfo 2024-03-25T17:55:47.745 INFO:teuthology.orchestra.run.smithi001.stdout:Last metadata expiration check: 1:22:58 ago on Mon 25 Mar 2024 04:32:49 PM UTC. 2024-03-25T17:55:47.900 INFO:teuthology.orchestra.run.smithi001.stdout:Dependencies resolved. 2024-03-25T17:55:47.901 INFO:teuthology.orchestra.run.smithi001.stdout:================================================================================ 2024-03-25T17:55:47.901 INFO:teuthology.orchestra.run.smithi001.stdout: Package Arch Version Repo Size 2024-03-25T17:55:47.901 INFO:teuthology.orchestra.run.smithi001.stdout:================================================================================ 2024-03-25T17:55:47.901 INFO:teuthology.orchestra.run.smithi001.stdout:Installing: 2024-03-25T17:55:47.901 INFO:teuthology.orchestra.run.smithi001.stdout: ceph-mgr-debuginfo x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 20 M 2024-03-25T17:55:47.902 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-25T17:55:47.902 INFO:teuthology.orchestra.run.smithi001.stdout:Transaction Summary 2024-03-25T17:55:47.902 INFO:teuthology.orchestra.run.smithi001.stdout:================================================================================ 2024-03-25T17:55:47.902 INFO:teuthology.orchestra.run.smithi001.stdout:Install 1 Package 2024-03-25T17:55:47.902 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-25T17:55:47.902 INFO:teuthology.orchestra.run.smithi001.stdout:Total download size: 20 M 2024-03-25T17:55:47.902 INFO:teuthology.orchestra.run.smithi001.stdout:Installed size: 103 M 2024-03-25T17:55:47.902 INFO:teuthology.orchestra.run.smithi001.stdout:Downloading Packages: 2024-03-25T17:56:33.520 INFO:teuthology.orchestra.run.smithi001.stdout:ceph-mgr-debuginfo-19.0.0-1439.g5dcb190d.el9.x8 442 kB/s | 20 MB 00:45 2024-03-25T17:56:33.521 INFO:teuthology.orchestra.run.smithi001.stdout:-------------------------------------------------------------------------------- 2024-03-25T17:56:33.521 INFO:teuthology.orchestra.run.smithi001.stdout:Total 442 kB/s | 20 MB 00:45 2024-03-25T17:56:33.521 INFO:teuthology.orchestra.run.smithi001.stdout:Running transaction check 2024-03-25T17:56:33.524 INFO:teuthology.orchestra.run.smithi001.stdout:Transaction check succeeded. 2024-03-25T17:56:33.524 INFO:teuthology.orchestra.run.smithi001.stdout:Running transaction test 2024-03-25T17:56:33.680 INFO:teuthology.orchestra.run.smithi001.stdout:Transaction test succeeded. 2024-03-25T17:56:33.680 INFO:teuthology.orchestra.run.smithi001.stdout:Running transaction 2024-03-25T17:56:36.030 INFO:teuthology.orchestra.run.smithi001.stdout: Preparing : 1/1 2024-03-25T17:56:36.145 INFO:teuthology.orchestra.run.smithi001.stdout: Installing : ceph-mgr-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_6 1/1 2024-03-25T17:56:37.071 INFO:teuthology.orchestra.run.smithi001.stdout: Running scriptlet: ceph-mgr-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_6 1/1 2024-03-25T17:56:37.340 INFO:teuthology.orchestra.run.smithi001.stdout: Verifying : ceph-mgr-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_6 1/1 2024-03-25T17:56:37.340 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-25T17:56:37.340 INFO:teuthology.orchestra.run.smithi001.stdout:Installed: 2024-03-25T17:56:37.341 INFO:teuthology.orchestra.run.smithi001.stdout: ceph-mgr-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T17:56:37.341 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-25T17:56:37.341 INFO:teuthology.orchestra.run.smithi001.stdout:Complete! 2024-03-25T17:56:37.583 DEBUG:teuthology.orchestra.run.smithi001:> sudo yum -y install ceph-mds-debuginfo 2024-03-25T17:56:38.158 INFO:teuthology.orchestra.run.smithi001.stdout:Last metadata expiration check: 1:23:49 ago on Mon 25 Mar 2024 04:32:49 PM UTC. 2024-03-25T17:56:38.316 INFO:teuthology.orchestra.run.smithi001.stdout:Dependencies resolved. 2024-03-25T17:56:38.317 INFO:teuthology.orchestra.run.smithi001.stdout:================================================================================ 2024-03-25T17:56:38.317 INFO:teuthology.orchestra.run.smithi001.stdout: Package Arch Version Repo Size 2024-03-25T17:56:38.317 INFO:teuthology.orchestra.run.smithi001.stdout:================================================================================ 2024-03-25T17:56:38.317 INFO:teuthology.orchestra.run.smithi001.stdout:Installing: 2024-03-25T17:56:38.317 INFO:teuthology.orchestra.run.smithi001.stdout: ceph-mds-debuginfo x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 48 M 2024-03-25T17:56:38.317 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-25T17:56:38.317 INFO:teuthology.orchestra.run.smithi001.stdout:Transaction Summary 2024-03-25T17:56:38.317 INFO:teuthology.orchestra.run.smithi001.stdout:================================================================================ 2024-03-25T17:56:38.317 INFO:teuthology.orchestra.run.smithi001.stdout:Install 1 Package 2024-03-25T17:56:38.317 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-25T17:56:38.318 INFO:teuthology.orchestra.run.smithi001.stdout:Total download size: 48 M 2024-03-25T17:56:38.318 INFO:teuthology.orchestra.run.smithi001.stdout:Installed size: 173 M 2024-03-25T17:56:38.318 INFO:teuthology.orchestra.run.smithi001.stdout:Downloading Packages: 2024-03-25T17:56:55.266 INFO:teuthology.orchestra.run.smithi115.stdout:ceph-osd-debuginfo-19.0.0-1439.g5dcb190d.el9.x8 459 kB/s | 287 MB 10:40 2024-03-25T17:56:55.267 INFO:teuthology.orchestra.run.smithi115.stdout:-------------------------------------------------------------------------------- 2024-03-25T17:56:55.267 INFO:teuthology.orchestra.run.smithi115.stdout:Total 459 kB/s | 287 MB 10:40 2024-03-25T17:56:55.267 INFO:teuthology.orchestra.run.smithi115.stdout:Running transaction check 2024-03-25T17:56:55.271 INFO:teuthology.orchestra.run.smithi115.stdout:Transaction check succeeded. 2024-03-25T17:56:55.271 INFO:teuthology.orchestra.run.smithi115.stdout:Running transaction test 2024-03-25T17:56:56.978 INFO:teuthology.orchestra.run.smithi115.stdout:Transaction test succeeded. 2024-03-25T17:56:56.978 INFO:teuthology.orchestra.run.smithi115.stdout:Running transaction 2024-03-25T17:57:26.339 INFO:teuthology.orchestra.run.smithi115.stdout: Preparing : 1/1 2024-03-25T17:57:26.443 INFO:teuthology.orchestra.run.smithi115.stdout: Installing : ceph-osd-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_6 1/1 2024-03-25T17:57:33.995 INFO:teuthology.orchestra.run.smithi115.stdout: Running scriptlet: ceph-osd-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_6 1/1 2024-03-25T17:57:34.278 INFO:teuthology.orchestra.run.smithi115.stdout: Verifying : ceph-osd-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_6 1/1 2024-03-25T17:57:34.278 INFO:teuthology.orchestra.run.smithi115.stdout: 2024-03-25T17:57:34.278 INFO:teuthology.orchestra.run.smithi115.stdout:Installed: 2024-03-25T17:57:34.278 INFO:teuthology.orchestra.run.smithi115.stdout: ceph-osd-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T17:57:34.279 INFO:teuthology.orchestra.run.smithi115.stdout: 2024-03-25T17:57:34.279 INFO:teuthology.orchestra.run.smithi115.stdout:Complete! 2024-03-25T17:57:34.463 DEBUG:teuthology.orchestra.run.smithi115:> sudo yum -y install ceph-fuse-debuginfo 2024-03-25T17:57:35.035 INFO:teuthology.orchestra.run.smithi115.stdout:Last metadata expiration check: 1:24:50 ago on Mon 25 Mar 2024 04:32:45 PM UTC. 2024-03-25T17:57:35.182 INFO:teuthology.orchestra.run.smithi115.stdout:Dependencies resolved. 2024-03-25T17:57:35.183 INFO:teuthology.orchestra.run.smithi115.stdout:================================================================================ 2024-03-25T17:57:35.183 INFO:teuthology.orchestra.run.smithi115.stdout: Package Arch Version Repo Size 2024-03-25T17:57:35.183 INFO:teuthology.orchestra.run.smithi115.stdout:================================================================================ 2024-03-25T17:57:35.183 INFO:teuthology.orchestra.run.smithi115.stdout:Installing: 2024-03-25T17:57:35.184 INFO:teuthology.orchestra.run.smithi115.stdout: ceph-fuse-debuginfo x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 5.9 M 2024-03-25T17:57:35.184 INFO:teuthology.orchestra.run.smithi115.stdout: 2024-03-25T17:57:35.184 INFO:teuthology.orchestra.run.smithi115.stdout:Transaction Summary 2024-03-25T17:57:35.184 INFO:teuthology.orchestra.run.smithi115.stdout:================================================================================ 2024-03-25T17:57:35.184 INFO:teuthology.orchestra.run.smithi115.stdout:Install 1 Package 2024-03-25T17:57:35.184 INFO:teuthology.orchestra.run.smithi115.stdout: 2024-03-25T17:57:35.184 INFO:teuthology.orchestra.run.smithi115.stdout:Total download size: 5.9 M 2024-03-25T17:57:35.184 INFO:teuthology.orchestra.run.smithi115.stdout:Installed size: 28 M 2024-03-25T17:57:35.184 INFO:teuthology.orchestra.run.smithi115.stdout:Downloading Packages: 2024-03-25T17:57:56.975 INFO:teuthology.orchestra.run.smithi115.stdout:ceph-fuse-debuginfo-19.0.0-1439.g5dcb190d.el9.x 279 kB/s | 5.9 MB 00:21 2024-03-25T17:57:56.976 INFO:teuthology.orchestra.run.smithi115.stdout:-------------------------------------------------------------------------------- 2024-03-25T17:57:56.976 INFO:teuthology.orchestra.run.smithi115.stdout:Total 279 kB/s | 5.9 MB 00:21 2024-03-25T17:57:56.977 INFO:teuthology.orchestra.run.smithi115.stdout:Running transaction check 2024-03-25T17:57:56.980 INFO:teuthology.orchestra.run.smithi115.stdout:Transaction check succeeded. 2024-03-25T17:57:56.981 INFO:teuthology.orchestra.run.smithi115.stdout:Running transaction test 2024-03-25T17:57:57.053 INFO:teuthology.orchestra.run.smithi115.stdout:Transaction test succeeded. 2024-03-25T17:57:57.054 INFO:teuthology.orchestra.run.smithi115.stdout:Running transaction 2024-03-25T17:57:57.787 INFO:teuthology.orchestra.run.smithi115.stdout: Preparing : 1/1 2024-03-25T17:57:57.932 INFO:teuthology.orchestra.run.smithi115.stdout: Installing : ceph-fuse-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_ 1/1 2024-03-25T17:57:58.533 INFO:teuthology.orchestra.run.smithi115.stdout: Running scriptlet: ceph-fuse-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_ 1/1 2024-03-25T17:57:58.790 INFO:teuthology.orchestra.run.smithi115.stdout: Verifying : ceph-fuse-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_ 1/1 2024-03-25T17:57:58.790 INFO:teuthology.orchestra.run.smithi115.stdout: 2024-03-25T17:57:58.790 INFO:teuthology.orchestra.run.smithi115.stdout:Installed: 2024-03-25T17:57:58.790 INFO:teuthology.orchestra.run.smithi115.stdout: ceph-fuse-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T17:57:58.790 INFO:teuthology.orchestra.run.smithi115.stdout: 2024-03-25T17:57:58.791 INFO:teuthology.orchestra.run.smithi115.stdout:Complete! 2024-03-25T17:57:58.967 DEBUG:teuthology.orchestra.run.smithi115:> sudo yum -y install librados-devel-debuginfo 2024-03-25T17:57:59.524 INFO:teuthology.orchestra.run.smithi115.stdout:Last metadata expiration check: 1:25:14 ago on Mon 25 Mar 2024 04:32:45 PM UTC. 2024-03-25T17:57:59.671 INFO:teuthology.orchestra.run.smithi115.stdout:Dependencies resolved. 2024-03-25T17:57:59.671 INFO:teuthology.orchestra.run.smithi115.stdout:================================================================================ 2024-03-25T17:57:59.672 INFO:teuthology.orchestra.run.smithi115.stdout: Package Arch Version Repo Size 2024-03-25T17:57:59.672 INFO:teuthology.orchestra.run.smithi115.stdout:================================================================================ 2024-03-25T17:57:59.672 INFO:teuthology.orchestra.run.smithi115.stdout:Installing: 2024-03-25T17:57:59.672 INFO:teuthology.orchestra.run.smithi115.stdout: librados-devel-debuginfo x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 109 k 2024-03-25T17:57:59.672 INFO:teuthology.orchestra.run.smithi115.stdout: 2024-03-25T17:57:59.672 INFO:teuthology.orchestra.run.smithi115.stdout:Transaction Summary 2024-03-25T17:57:59.672 INFO:teuthology.orchestra.run.smithi115.stdout:================================================================================ 2024-03-25T17:57:59.672 INFO:teuthology.orchestra.run.smithi115.stdout:Install 1 Package 2024-03-25T17:57:59.672 INFO:teuthology.orchestra.run.smithi115.stdout: 2024-03-25T17:57:59.672 INFO:teuthology.orchestra.run.smithi115.stdout:Total download size: 109 k 2024-03-25T17:57:59.672 INFO:teuthology.orchestra.run.smithi115.stdout:Installed size: 440 k 2024-03-25T17:57:59.672 INFO:teuthology.orchestra.run.smithi115.stdout:Downloading Packages: 2024-03-25T17:58:00.394 INFO:teuthology.orchestra.run.smithi115.stdout:librados-devel-debuginfo-19.0.0-1439.g5dcb190d. 151 kB/s | 109 kB 00:00 2024-03-25T17:58:00.395 INFO:teuthology.orchestra.run.smithi115.stdout:-------------------------------------------------------------------------------- 2024-03-25T17:58:00.395 INFO:teuthology.orchestra.run.smithi115.stdout:Total 150 kB/s | 109 kB 00:00 2024-03-25T17:58:00.395 INFO:teuthology.orchestra.run.smithi115.stdout:Running transaction check 2024-03-25T17:58:00.399 INFO:teuthology.orchestra.run.smithi115.stdout:Transaction check succeeded. 2024-03-25T17:58:00.399 INFO:teuthology.orchestra.run.smithi115.stdout:Running transaction test 2024-03-25T17:58:00.434 INFO:teuthology.orchestra.run.smithi115.stdout:Transaction test succeeded. 2024-03-25T17:58:00.434 INFO:teuthology.orchestra.run.smithi115.stdout:Running transaction 2024-03-25T17:58:00.551 INFO:teuthology.orchestra.run.smithi115.stdout: Preparing : 1/1 2024-03-25T17:58:00.667 INFO:teuthology.orchestra.run.smithi115.stdout: Installing : librados-devel-debuginfo-2:19.0.0-1439.g5dcb190d.el9 1/1 2024-03-25T17:58:01.074 INFO:teuthology.orchestra.run.smithi115.stdout: Running scriptlet: librados-devel-debuginfo-2:19.0.0-1439.g5dcb190d.el9 1/1 2024-03-25T17:58:01.340 INFO:teuthology.orchestra.run.smithi115.stdout: Verifying : librados-devel-debuginfo-2:19.0.0-1439.g5dcb190d.el9 1/1 2024-03-25T17:58:01.340 INFO:teuthology.orchestra.run.smithi115.stdout: 2024-03-25T17:58:01.340 INFO:teuthology.orchestra.run.smithi115.stdout:Installed: 2024-03-25T17:58:01.340 INFO:teuthology.orchestra.run.smithi115.stdout: librados-devel-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T17:58:01.340 INFO:teuthology.orchestra.run.smithi115.stdout: 2024-03-25T17:58:01.341 INFO:teuthology.orchestra.run.smithi115.stdout:Complete! 2024-03-25T17:58:01.497 DEBUG:teuthology.orchestra.run.smithi115:> sudo yum -y install libcephfs2-debuginfo 2024-03-25T17:58:02.062 INFO:teuthology.orchestra.run.smithi115.stdout:Last metadata expiration check: 1:25:17 ago on Mon 25 Mar 2024 04:32:45 PM UTC. 2024-03-25T17:58:02.210 INFO:teuthology.orchestra.run.smithi115.stdout:Dependencies resolved. 2024-03-25T17:58:02.211 INFO:teuthology.orchestra.run.smithi115.stdout:================================================================================ 2024-03-25T17:58:02.211 INFO:teuthology.orchestra.run.smithi115.stdout: Package Arch Version Repo Size 2024-03-25T17:58:02.211 INFO:teuthology.orchestra.run.smithi115.stdout:================================================================================ 2024-03-25T17:58:02.211 INFO:teuthology.orchestra.run.smithi115.stdout:Installing: 2024-03-25T17:58:02.211 INFO:teuthology.orchestra.run.smithi115.stdout: libcephfs2-debuginfo x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 5.4 M 2024-03-25T17:58:02.211 INFO:teuthology.orchestra.run.smithi115.stdout: 2024-03-25T17:58:02.211 INFO:teuthology.orchestra.run.smithi115.stdout:Transaction Summary 2024-03-25T17:58:02.211 INFO:teuthology.orchestra.run.smithi115.stdout:================================================================================ 2024-03-25T17:58:02.211 INFO:teuthology.orchestra.run.smithi115.stdout:Install 1 Package 2024-03-25T17:58:02.211 INFO:teuthology.orchestra.run.smithi115.stdout: 2024-03-25T17:58:02.211 INFO:teuthology.orchestra.run.smithi115.stdout:Total download size: 5.4 M 2024-03-25T17:58:02.211 INFO:teuthology.orchestra.run.smithi115.stdout:Installed size: 25 M 2024-03-25T17:58:02.211 INFO:teuthology.orchestra.run.smithi115.stdout:Downloading Packages: 2024-03-25T17:58:16.120 INFO:teuthology.orchestra.run.smithi115.stdout:libcephfs2-debuginfo-19.0.0-1439.g5dcb190d.el9. 394 kB/s | 5.4 MB 00:13 2024-03-25T17:58:16.121 INFO:teuthology.orchestra.run.smithi115.stdout:-------------------------------------------------------------------------------- 2024-03-25T17:58:16.121 INFO:teuthology.orchestra.run.smithi115.stdout:Total 394 kB/s | 5.4 MB 00:13 2024-03-25T17:58:16.121 INFO:teuthology.orchestra.run.smithi115.stdout:Running transaction check 2024-03-25T17:58:16.125 INFO:teuthology.orchestra.run.smithi115.stdout:Transaction check succeeded. 2024-03-25T17:58:16.125 INFO:teuthology.orchestra.run.smithi115.stdout:Running transaction test 2024-03-25T17:58:16.193 INFO:teuthology.orchestra.run.smithi115.stdout:Transaction test succeeded. 2024-03-25T17:58:16.193 INFO:teuthology.orchestra.run.smithi115.stdout:Running transaction 2024-03-25T17:58:16.868 INFO:teuthology.orchestra.run.smithi115.stdout: Preparing : 1/1 2024-03-25T17:58:16.968 INFO:teuthology.orchestra.run.smithi115.stdout: Installing : libcephfs2-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86 1/1 2024-03-25T17:58:17.510 INFO:teuthology.orchestra.run.smithi115.stdout: Running scriptlet: libcephfs2-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86 1/1 2024-03-25T17:58:17.760 INFO:teuthology.orchestra.run.smithi115.stdout: Verifying : libcephfs2-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86 1/1 2024-03-25T17:58:17.760 INFO:teuthology.orchestra.run.smithi115.stdout: 2024-03-25T17:58:17.760 INFO:teuthology.orchestra.run.smithi115.stdout:Installed: 2024-03-25T17:58:17.760 INFO:teuthology.orchestra.run.smithi115.stdout: libcephfs2-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T17:58:17.760 INFO:teuthology.orchestra.run.smithi115.stdout: 2024-03-25T17:58:17.761 INFO:teuthology.orchestra.run.smithi115.stdout:Complete! 2024-03-25T17:58:17.941 DEBUG:teuthology.orchestra.run.smithi115:> sudo yum -y install librados2-debuginfo 2024-03-25T17:58:18.504 INFO:teuthology.orchestra.run.smithi115.stdout:Last metadata expiration check: 1:25:33 ago on Mon 25 Mar 2024 04:32:45 PM UTC. 2024-03-25T17:58:18.652 INFO:teuthology.orchestra.run.smithi115.stdout:Dependencies resolved. 2024-03-25T17:58:18.653 INFO:teuthology.orchestra.run.smithi115.stdout:================================================================================ 2024-03-25T17:58:18.653 INFO:teuthology.orchestra.run.smithi115.stdout: Package Arch Version Repo Size 2024-03-25T17:58:18.653 INFO:teuthology.orchestra.run.smithi115.stdout:================================================================================ 2024-03-25T17:58:18.653 INFO:teuthology.orchestra.run.smithi115.stdout:Installing: 2024-03-25T17:58:18.653 INFO:teuthology.orchestra.run.smithi115.stdout: librados2-debuginfo x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 44 M 2024-03-25T17:58:18.653 INFO:teuthology.orchestra.run.smithi115.stdout: 2024-03-25T17:58:18.653 INFO:teuthology.orchestra.run.smithi115.stdout:Transaction Summary 2024-03-25T17:58:18.653 INFO:teuthology.orchestra.run.smithi115.stdout:================================================================================ 2024-03-25T17:58:18.653 INFO:teuthology.orchestra.run.smithi115.stdout:Install 1 Package 2024-03-25T17:58:18.653 INFO:teuthology.orchestra.run.smithi115.stdout: 2024-03-25T17:58:18.654 INFO:teuthology.orchestra.run.smithi115.stdout:Total download size: 44 M 2024-03-25T17:58:18.654 INFO:teuthology.orchestra.run.smithi115.stdout:Installed size: 253 M 2024-03-25T17:58:18.654 INFO:teuthology.orchestra.run.smithi115.stdout:Downloading Packages: 2024-03-25T17:58:52.474 INFO:teuthology.orchestra.run.smithi001.stdout:ceph-mds-debuginfo-19.0.0-1439.g5dcb190d.el9.x8 364 kB/s | 48 MB 02:14 2024-03-25T17:58:52.475 INFO:teuthology.orchestra.run.smithi001.stdout:-------------------------------------------------------------------------------- 2024-03-25T17:58:52.475 INFO:teuthology.orchestra.run.smithi001.stdout:Total 364 kB/s | 48 MB 02:14 2024-03-25T17:58:52.475 INFO:teuthology.orchestra.run.smithi001.stdout:Running transaction check 2024-03-25T17:58:52.478 INFO:teuthology.orchestra.run.smithi001.stdout:Transaction check succeeded. 2024-03-25T17:58:52.478 INFO:teuthology.orchestra.run.smithi001.stdout:Running transaction test 2024-03-25T17:58:52.805 INFO:teuthology.orchestra.run.smithi001.stdout:Transaction test succeeded. 2024-03-25T17:58:52.805 INFO:teuthology.orchestra.run.smithi001.stdout:Running transaction 2024-03-25T17:58:57.810 INFO:teuthology.orchestra.run.smithi001.stdout: Preparing : 1/1 2024-03-25T17:58:57.909 INFO:teuthology.orchestra.run.smithi001.stdout: Installing : ceph-mds-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_6 1/1 2024-03-25T17:58:59.373 INFO:teuthology.orchestra.run.smithi001.stdout: Running scriptlet: ceph-mds-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_6 1/1 2024-03-25T17:58:59.635 INFO:teuthology.orchestra.run.smithi001.stdout: Verifying : ceph-mds-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_6 1/1 2024-03-25T17:58:59.635 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-25T17:58:59.635 INFO:teuthology.orchestra.run.smithi001.stdout:Installed: 2024-03-25T17:58:59.635 INFO:teuthology.orchestra.run.smithi001.stdout: ceph-mds-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T17:58:59.635 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-25T17:58:59.635 INFO:teuthology.orchestra.run.smithi001.stdout:Complete! 2024-03-25T17:58:59.977 DEBUG:teuthology.orchestra.run.smithi001:> sudo yum -y install ceph-mon-debuginfo 2024-03-25T17:59:00.553 INFO:teuthology.orchestra.run.smithi001.stdout:Last metadata expiration check: 1:26:11 ago on Mon 25 Mar 2024 04:32:49 PM UTC. 2024-03-25T17:59:00.707 INFO:teuthology.orchestra.run.smithi001.stdout:Dependencies resolved. 2024-03-25T17:59:00.708 INFO:teuthology.orchestra.run.smithi001.stdout:================================================================================ 2024-03-25T17:59:00.708 INFO:teuthology.orchestra.run.smithi001.stdout: Package Arch Version Repo Size 2024-03-25T17:59:00.708 INFO:teuthology.orchestra.run.smithi001.stdout:================================================================================ 2024-03-25T17:59:00.708 INFO:teuthology.orchestra.run.smithi001.stdout:Installing: 2024-03-25T17:59:00.709 INFO:teuthology.orchestra.run.smithi001.stdout: ceph-mon-debuginfo x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 99 M 2024-03-25T17:59:00.709 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-25T17:59:00.709 INFO:teuthology.orchestra.run.smithi001.stdout:Transaction Summary 2024-03-25T17:59:00.709 INFO:teuthology.orchestra.run.smithi001.stdout:================================================================================ 2024-03-25T17:59:00.709 INFO:teuthology.orchestra.run.smithi001.stdout:Install 1 Package 2024-03-25T17:59:00.709 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-25T17:59:00.709 INFO:teuthology.orchestra.run.smithi001.stdout:Total download size: 99 M 2024-03-25T17:59:00.709 INFO:teuthology.orchestra.run.smithi001.stdout:Installed size: 449 M 2024-03-25T17:59:00.709 INFO:teuthology.orchestra.run.smithi001.stdout:Downloading Packages: 2024-03-25T18:00:49.827 INFO:teuthology.orchestra.run.smithi115.stdout:librados2-debuginfo-19.0.0-1439.g5dcb190d.el9.x 300 kB/s | 44 MB 02:31 2024-03-25T18:00:49.827 INFO:teuthology.orchestra.run.smithi115.stdout:-------------------------------------------------------------------------------- 2024-03-25T18:00:49.827 INFO:teuthology.orchestra.run.smithi115.stdout:Total 300 kB/s | 44 MB 02:31 2024-03-25T18:00:49.827 INFO:teuthology.orchestra.run.smithi115.stdout:Running transaction check 2024-03-25T18:00:49.830 INFO:teuthology.orchestra.run.smithi115.stdout:Transaction check succeeded. 2024-03-25T18:00:49.830 INFO:teuthology.orchestra.run.smithi115.stdout:Running transaction test 2024-03-25T18:00:50.127 INFO:teuthology.orchestra.run.smithi115.stdout:Transaction test succeeded. 2024-03-25T18:00:50.128 INFO:teuthology.orchestra.run.smithi115.stdout:Running transaction 2024-03-25T18:00:55.158 INFO:teuthology.orchestra.run.smithi115.stdout: Preparing : 1/1 2024-03-25T18:00:55.263 INFO:teuthology.orchestra.run.smithi115.stdout: Installing : librados2-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_ 1/1 2024-03-25T18:00:57.057 INFO:teuthology.orchestra.run.smithi115.stdout: Running scriptlet: librados2-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_ 1/1 2024-03-25T18:00:57.308 INFO:teuthology.orchestra.run.smithi115.stdout: Verifying : librados2-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_ 1/1 2024-03-25T18:00:57.309 INFO:teuthology.orchestra.run.smithi115.stdout: 2024-03-25T18:00:57.309 INFO:teuthology.orchestra.run.smithi115.stdout:Installed: 2024-03-25T18:00:57.309 INFO:teuthology.orchestra.run.smithi115.stdout: librados2-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T18:00:57.309 INFO:teuthology.orchestra.run.smithi115.stdout: 2024-03-25T18:00:57.309 INFO:teuthology.orchestra.run.smithi115.stdout:Complete! 2024-03-25T18:00:57.478 DEBUG:teuthology.orchestra.run.smithi115:> sudo yum -y install librbd1-debuginfo 2024-03-25T18:00:58.041 INFO:teuthology.orchestra.run.smithi115.stdout:Last metadata expiration check: 1:28:13 ago on Mon 25 Mar 2024 04:32:45 PM UTC. 2024-03-25T18:00:58.191 INFO:teuthology.orchestra.run.smithi115.stdout:Dependencies resolved. 2024-03-25T18:00:58.192 INFO:teuthology.orchestra.run.smithi115.stdout:================================================================================ 2024-03-25T18:00:58.192 INFO:teuthology.orchestra.run.smithi115.stdout: Package Arch Version Repo Size 2024-03-25T18:00:58.192 INFO:teuthology.orchestra.run.smithi115.stdout:================================================================================ 2024-03-25T18:00:58.192 INFO:teuthology.orchestra.run.smithi115.stdout:Installing: 2024-03-25T18:00:58.192 INFO:teuthology.orchestra.run.smithi115.stdout: librbd1-debuginfo x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 52 M 2024-03-25T18:00:58.192 INFO:teuthology.orchestra.run.smithi115.stdout: 2024-03-25T18:00:58.192 INFO:teuthology.orchestra.run.smithi115.stdout:Transaction Summary 2024-03-25T18:00:58.192 INFO:teuthology.orchestra.run.smithi115.stdout:================================================================================ 2024-03-25T18:00:58.192 INFO:teuthology.orchestra.run.smithi115.stdout:Install 1 Package 2024-03-25T18:00:58.192 INFO:teuthology.orchestra.run.smithi115.stdout: 2024-03-25T18:00:58.192 INFO:teuthology.orchestra.run.smithi115.stdout:Total download size: 52 M 2024-03-25T18:00:58.193 INFO:teuthology.orchestra.run.smithi115.stdout:Installed size: 308 M 2024-03-25T18:00:58.193 INFO:teuthology.orchestra.run.smithi115.stdout:Downloading Packages: 2024-03-25T18:02:58.905 INFO:teuthology.orchestra.run.smithi115.stdout:librbd1-debuginfo-19.0.0-1439.g5dcb190d.el9.x86 440 kB/s | 52 MB 02:00 2024-03-25T18:02:58.905 INFO:teuthology.orchestra.run.smithi115.stdout:-------------------------------------------------------------------------------- 2024-03-25T18:02:58.906 INFO:teuthology.orchestra.run.smithi115.stdout:Total 440 kB/s | 52 MB 02:00 2024-03-25T18:02:58.906 INFO:teuthology.orchestra.run.smithi115.stdout:Running transaction check 2024-03-25T18:02:58.909 INFO:teuthology.orchestra.run.smithi115.stdout:Transaction check succeeded. 2024-03-25T18:02:58.909 INFO:teuthology.orchestra.run.smithi115.stdout:Running transaction test 2024-03-25T18:02:59.251 INFO:teuthology.orchestra.run.smithi115.stdout:Transaction test succeeded. 2024-03-25T18:02:59.252 INFO:teuthology.orchestra.run.smithi115.stdout:Running transaction 2024-03-25T18:03:05.230 INFO:teuthology.orchestra.run.smithi115.stdout: Preparing : 1/1 2024-03-25T18:03:05.361 INFO:teuthology.orchestra.run.smithi115.stdout: Installing : librbd1-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/1 2024-03-25T18:03:07.550 INFO:teuthology.orchestra.run.smithi115.stdout: Running scriptlet: librbd1-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/1 2024-03-25T18:03:07.813 INFO:teuthology.orchestra.run.smithi115.stdout: Verifying : librbd1-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/1 2024-03-25T18:03:07.813 INFO:teuthology.orchestra.run.smithi115.stdout: 2024-03-25T18:03:07.813 INFO:teuthology.orchestra.run.smithi115.stdout:Installed: 2024-03-25T18:03:07.813 INFO:teuthology.orchestra.run.smithi115.stdout: librbd1-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T18:03:07.814 INFO:teuthology.orchestra.run.smithi115.stdout: 2024-03-25T18:03:07.814 INFO:teuthology.orchestra.run.smithi115.stdout:Complete! 2024-03-25T18:03:07.993 DEBUG:teuthology.orchestra.run.smithi115:> sudo yum -y install python3-cephfs-debuginfo 2024-03-25T18:03:08.552 INFO:teuthology.orchestra.run.smithi115.stdout:Last metadata expiration check: 1:30:23 ago on Mon 25 Mar 2024 04:32:45 PM UTC. 2024-03-25T18:03:08.702 INFO:teuthology.orchestra.run.smithi115.stdout:Dependencies resolved. 2024-03-25T18:03:08.702 INFO:teuthology.orchestra.run.smithi115.stdout:================================================================================ 2024-03-25T18:03:08.702 INFO:teuthology.orchestra.run.smithi115.stdout: Package Arch Version Repo Size 2024-03-25T18:03:08.703 INFO:teuthology.orchestra.run.smithi115.stdout:================================================================================ 2024-03-25T18:03:08.703 INFO:teuthology.orchestra.run.smithi115.stdout:Installing: 2024-03-25T18:03:08.703 INFO:teuthology.orchestra.run.smithi115.stdout: python3-cephfs-debuginfo x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 615 k 2024-03-25T18:03:08.703 INFO:teuthology.orchestra.run.smithi115.stdout: 2024-03-25T18:03:08.703 INFO:teuthology.orchestra.run.smithi115.stdout:Transaction Summary 2024-03-25T18:03:08.703 INFO:teuthology.orchestra.run.smithi115.stdout:================================================================================ 2024-03-25T18:03:08.703 INFO:teuthology.orchestra.run.smithi115.stdout:Install 1 Package 2024-03-25T18:03:08.703 INFO:teuthology.orchestra.run.smithi115.stdout: 2024-03-25T18:03:08.704 INFO:teuthology.orchestra.run.smithi115.stdout:Total download size: 615 k 2024-03-25T18:03:08.704 INFO:teuthology.orchestra.run.smithi115.stdout:Installed size: 2.1 M 2024-03-25T18:03:08.704 INFO:teuthology.orchestra.run.smithi115.stdout:Downloading Packages: 2024-03-25T18:03:10.989 INFO:teuthology.orchestra.run.smithi115.stdout:python3-cephfs-debuginfo-19.0.0-1439.g5dcb190d. 269 kB/s | 615 kB 00:02 2024-03-25T18:03:10.989 INFO:teuthology.orchestra.run.smithi115.stdout:-------------------------------------------------------------------------------- 2024-03-25T18:03:10.990 INFO:teuthology.orchestra.run.smithi115.stdout:Total 269 kB/s | 615 kB 00:02 2024-03-25T18:03:10.990 INFO:teuthology.orchestra.run.smithi115.stdout:Running transaction check 2024-03-25T18:03:10.994 INFO:teuthology.orchestra.run.smithi115.stdout:Transaction check succeeded. 2024-03-25T18:03:10.994 INFO:teuthology.orchestra.run.smithi115.stdout:Running transaction test 2024-03-25T18:03:11.036 INFO:teuthology.orchestra.run.smithi115.stdout:Transaction test succeeded. 2024-03-25T18:03:11.036 INFO:teuthology.orchestra.run.smithi115.stdout:Running transaction 2024-03-25T18:03:11.202 INFO:teuthology.orchestra.run.smithi115.stdout: Preparing : 1/1 2024-03-25T18:03:11.297 INFO:teuthology.orchestra.run.smithi115.stdout: Installing : python3-cephfs-debuginfo-2:19.0.0-1439.g5dcb190d.el9 1/1 2024-03-25T18:03:11.719 INFO:teuthology.orchestra.run.smithi115.stdout: Running scriptlet: python3-cephfs-debuginfo-2:19.0.0-1439.g5dcb190d.el9 1/1 2024-03-25T18:03:11.962 INFO:teuthology.orchestra.run.smithi115.stdout: Verifying : python3-cephfs-debuginfo-2:19.0.0-1439.g5dcb190d.el9 1/1 2024-03-25T18:03:11.962 INFO:teuthology.orchestra.run.smithi115.stdout: 2024-03-25T18:03:11.962 INFO:teuthology.orchestra.run.smithi115.stdout:Installed: 2024-03-25T18:03:11.962 INFO:teuthology.orchestra.run.smithi115.stdout: python3-cephfs-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T18:03:11.963 INFO:teuthology.orchestra.run.smithi115.stdout: 2024-03-25T18:03:11.963 INFO:teuthology.orchestra.run.smithi115.stdout:Complete! 2024-03-25T18:03:12.140 DEBUG:teuthology.orchestra.run.smithi115:> sudo yum -y install python3-rados-debuginfo 2024-03-25T18:03:12.706 INFO:teuthology.orchestra.run.smithi115.stdout:Last metadata expiration check: 1:30:27 ago on Mon 25 Mar 2024 04:32:45 PM UTC. 2024-03-25T18:03:12.856 INFO:teuthology.orchestra.run.smithi115.stdout:Dependencies resolved. 2024-03-25T18:03:12.856 INFO:teuthology.orchestra.run.smithi115.stdout:================================================================================ 2024-03-25T18:03:12.856 INFO:teuthology.orchestra.run.smithi115.stdout: Package Arch Version Repo Size 2024-03-25T18:03:12.857 INFO:teuthology.orchestra.run.smithi115.stdout:================================================================================ 2024-03-25T18:03:12.857 INFO:teuthology.orchestra.run.smithi115.stdout:Installing: 2024-03-25T18:03:12.857 INFO:teuthology.orchestra.run.smithi115.stdout: python3-rados-debuginfo x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 771 k 2024-03-25T18:03:12.857 INFO:teuthology.orchestra.run.smithi115.stdout: 2024-03-25T18:03:12.857 INFO:teuthology.orchestra.run.smithi115.stdout:Transaction Summary 2024-03-25T18:03:12.857 INFO:teuthology.orchestra.run.smithi115.stdout:================================================================================ 2024-03-25T18:03:12.857 INFO:teuthology.orchestra.run.smithi115.stdout:Install 1 Package 2024-03-25T18:03:12.857 INFO:teuthology.orchestra.run.smithi115.stdout: 2024-03-25T18:03:12.857 INFO:teuthology.orchestra.run.smithi115.stdout:Total download size: 771 k 2024-03-25T18:03:12.857 INFO:teuthology.orchestra.run.smithi115.stdout:Installed size: 2.7 M 2024-03-25T18:03:12.857 INFO:teuthology.orchestra.run.smithi115.stdout:Downloading Packages: 2024-03-25T18:03:16.216 INFO:teuthology.orchestra.run.smithi115.stdout:python3-rados-debuginfo-19.0.0-1439.g5dcb190d.e 230 kB/s | 771 kB 00:03 2024-03-25T18:03:16.216 INFO:teuthology.orchestra.run.smithi115.stdout:-------------------------------------------------------------------------------- 2024-03-25T18:03:16.217 INFO:teuthology.orchestra.run.smithi115.stdout:Total 229 kB/s | 771 kB 00:03 2024-03-25T18:03:16.217 INFO:teuthology.orchestra.run.smithi115.stdout:Running transaction check 2024-03-25T18:03:16.221 INFO:teuthology.orchestra.run.smithi115.stdout:Transaction check succeeded. 2024-03-25T18:03:16.221 INFO:teuthology.orchestra.run.smithi115.stdout:Running transaction test 2024-03-25T18:03:16.262 INFO:teuthology.orchestra.run.smithi115.stdout:Transaction test succeeded. 2024-03-25T18:03:16.263 INFO:teuthology.orchestra.run.smithi115.stdout:Running transaction 2024-03-25T18:03:16.448 INFO:teuthology.orchestra.run.smithi115.stdout: Preparing : 1/1 2024-03-25T18:03:16.548 INFO:teuthology.orchestra.run.smithi115.stdout: Installing : python3-rados-debuginfo-2:19.0.0-1439.g5dcb190d.el9. 1/1 2024-03-25T18:03:16.958 INFO:teuthology.orchestra.run.smithi115.stdout: Running scriptlet: python3-rados-debuginfo-2:19.0.0-1439.g5dcb190d.el9. 1/1 2024-03-25T18:03:17.220 INFO:teuthology.orchestra.run.smithi115.stdout: Verifying : python3-rados-debuginfo-2:19.0.0-1439.g5dcb190d.el9. 1/1 2024-03-25T18:03:17.220 INFO:teuthology.orchestra.run.smithi115.stdout: 2024-03-25T18:03:17.220 INFO:teuthology.orchestra.run.smithi115.stdout:Installed: 2024-03-25T18:03:17.220 INFO:teuthology.orchestra.run.smithi115.stdout: python3-rados-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T18:03:17.220 INFO:teuthology.orchestra.run.smithi115.stdout: 2024-03-25T18:03:17.220 INFO:teuthology.orchestra.run.smithi115.stdout:Complete! 2024-03-25T18:03:17.405 DEBUG:teuthology.orchestra.run.smithi115:> sudo yum -y install python3-rbd-debuginfo 2024-03-25T18:03:17.963 INFO:teuthology.orchestra.run.smithi115.stdout:Last metadata expiration check: 1:30:32 ago on Mon 25 Mar 2024 04:32:45 PM UTC. 2024-03-25T18:03:18.113 INFO:teuthology.orchestra.run.smithi115.stdout:Dependencies resolved. 2024-03-25T18:03:18.114 INFO:teuthology.orchestra.run.smithi115.stdout:================================================================================ 2024-03-25T18:03:18.114 INFO:teuthology.orchestra.run.smithi115.stdout: Package Arch Version Repo Size 2024-03-25T18:03:18.114 INFO:teuthology.orchestra.run.smithi115.stdout:================================================================================ 2024-03-25T18:03:18.114 INFO:teuthology.orchestra.run.smithi115.stdout:Installing: 2024-03-25T18:03:18.114 INFO:teuthology.orchestra.run.smithi115.stdout: python3-rbd-debuginfo x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 1.1 M 2024-03-25T18:03:18.114 INFO:teuthology.orchestra.run.smithi115.stdout: 2024-03-25T18:03:18.114 INFO:teuthology.orchestra.run.smithi115.stdout:Transaction Summary 2024-03-25T18:03:18.115 INFO:teuthology.orchestra.run.smithi115.stdout:================================================================================ 2024-03-25T18:03:18.115 INFO:teuthology.orchestra.run.smithi115.stdout:Install 1 Package 2024-03-25T18:03:18.115 INFO:teuthology.orchestra.run.smithi115.stdout: 2024-03-25T18:03:18.115 INFO:teuthology.orchestra.run.smithi115.stdout:Total download size: 1.1 M 2024-03-25T18:03:18.115 INFO:teuthology.orchestra.run.smithi115.stdout:Installed size: 4.3 M 2024-03-25T18:03:18.115 INFO:teuthology.orchestra.run.smithi115.stdout:Downloading Packages: 2024-03-25T18:03:22.369 INFO:teuthology.orchestra.run.smithi115.stdout:python3-rbd-debuginfo-19.0.0-1439.g5dcb190d.el9 271 kB/s | 1.1 MB 00:04 2024-03-25T18:03:22.370 INFO:teuthology.orchestra.run.smithi115.stdout:-------------------------------------------------------------------------------- 2024-03-25T18:03:22.370 INFO:teuthology.orchestra.run.smithi115.stdout:Total 270 kB/s | 1.1 MB 00:04 2024-03-25T18:03:22.370 INFO:teuthology.orchestra.run.smithi115.stdout:Running transaction check 2024-03-25T18:03:22.374 INFO:teuthology.orchestra.run.smithi115.stdout:Transaction check succeeded. 2024-03-25T18:03:22.374 INFO:teuthology.orchestra.run.smithi115.stdout:Running transaction test 2024-03-25T18:03:22.417 INFO:teuthology.orchestra.run.smithi115.stdout:Transaction test succeeded. 2024-03-25T18:03:22.418 INFO:teuthology.orchestra.run.smithi115.stdout:Running transaction 2024-03-25T18:03:22.652 INFO:teuthology.orchestra.run.smithi115.stdout: Preparing : 1/1 2024-03-25T18:03:22.743 INFO:teuthology.orchestra.run.smithi115.stdout: Installing : python3-rbd-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x8 1/1 2024-03-25T18:03:23.150 INFO:teuthology.orchestra.run.smithi115.stdout: Running scriptlet: python3-rbd-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x8 1/1 2024-03-25T18:03:23.392 INFO:teuthology.orchestra.run.smithi115.stdout: Verifying : python3-rbd-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x8 1/1 2024-03-25T18:03:23.392 INFO:teuthology.orchestra.run.smithi115.stdout: 2024-03-25T18:03:23.392 INFO:teuthology.orchestra.run.smithi115.stdout:Installed: 2024-03-25T18:03:23.392 INFO:teuthology.orchestra.run.smithi115.stdout: python3-rbd-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T18:03:23.392 INFO:teuthology.orchestra.run.smithi115.stdout: 2024-03-25T18:03:23.392 INFO:teuthology.orchestra.run.smithi115.stdout:Complete! 2024-03-25T18:03:23.567 DEBUG:teuthology.orchestra.run.smithi115:> sudo yum -y install python3-rgw-debuginfo 2024-03-25T18:03:24.132 INFO:teuthology.orchestra.run.smithi115.stdout:Last metadata expiration check: 1:30:39 ago on Mon 25 Mar 2024 04:32:45 PM UTC. 2024-03-25T18:03:24.282 INFO:teuthology.orchestra.run.smithi115.stdout:Dependencies resolved. 2024-03-25T18:03:24.282 INFO:teuthology.orchestra.run.smithi115.stdout:================================================================================ 2024-03-25T18:03:24.283 INFO:teuthology.orchestra.run.smithi115.stdout: Package Arch Version Repo Size 2024-03-25T18:03:24.283 INFO:teuthology.orchestra.run.smithi115.stdout:================================================================================ 2024-03-25T18:03:24.283 INFO:teuthology.orchestra.run.smithi115.stdout:Installing: 2024-03-25T18:03:24.283 INFO:teuthology.orchestra.run.smithi115.stdout: python3-rgw-debuginfo x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 337 k 2024-03-25T18:03:24.283 INFO:teuthology.orchestra.run.smithi115.stdout: 2024-03-25T18:03:24.283 INFO:teuthology.orchestra.run.smithi115.stdout:Transaction Summary 2024-03-25T18:03:24.283 INFO:teuthology.orchestra.run.smithi115.stdout:================================================================================ 2024-03-25T18:03:24.283 INFO:teuthology.orchestra.run.smithi115.stdout:Install 1 Package 2024-03-25T18:03:24.283 INFO:teuthology.orchestra.run.smithi115.stdout: 2024-03-25T18:03:24.283 INFO:teuthology.orchestra.run.smithi115.stdout:Total download size: 337 k 2024-03-25T18:03:24.283 INFO:teuthology.orchestra.run.smithi115.stdout:Installed size: 1.0 M 2024-03-25T18:03:24.283 INFO:teuthology.orchestra.run.smithi115.stdout:Downloading Packages: 2024-03-25T18:03:26.213 INFO:teuthology.orchestra.run.smithi115.stdout:python3-rgw-debuginfo-19.0.0-1439.g5dcb190d.el9 175 kB/s | 337 kB 00:01 2024-03-25T18:03:26.213 INFO:teuthology.orchestra.run.smithi115.stdout:-------------------------------------------------------------------------------- 2024-03-25T18:03:26.213 INFO:teuthology.orchestra.run.smithi115.stdout:Total 175 kB/s | 337 kB 00:01 2024-03-25T18:03:26.214 INFO:teuthology.orchestra.run.smithi115.stdout:Running transaction check 2024-03-25T18:03:26.217 INFO:teuthology.orchestra.run.smithi115.stdout:Transaction check succeeded. 2024-03-25T18:03:26.218 INFO:teuthology.orchestra.run.smithi115.stdout:Running transaction test 2024-03-25T18:03:26.257 INFO:teuthology.orchestra.run.smithi115.stdout:Transaction test succeeded. 2024-03-25T18:03:26.257 INFO:teuthology.orchestra.run.smithi115.stdout:Running transaction 2024-03-25T18:03:26.394 INFO:teuthology.orchestra.run.smithi115.stdout: Preparing : 1/1 2024-03-25T18:03:26.511 INFO:teuthology.orchestra.run.smithi115.stdout: Installing : python3-rgw-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x8 1/1 2024-03-25T18:03:26.897 INFO:teuthology.orchestra.run.smithi115.stdout: Running scriptlet: python3-rgw-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x8 1/1 2024-03-25T18:03:27.150 INFO:teuthology.orchestra.run.smithi115.stdout: Verifying : python3-rgw-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x8 1/1 2024-03-25T18:03:27.150 INFO:teuthology.orchestra.run.smithi115.stdout: 2024-03-25T18:03:27.150 INFO:teuthology.orchestra.run.smithi115.stdout:Installed: 2024-03-25T18:03:27.151 INFO:teuthology.orchestra.run.smithi115.stdout: python3-rgw-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T18:03:27.151 INFO:teuthology.orchestra.run.smithi115.stdout: 2024-03-25T18:03:27.151 INFO:teuthology.orchestra.run.smithi115.stdout:Complete! 2024-03-25T18:03:27.335 DEBUG:teuthology.orchestra.run.smithi115:> sudo yum -y install rbd-fuse-debuginfo 2024-03-25T18:03:27.901 INFO:teuthology.orchestra.run.smithi115.stdout:Last metadata expiration check: 1:30:42 ago on Mon 25 Mar 2024 04:32:45 PM UTC. 2024-03-25T18:03:28.051 INFO:teuthology.orchestra.run.smithi115.stdout:Dependencies resolved. 2024-03-25T18:03:28.052 INFO:teuthology.orchestra.run.smithi115.stdout:================================================================================ 2024-03-25T18:03:28.052 INFO:teuthology.orchestra.run.smithi115.stdout: Package Arch Version Repo Size 2024-03-25T18:03:28.052 INFO:teuthology.orchestra.run.smithi115.stdout:================================================================================ 2024-03-25T18:03:28.052 INFO:teuthology.orchestra.run.smithi115.stdout:Installing: 2024-03-25T18:03:28.052 INFO:teuthology.orchestra.run.smithi115.stdout: rbd-fuse-debuginfo x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 509 k 2024-03-25T18:03:28.052 INFO:teuthology.orchestra.run.smithi115.stdout: 2024-03-25T18:03:28.052 INFO:teuthology.orchestra.run.smithi115.stdout:Transaction Summary 2024-03-25T18:03:28.053 INFO:teuthology.orchestra.run.smithi115.stdout:================================================================================ 2024-03-25T18:03:28.053 INFO:teuthology.orchestra.run.smithi115.stdout:Install 1 Package 2024-03-25T18:03:28.053 INFO:teuthology.orchestra.run.smithi115.stdout: 2024-03-25T18:03:28.053 INFO:teuthology.orchestra.run.smithi115.stdout:Total download size: 509 k 2024-03-25T18:03:28.053 INFO:teuthology.orchestra.run.smithi115.stdout:Installed size: 2.5 M 2024-03-25T18:03:28.053 INFO:teuthology.orchestra.run.smithi115.stdout:Downloading Packages: 2024-03-25T18:03:29.632 INFO:teuthology.orchestra.run.smithi115.stdout:rbd-fuse-debuginfo-19.0.0-1439.g5dcb190d.el9.x8 322 kB/s | 509 kB 00:01 2024-03-25T18:03:29.633 INFO:teuthology.orchestra.run.smithi115.stdout:-------------------------------------------------------------------------------- 2024-03-25T18:03:29.634 INFO:teuthology.orchestra.run.smithi115.stdout:Total 322 kB/s | 509 kB 00:01 2024-03-25T18:03:29.634 INFO:teuthology.orchestra.run.smithi115.stdout:Running transaction check 2024-03-25T18:03:29.638 INFO:teuthology.orchestra.run.smithi115.stdout:Transaction check succeeded. 2024-03-25T18:03:29.638 INFO:teuthology.orchestra.run.smithi115.stdout:Running transaction test 2024-03-25T18:03:29.677 INFO:teuthology.orchestra.run.smithi115.stdout:Transaction test succeeded. 2024-03-25T18:03:29.677 INFO:teuthology.orchestra.run.smithi115.stdout:Running transaction 2024-03-25T18:03:29.840 INFO:teuthology.orchestra.run.smithi115.stdout: Preparing : 1/1 2024-03-25T18:03:29.956 INFO:teuthology.orchestra.run.smithi115.stdout: Installing : rbd-fuse-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_6 1/1 2024-03-25T18:03:30.360 INFO:teuthology.orchestra.run.smithi115.stdout: Running scriptlet: rbd-fuse-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_6 1/1 2024-03-25T18:03:30.619 INFO:teuthology.orchestra.run.smithi115.stdout: Verifying : rbd-fuse-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_6 1/1 2024-03-25T18:03:30.619 INFO:teuthology.orchestra.run.smithi115.stdout: 2024-03-25T18:03:30.619 INFO:teuthology.orchestra.run.smithi115.stdout:Installed: 2024-03-25T18:03:30.619 INFO:teuthology.orchestra.run.smithi115.stdout: rbd-fuse-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T18:03:30.619 INFO:teuthology.orchestra.run.smithi115.stdout: 2024-03-25T18:03:30.619 INFO:teuthology.orchestra.run.smithi115.stdout:Complete! 2024-03-25T18:03:30.812 DEBUG:teuthology.orchestra.run.smithi115:> sudo yum -y install rbd-mirror-debuginfo 2024-03-25T18:03:31.371 INFO:teuthology.orchestra.run.smithi115.stdout:Last metadata expiration check: 1:30:46 ago on Mon 25 Mar 2024 04:32:45 PM UTC. 2024-03-25T18:03:31.519 INFO:teuthology.orchestra.run.smithi115.stdout:Dependencies resolved. 2024-03-25T18:03:31.520 INFO:teuthology.orchestra.run.smithi115.stdout:================================================================================ 2024-03-25T18:03:31.520 INFO:teuthology.orchestra.run.smithi115.stdout: Package Arch Version Repo Size 2024-03-25T18:03:31.520 INFO:teuthology.orchestra.run.smithi115.stdout:================================================================================ 2024-03-25T18:03:31.520 INFO:teuthology.orchestra.run.smithi115.stdout:Installing: 2024-03-25T18:03:31.520 INFO:teuthology.orchestra.run.smithi115.stdout: rbd-mirror-debuginfo x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 52 M 2024-03-25T18:03:31.520 INFO:teuthology.orchestra.run.smithi115.stdout: 2024-03-25T18:03:31.520 INFO:teuthology.orchestra.run.smithi115.stdout:Transaction Summary 2024-03-25T18:03:31.521 INFO:teuthology.orchestra.run.smithi115.stdout:================================================================================ 2024-03-25T18:03:31.521 INFO:teuthology.orchestra.run.smithi115.stdout:Install 1 Package 2024-03-25T18:03:31.521 INFO:teuthology.orchestra.run.smithi115.stdout: 2024-03-25T18:03:31.521 INFO:teuthology.orchestra.run.smithi115.stdout:Total download size: 52 M 2024-03-25T18:03:31.521 INFO:teuthology.orchestra.run.smithi115.stdout:Installed size: 298 M 2024-03-25T18:03:31.521 INFO:teuthology.orchestra.run.smithi115.stdout:Downloading Packages: 2024-03-25T18:04:51.820 INFO:teuthology.orchestra.run.smithi115.stdout:rbd-mirror-debuginfo-19.0.0-1439.g5dcb190d.el9. 657 kB/s | 52 MB 01:20 2024-03-25T18:04:51.821 INFO:teuthology.orchestra.run.smithi115.stdout:-------------------------------------------------------------------------------- 2024-03-25T18:04:51.821 INFO:teuthology.orchestra.run.smithi115.stdout:Total 657 kB/s | 52 MB 01:20 2024-03-25T18:04:51.821 INFO:teuthology.orchestra.run.smithi115.stdout:Running transaction check 2024-03-25T18:04:51.824 INFO:teuthology.orchestra.run.smithi115.stdout:Transaction check succeeded. 2024-03-25T18:04:51.824 INFO:teuthology.orchestra.run.smithi115.stdout:Running transaction test 2024-03-25T18:04:52.157 INFO:teuthology.orchestra.run.smithi115.stdout:Transaction test succeeded. 2024-03-25T18:04:52.157 INFO:teuthology.orchestra.run.smithi115.stdout:Running transaction 2024-03-25T18:04:58.049 INFO:teuthology.orchestra.run.smithi115.stdout: Preparing : 1/1 2024-03-25T18:04:58.291 INFO:teuthology.orchestra.run.smithi115.stdout: Installing : rbd-mirror-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86 1/1 2024-03-25T18:05:00.359 INFO:teuthology.orchestra.run.smithi115.stdout: Running scriptlet: rbd-mirror-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86 1/1 2024-03-25T18:05:00.628 INFO:teuthology.orchestra.run.smithi115.stdout: Verifying : rbd-mirror-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86 1/1 2024-03-25T18:05:00.629 INFO:teuthology.orchestra.run.smithi115.stdout: 2024-03-25T18:05:00.629 INFO:teuthology.orchestra.run.smithi115.stdout:Installed: 2024-03-25T18:05:00.629 INFO:teuthology.orchestra.run.smithi115.stdout: rbd-mirror-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T18:05:00.629 INFO:teuthology.orchestra.run.smithi115.stdout: 2024-03-25T18:05:00.629 INFO:teuthology.orchestra.run.smithi115.stdout:Complete! 2024-03-25T18:05:00.805 DEBUG:teuthology.orchestra.run.smithi115:> sudo yum -y install rbd-nbd-debuginfo 2024-03-25T18:05:01.371 INFO:teuthology.orchestra.run.smithi115.stdout:Last metadata expiration check: 1:32:16 ago on Mon 25 Mar 2024 04:32:45 PM UTC. 2024-03-25T18:05:01.519 INFO:teuthology.orchestra.run.smithi115.stdout:Dependencies resolved. 2024-03-25T18:05:01.520 INFO:teuthology.orchestra.run.smithi115.stdout:================================================================================ 2024-03-25T18:05:01.520 INFO:teuthology.orchestra.run.smithi115.stdout: Package Arch Version Repo Size 2024-03-25T18:05:01.520 INFO:teuthology.orchestra.run.smithi115.stdout:================================================================================ 2024-03-25T18:05:01.520 INFO:teuthology.orchestra.run.smithi115.stdout:Installing: 2024-03-25T18:05:01.520 INFO:teuthology.orchestra.run.smithi115.stdout: rbd-nbd-debuginfo x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 1.3 M 2024-03-25T18:05:01.520 INFO:teuthology.orchestra.run.smithi115.stdout: 2024-03-25T18:05:01.520 INFO:teuthology.orchestra.run.smithi115.stdout:Transaction Summary 2024-03-25T18:05:01.520 INFO:teuthology.orchestra.run.smithi115.stdout:================================================================================ 2024-03-25T18:05:01.520 INFO:teuthology.orchestra.run.smithi115.stdout:Install 1 Package 2024-03-25T18:05:01.520 INFO:teuthology.orchestra.run.smithi115.stdout: 2024-03-25T18:05:01.520 INFO:teuthology.orchestra.run.smithi115.stdout:Total download size: 1.3 M 2024-03-25T18:05:01.520 INFO:teuthology.orchestra.run.smithi115.stdout:Installed size: 5.5 M 2024-03-25T18:05:01.520 INFO:teuthology.orchestra.run.smithi115.stdout:Downloading Packages: 2024-03-25T18:05:05.562 INFO:teuthology.orchestra.run.smithi115.stdout:rbd-nbd-debuginfo-19.0.0-1439.g5dcb190d.el9.x86 317 kB/s | 1.3 MB 00:04 2024-03-25T18:05:05.563 INFO:teuthology.orchestra.run.smithi115.stdout:-------------------------------------------------------------------------------- 2024-03-25T18:05:05.563 INFO:teuthology.orchestra.run.smithi115.stdout:Total 317 kB/s | 1.3 MB 00:04 2024-03-25T18:05:05.563 INFO:teuthology.orchestra.run.smithi115.stdout:Running transaction check 2024-03-25T18:05:05.567 INFO:teuthology.orchestra.run.smithi115.stdout:Transaction check succeeded. 2024-03-25T18:05:05.567 INFO:teuthology.orchestra.run.smithi115.stdout:Running transaction test 2024-03-25T18:05:05.611 INFO:teuthology.orchestra.run.smithi115.stdout:Transaction test succeeded. 2024-03-25T18:05:05.612 INFO:teuthology.orchestra.run.smithi115.stdout:Running transaction 2024-03-25T18:05:05.863 INFO:teuthology.orchestra.run.smithi115.stdout: Preparing : 1/1 2024-03-25T18:05:05.969 INFO:teuthology.orchestra.run.smithi115.stdout: Installing : rbd-nbd-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/1 2024-03-25T18:05:06.407 INFO:teuthology.orchestra.run.smithi115.stdout: Running scriptlet: rbd-nbd-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/1 2024-03-25T18:05:06.660 INFO:teuthology.orchestra.run.smithi115.stdout: Verifying : rbd-nbd-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/1 2024-03-25T18:05:06.661 INFO:teuthology.orchestra.run.smithi115.stdout: 2024-03-25T18:05:06.661 INFO:teuthology.orchestra.run.smithi115.stdout:Installed: 2024-03-25T18:05:06.661 INFO:teuthology.orchestra.run.smithi115.stdout: rbd-nbd-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T18:05:06.661 INFO:teuthology.orchestra.run.smithi115.stdout: 2024-03-25T18:05:06.661 INFO:teuthology.orchestra.run.smithi115.stdout:Complete! 2024-03-25T18:05:06.832 DEBUG:teuthology.parallel:result is None 2024-03-25T18:05:32.594 INFO:teuthology.orchestra.run.smithi001.stdout:ceph-mon-debuginfo-19.0.0-1439.g5dcb190d.el9.x8 257 kB/s | 99 MB 06:31 2024-03-25T18:05:32.595 INFO:teuthology.orchestra.run.smithi001.stdout:-------------------------------------------------------------------------------- 2024-03-25T18:05:32.595 INFO:teuthology.orchestra.run.smithi001.stdout:Total 257 kB/s | 99 MB 06:31 2024-03-25T18:05:32.595 INFO:teuthology.orchestra.run.smithi001.stdout:Running transaction check 2024-03-25T18:05:32.598 INFO:teuthology.orchestra.run.smithi001.stdout:Transaction check succeeded. 2024-03-25T18:05:32.599 INFO:teuthology.orchestra.run.smithi001.stdout:Running transaction test 2024-03-25T18:05:33.239 INFO:teuthology.orchestra.run.smithi001.stdout:Transaction test succeeded. 2024-03-25T18:05:33.239 INFO:teuthology.orchestra.run.smithi001.stdout:Running transaction 2024-03-25T18:05:44.105 INFO:teuthology.orchestra.run.smithi001.stdout: Preparing : 1/1 2024-03-25T18:05:44.211 INFO:teuthology.orchestra.run.smithi001.stdout: Installing : ceph-mon-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_6 1/1 2024-03-25T18:05:47.090 INFO:teuthology.orchestra.run.smithi001.stdout: Running scriptlet: ceph-mon-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_6 1/1 2024-03-25T18:05:47.351 INFO:teuthology.orchestra.run.smithi001.stdout: Verifying : ceph-mon-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_6 1/1 2024-03-25T18:05:47.351 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-25T18:05:47.351 INFO:teuthology.orchestra.run.smithi001.stdout:Installed: 2024-03-25T18:05:47.351 INFO:teuthology.orchestra.run.smithi001.stdout: ceph-mon-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T18:05:47.351 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-25T18:05:47.351 INFO:teuthology.orchestra.run.smithi001.stdout:Complete! 2024-03-25T18:05:47.566 DEBUG:teuthology.orchestra.run.smithi001:> sudo yum -y install ceph-osd-debuginfo 2024-03-25T18:05:48.142 INFO:teuthology.orchestra.run.smithi001.stdout:Last metadata expiration check: 1:32:59 ago on Mon 25 Mar 2024 04:32:49 PM UTC. 2024-03-25T18:05:48.296 INFO:teuthology.orchestra.run.smithi001.stdout:Dependencies resolved. 2024-03-25T18:05:48.297 INFO:teuthology.orchestra.run.smithi001.stdout:================================================================================ 2024-03-25T18:05:48.297 INFO:teuthology.orchestra.run.smithi001.stdout: Package Arch Version Repo Size 2024-03-25T18:05:48.297 INFO:teuthology.orchestra.run.smithi001.stdout:================================================================================ 2024-03-25T18:05:48.297 INFO:teuthology.orchestra.run.smithi001.stdout:Installing: 2024-03-25T18:05:48.297 INFO:teuthology.orchestra.run.smithi001.stdout: ceph-osd-debuginfo x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 287 M 2024-03-25T18:05:48.297 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-25T18:05:48.298 INFO:teuthology.orchestra.run.smithi001.stdout:Transaction Summary 2024-03-25T18:05:48.298 INFO:teuthology.orchestra.run.smithi001.stdout:================================================================================ 2024-03-25T18:05:48.298 INFO:teuthology.orchestra.run.smithi001.stdout:Install 1 Package 2024-03-25T18:05:48.298 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-25T18:05:48.298 INFO:teuthology.orchestra.run.smithi001.stdout:Total download size: 287 M 2024-03-25T18:05:48.298 INFO:teuthology.orchestra.run.smithi001.stdout:Installed size: 1.2 G 2024-03-25T18:05:48.298 INFO:teuthology.orchestra.run.smithi001.stdout:Downloading Packages: 2024-03-25T18:18:48.278 INFO:teuthology.orchestra.run.smithi001.stdout:ceph-osd-debuginfo-19.0.0-1439.g5dcb190d.el9.x8 377 kB/s | 287 MB 12:59 2024-03-25T18:18:48.279 INFO:teuthology.orchestra.run.smithi001.stdout:-------------------------------------------------------------------------------- 2024-03-25T18:18:48.279 INFO:teuthology.orchestra.run.smithi001.stdout:Total 377 kB/s | 287 MB 12:59 2024-03-25T18:18:48.279 INFO:teuthology.orchestra.run.smithi001.stdout:Running transaction check 2024-03-25T18:18:48.282 INFO:teuthology.orchestra.run.smithi001.stdout:Transaction check succeeded. 2024-03-25T18:18:48.282 INFO:teuthology.orchestra.run.smithi001.stdout:Running transaction test 2024-03-25T18:18:50.073 INFO:teuthology.orchestra.run.smithi001.stdout:Transaction test succeeded. 2024-03-25T18:18:50.073 INFO:teuthology.orchestra.run.smithi001.stdout:Running transaction 2024-03-25T18:19:21.273 INFO:teuthology.orchestra.run.smithi001.stdout: Preparing : 1/1 2024-03-25T18:19:21.377 INFO:teuthology.orchestra.run.smithi001.stdout: Installing : ceph-osd-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_6 1/1 2024-03-25T18:19:28.890 INFO:teuthology.orchestra.run.smithi001.stdout: Running scriptlet: ceph-osd-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_6 1/1 2024-03-25T18:19:29.163 INFO:teuthology.orchestra.run.smithi001.stdout: Verifying : ceph-osd-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_6 1/1 2024-03-25T18:19:29.163 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-25T18:19:29.163 INFO:teuthology.orchestra.run.smithi001.stdout:Installed: 2024-03-25T18:19:29.163 INFO:teuthology.orchestra.run.smithi001.stdout: ceph-osd-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T18:19:29.163 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-25T18:19:29.163 INFO:teuthology.orchestra.run.smithi001.stdout:Complete! 2024-03-25T18:19:29.505 DEBUG:teuthology.orchestra.run.smithi001:> sudo yum -y install ceph-fuse-debuginfo 2024-03-25T18:19:30.082 INFO:teuthology.orchestra.run.smithi001.stdout:Last metadata expiration check: 1:46:41 ago on Mon 25 Mar 2024 04:32:49 PM UTC. 2024-03-25T18:19:30.236 INFO:teuthology.orchestra.run.smithi001.stdout:Dependencies resolved. 2024-03-25T18:19:30.237 INFO:teuthology.orchestra.run.smithi001.stdout:================================================================================ 2024-03-25T18:19:30.237 INFO:teuthology.orchestra.run.smithi001.stdout: Package Arch Version Repo Size 2024-03-25T18:19:30.237 INFO:teuthology.orchestra.run.smithi001.stdout:================================================================================ 2024-03-25T18:19:30.237 INFO:teuthology.orchestra.run.smithi001.stdout:Installing: 2024-03-25T18:19:30.237 INFO:teuthology.orchestra.run.smithi001.stdout: ceph-fuse-debuginfo x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 5.9 M 2024-03-25T18:19:30.237 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-25T18:19:30.237 INFO:teuthology.orchestra.run.smithi001.stdout:Transaction Summary 2024-03-25T18:19:30.237 INFO:teuthology.orchestra.run.smithi001.stdout:================================================================================ 2024-03-25T18:19:30.237 INFO:teuthology.orchestra.run.smithi001.stdout:Install 1 Package 2024-03-25T18:19:30.237 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-25T18:19:30.238 INFO:teuthology.orchestra.run.smithi001.stdout:Total download size: 5.9 M 2024-03-25T18:19:30.238 INFO:teuthology.orchestra.run.smithi001.stdout:Installed size: 28 M 2024-03-25T18:19:30.238 INFO:teuthology.orchestra.run.smithi001.stdout:Downloading Packages: 2024-03-25T18:19:43.943 INFO:teuthology.orchestra.run.smithi001.stdout:ceph-fuse-debuginfo-19.0.0-1439.g5dcb190d.el9.x 444 kB/s | 5.9 MB 00:13 2024-03-25T18:19:43.943 INFO:teuthology.orchestra.run.smithi001.stdout:-------------------------------------------------------------------------------- 2024-03-25T18:19:43.944 INFO:teuthology.orchestra.run.smithi001.stdout:Total 444 kB/s | 5.9 MB 00:13 2024-03-25T18:19:43.944 INFO:teuthology.orchestra.run.smithi001.stdout:Running transaction check 2024-03-25T18:19:43.948 INFO:teuthology.orchestra.run.smithi001.stdout:Transaction check succeeded. 2024-03-25T18:19:43.948 INFO:teuthology.orchestra.run.smithi001.stdout:Running transaction test 2024-03-25T18:19:44.023 INFO:teuthology.orchestra.run.smithi001.stdout:Transaction test succeeded. 2024-03-25T18:19:44.023 INFO:teuthology.orchestra.run.smithi001.stdout:Running transaction 2024-03-25T18:19:44.796 INFO:teuthology.orchestra.run.smithi001.stdout: Preparing : 1/1 2024-03-25T18:19:44.910 INFO:teuthology.orchestra.run.smithi001.stdout: Installing : ceph-fuse-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_ 1/1 2024-03-25T18:19:45.467 INFO:teuthology.orchestra.run.smithi001.stdout: Running scriptlet: ceph-fuse-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_ 1/1 2024-03-25T18:19:45.727 INFO:teuthology.orchestra.run.smithi001.stdout: Verifying : ceph-fuse-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_ 1/1 2024-03-25T18:19:45.727 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-25T18:19:45.727 INFO:teuthology.orchestra.run.smithi001.stdout:Installed: 2024-03-25T18:19:45.727 INFO:teuthology.orchestra.run.smithi001.stdout: ceph-fuse-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T18:19:45.727 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-25T18:19:45.727 INFO:teuthology.orchestra.run.smithi001.stdout:Complete! 2024-03-25T18:19:45.951 DEBUG:teuthology.orchestra.run.smithi001:> sudo yum -y install librados-devel-debuginfo 2024-03-25T18:19:46.526 INFO:teuthology.orchestra.run.smithi001.stdout:Last metadata expiration check: 1:46:57 ago on Mon 25 Mar 2024 04:32:49 PM UTC. 2024-03-25T18:19:46.681 INFO:teuthology.orchestra.run.smithi001.stdout:Dependencies resolved. 2024-03-25T18:19:46.682 INFO:teuthology.orchestra.run.smithi001.stdout:================================================================================ 2024-03-25T18:19:46.682 INFO:teuthology.orchestra.run.smithi001.stdout: Package Arch Version Repo Size 2024-03-25T18:19:46.682 INFO:teuthology.orchestra.run.smithi001.stdout:================================================================================ 2024-03-25T18:19:46.682 INFO:teuthology.orchestra.run.smithi001.stdout:Installing: 2024-03-25T18:19:46.682 INFO:teuthology.orchestra.run.smithi001.stdout: librados-devel-debuginfo x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 109 k 2024-03-25T18:19:46.683 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-25T18:19:46.683 INFO:teuthology.orchestra.run.smithi001.stdout:Transaction Summary 2024-03-25T18:19:46.683 INFO:teuthology.orchestra.run.smithi001.stdout:================================================================================ 2024-03-25T18:19:46.683 INFO:teuthology.orchestra.run.smithi001.stdout:Install 1 Package 2024-03-25T18:19:46.683 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-25T18:19:46.683 INFO:teuthology.orchestra.run.smithi001.stdout:Total download size: 109 k 2024-03-25T18:19:46.683 INFO:teuthology.orchestra.run.smithi001.stdout:Installed size: 440 k 2024-03-25T18:19:46.683 INFO:teuthology.orchestra.run.smithi001.stdout:Downloading Packages: 2024-03-25T18:19:47.387 INFO:teuthology.orchestra.run.smithi001.stdout:librados-devel-debuginfo-19.0.0-1439.g5dcb190d. 154 kB/s | 109 kB 00:00 2024-03-25T18:19:47.388 INFO:teuthology.orchestra.run.smithi001.stdout:-------------------------------------------------------------------------------- 2024-03-25T18:19:47.388 INFO:teuthology.orchestra.run.smithi001.stdout:Total 154 kB/s | 109 kB 00:00 2024-03-25T18:19:47.388 INFO:teuthology.orchestra.run.smithi001.stdout:Running transaction check 2024-03-25T18:19:47.392 INFO:teuthology.orchestra.run.smithi001.stdout:Transaction check succeeded. 2024-03-25T18:19:47.392 INFO:teuthology.orchestra.run.smithi001.stdout:Running transaction test 2024-03-25T18:19:47.430 INFO:teuthology.orchestra.run.smithi001.stdout:Transaction test succeeded. 2024-03-25T18:19:47.430 INFO:teuthology.orchestra.run.smithi001.stdout:Running transaction 2024-03-25T18:19:47.550 INFO:teuthology.orchestra.run.smithi001.stdout: Preparing : 1/1 2024-03-25T18:19:47.673 INFO:teuthology.orchestra.run.smithi001.stdout: Installing : librados-devel-debuginfo-2:19.0.0-1439.g5dcb190d.el9 1/1 2024-03-25T18:19:48.039 INFO:teuthology.orchestra.run.smithi001.stdout: Running scriptlet: librados-devel-debuginfo-2:19.0.0-1439.g5dcb190d.el9 1/1 2024-03-25T18:19:48.295 INFO:teuthology.orchestra.run.smithi001.stdout: Verifying : librados-devel-debuginfo-2:19.0.0-1439.g5dcb190d.el9 1/1 2024-03-25T18:19:48.295 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-25T18:19:48.296 INFO:teuthology.orchestra.run.smithi001.stdout:Installed: 2024-03-25T18:19:48.296 INFO:teuthology.orchestra.run.smithi001.stdout: librados-devel-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T18:19:48.296 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-25T18:19:48.296 INFO:teuthology.orchestra.run.smithi001.stdout:Complete! 2024-03-25T18:19:48.515 DEBUG:teuthology.orchestra.run.smithi001:> sudo yum -y install libcephfs2-debuginfo 2024-03-25T18:19:49.091 INFO:teuthology.orchestra.run.smithi001.stdout:Last metadata expiration check: 1:47:00 ago on Mon 25 Mar 2024 04:32:49 PM UTC. 2024-03-25T18:19:49.246 INFO:teuthology.orchestra.run.smithi001.stdout:Dependencies resolved. 2024-03-25T18:19:49.247 INFO:teuthology.orchestra.run.smithi001.stdout:================================================================================ 2024-03-25T18:19:49.247 INFO:teuthology.orchestra.run.smithi001.stdout: Package Arch Version Repo Size 2024-03-25T18:19:49.247 INFO:teuthology.orchestra.run.smithi001.stdout:================================================================================ 2024-03-25T18:19:49.247 INFO:teuthology.orchestra.run.smithi001.stdout:Installing: 2024-03-25T18:19:49.247 INFO:teuthology.orchestra.run.smithi001.stdout: libcephfs2-debuginfo x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 5.4 M 2024-03-25T18:19:49.247 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-25T18:19:49.247 INFO:teuthology.orchestra.run.smithi001.stdout:Transaction Summary 2024-03-25T18:19:49.247 INFO:teuthology.orchestra.run.smithi001.stdout:================================================================================ 2024-03-25T18:19:49.247 INFO:teuthology.orchestra.run.smithi001.stdout:Install 1 Package 2024-03-25T18:19:49.248 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-25T18:19:49.248 INFO:teuthology.orchestra.run.smithi001.stdout:Total download size: 5.4 M 2024-03-25T18:19:49.248 INFO:teuthology.orchestra.run.smithi001.stdout:Installed size: 25 M 2024-03-25T18:19:49.248 INFO:teuthology.orchestra.run.smithi001.stdout:Downloading Packages: 2024-03-25T18:19:58.790 INFO:teuthology.orchestra.run.smithi001.stdout:libcephfs2-debuginfo-19.0.0-1439.g5dcb190d.el9. 574 kB/s | 5.4 MB 00:09 2024-03-25T18:19:58.790 INFO:teuthology.orchestra.run.smithi001.stdout:-------------------------------------------------------------------------------- 2024-03-25T18:19:58.791 INFO:teuthology.orchestra.run.smithi001.stdout:Total 574 kB/s | 5.4 MB 00:09 2024-03-25T18:19:58.791 INFO:teuthology.orchestra.run.smithi001.stdout:Running transaction check 2024-03-25T18:19:58.795 INFO:teuthology.orchestra.run.smithi001.stdout:Transaction check succeeded. 2024-03-25T18:19:58.796 INFO:teuthology.orchestra.run.smithi001.stdout:Running transaction test 2024-03-25T18:19:58.866 INFO:teuthology.orchestra.run.smithi001.stdout:Transaction test succeeded. 2024-03-25T18:19:58.867 INFO:teuthology.orchestra.run.smithi001.stdout:Running transaction 2024-03-25T18:19:59.600 INFO:teuthology.orchestra.run.smithi001.stdout: Preparing : 1/1 2024-03-25T18:19:59.708 INFO:teuthology.orchestra.run.smithi001.stdout: Installing : libcephfs2-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86 1/1 2024-03-25T18:20:00.219 INFO:teuthology.orchestra.run.smithi001.stdout: Running scriptlet: libcephfs2-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86 1/1 2024-03-25T18:20:00.476 INFO:teuthology.orchestra.run.smithi001.stdout: Verifying : libcephfs2-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86 1/1 2024-03-25T18:20:00.476 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-25T18:20:00.476 INFO:teuthology.orchestra.run.smithi001.stdout:Installed: 2024-03-25T18:20:00.476 INFO:teuthology.orchestra.run.smithi001.stdout: libcephfs2-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T18:20:00.476 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-25T18:20:00.476 INFO:teuthology.orchestra.run.smithi001.stdout:Complete! 2024-03-25T18:20:00.687 DEBUG:teuthology.orchestra.run.smithi001:> sudo yum -y install librados2-debuginfo 2024-03-25T18:20:01.263 INFO:teuthology.orchestra.run.smithi001.stdout:Last metadata expiration check: 1:47:12 ago on Mon 25 Mar 2024 04:32:49 PM UTC. 2024-03-25T18:20:01.422 INFO:teuthology.orchestra.run.smithi001.stdout:Dependencies resolved. 2024-03-25T18:20:01.422 INFO:teuthology.orchestra.run.smithi001.stdout:================================================================================ 2024-03-25T18:20:01.422 INFO:teuthology.orchestra.run.smithi001.stdout: Package Arch Version Repo Size 2024-03-25T18:20:01.422 INFO:teuthology.orchestra.run.smithi001.stdout:================================================================================ 2024-03-25T18:20:01.422 INFO:teuthology.orchestra.run.smithi001.stdout:Installing: 2024-03-25T18:20:01.422 INFO:teuthology.orchestra.run.smithi001.stdout: librados2-debuginfo x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 44 M 2024-03-25T18:20:01.423 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-25T18:20:01.423 INFO:teuthology.orchestra.run.smithi001.stdout:Transaction Summary 2024-03-25T18:20:01.423 INFO:teuthology.orchestra.run.smithi001.stdout:================================================================================ 2024-03-25T18:20:01.423 INFO:teuthology.orchestra.run.smithi001.stdout:Install 1 Package 2024-03-25T18:20:01.423 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-25T18:20:01.423 INFO:teuthology.orchestra.run.smithi001.stdout:Total download size: 44 M 2024-03-25T18:20:01.423 INFO:teuthology.orchestra.run.smithi001.stdout:Installed size: 253 M 2024-03-25T18:20:01.423 INFO:teuthology.orchestra.run.smithi001.stdout:Downloading Packages: 2024-03-25T18:21:12.285 INFO:teuthology.orchestra.run.smithi001.stdout:librados2-debuginfo-19.0.0-1439.g5dcb190d.el9.x 640 kB/s | 44 MB 01:10 2024-03-25T18:21:12.286 INFO:teuthology.orchestra.run.smithi001.stdout:-------------------------------------------------------------------------------- 2024-03-25T18:21:12.286 INFO:teuthology.orchestra.run.smithi001.stdout:Total 640 kB/s | 44 MB 01:10 2024-03-25T18:21:12.286 INFO:teuthology.orchestra.run.smithi001.stdout:Running transaction check 2024-03-25T18:21:12.290 INFO:teuthology.orchestra.run.smithi001.stdout:Transaction check succeeded. 2024-03-25T18:21:12.290 INFO:teuthology.orchestra.run.smithi001.stdout:Running transaction test 2024-03-25T18:21:12.603 INFO:teuthology.orchestra.run.smithi001.stdout:Transaction test succeeded. 2024-03-25T18:21:12.603 INFO:teuthology.orchestra.run.smithi001.stdout:Running transaction 2024-03-25T18:21:17.938 INFO:teuthology.orchestra.run.smithi001.stdout: Preparing : 1/1 2024-03-25T18:21:18.047 INFO:teuthology.orchestra.run.smithi001.stdout: Installing : librados2-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_ 1/1 2024-03-25T18:21:19.897 INFO:teuthology.orchestra.run.smithi001.stdout: Running scriptlet: librados2-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_ 1/1 2024-03-25T18:21:20.165 INFO:teuthology.orchestra.run.smithi001.stdout: Verifying : librados2-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_ 1/1 2024-03-25T18:21:20.165 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-25T18:21:20.166 INFO:teuthology.orchestra.run.smithi001.stdout:Installed: 2024-03-25T18:21:20.166 INFO:teuthology.orchestra.run.smithi001.stdout: librados2-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T18:21:20.166 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-25T18:21:20.166 INFO:teuthology.orchestra.run.smithi001.stdout:Complete! 2024-03-25T18:21:20.392 DEBUG:teuthology.orchestra.run.smithi001:> sudo yum -y install librbd1-debuginfo 2024-03-25T18:21:20.968 INFO:teuthology.orchestra.run.smithi001.stdout:Last metadata expiration check: 1:48:31 ago on Mon 25 Mar 2024 04:32:49 PM UTC. 2024-03-25T18:21:21.124 INFO:teuthology.orchestra.run.smithi001.stdout:Dependencies resolved. 2024-03-25T18:21:21.125 INFO:teuthology.orchestra.run.smithi001.stdout:================================================================================ 2024-03-25T18:21:21.126 INFO:teuthology.orchestra.run.smithi001.stdout: Package Arch Version Repo Size 2024-03-25T18:21:21.126 INFO:teuthology.orchestra.run.smithi001.stdout:================================================================================ 2024-03-25T18:21:21.126 INFO:teuthology.orchestra.run.smithi001.stdout:Installing: 2024-03-25T18:21:21.126 INFO:teuthology.orchestra.run.smithi001.stdout: librbd1-debuginfo x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 52 M 2024-03-25T18:21:21.126 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-25T18:21:21.126 INFO:teuthology.orchestra.run.smithi001.stdout:Transaction Summary 2024-03-25T18:21:21.126 INFO:teuthology.orchestra.run.smithi001.stdout:================================================================================ 2024-03-25T18:21:21.126 INFO:teuthology.orchestra.run.smithi001.stdout:Install 1 Package 2024-03-25T18:21:21.126 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-25T18:21:21.126 INFO:teuthology.orchestra.run.smithi001.stdout:Total download size: 52 M 2024-03-25T18:21:21.126 INFO:teuthology.orchestra.run.smithi001.stdout:Installed size: 308 M 2024-03-25T18:21:21.126 INFO:teuthology.orchestra.run.smithi001.stdout:Downloading Packages: 2024-03-25T18:23:17.629 INFO:teuthology.orchestra.run.smithi001.stdout:librbd1-debuginfo-19.0.0-1439.g5dcb190d.el9.x86 456 kB/s | 52 MB 01:56 2024-03-25T18:23:17.629 INFO:teuthology.orchestra.run.smithi001.stdout:-------------------------------------------------------------------------------- 2024-03-25T18:23:17.629 INFO:teuthology.orchestra.run.smithi001.stdout:Total 456 kB/s | 52 MB 01:56 2024-03-25T18:23:17.630 INFO:teuthology.orchestra.run.smithi001.stdout:Running transaction check 2024-03-25T18:23:17.632 INFO:teuthology.orchestra.run.smithi001.stdout:Transaction check succeeded. 2024-03-25T18:23:17.632 INFO:teuthology.orchestra.run.smithi001.stdout:Running transaction test 2024-03-25T18:23:17.991 INFO:teuthology.orchestra.run.smithi001.stdout:Transaction test succeeded. 2024-03-25T18:23:17.991 INFO:teuthology.orchestra.run.smithi001.stdout:Running transaction 2024-03-25T18:23:24.345 INFO:teuthology.orchestra.run.smithi001.stdout: Preparing : 1/1 2024-03-25T18:23:24.446 INFO:teuthology.orchestra.run.smithi001.stdout: Installing : librbd1-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/1 2024-03-25T18:23:26.694 INFO:teuthology.orchestra.run.smithi001.stdout: Running scriptlet: librbd1-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/1 2024-03-25T18:23:26.952 INFO:teuthology.orchestra.run.smithi001.stdout: Verifying : librbd1-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/1 2024-03-25T18:23:26.952 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-25T18:23:26.952 INFO:teuthology.orchestra.run.smithi001.stdout:Installed: 2024-03-25T18:23:26.952 INFO:teuthology.orchestra.run.smithi001.stdout: librbd1-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T18:23:26.952 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-25T18:23:26.953 INFO:teuthology.orchestra.run.smithi001.stdout:Complete! 2024-03-25T18:23:27.166 DEBUG:teuthology.orchestra.run.smithi001:> sudo yum -y install python3-cephfs-debuginfo 2024-03-25T18:23:27.736 INFO:teuthology.orchestra.run.smithi001.stdout:Last metadata expiration check: 1:50:38 ago on Mon 25 Mar 2024 04:32:49 PM UTC. 2024-03-25T18:23:27.892 INFO:teuthology.orchestra.run.smithi001.stdout:Dependencies resolved. 2024-03-25T18:23:27.892 INFO:teuthology.orchestra.run.smithi001.stdout:================================================================================ 2024-03-25T18:23:27.892 INFO:teuthology.orchestra.run.smithi001.stdout: Package Arch Version Repo Size 2024-03-25T18:23:27.892 INFO:teuthology.orchestra.run.smithi001.stdout:================================================================================ 2024-03-25T18:23:27.893 INFO:teuthology.orchestra.run.smithi001.stdout:Installing: 2024-03-25T18:23:27.893 INFO:teuthology.orchestra.run.smithi001.stdout: python3-cephfs-debuginfo x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 615 k 2024-03-25T18:23:27.893 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-25T18:23:27.893 INFO:teuthology.orchestra.run.smithi001.stdout:Transaction Summary 2024-03-25T18:23:27.893 INFO:teuthology.orchestra.run.smithi001.stdout:================================================================================ 2024-03-25T18:23:27.893 INFO:teuthology.orchestra.run.smithi001.stdout:Install 1 Package 2024-03-25T18:23:27.893 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-25T18:23:27.893 INFO:teuthology.orchestra.run.smithi001.stdout:Total download size: 615 k 2024-03-25T18:23:27.894 INFO:teuthology.orchestra.run.smithi001.stdout:Installed size: 2.1 M 2024-03-25T18:23:27.894 INFO:teuthology.orchestra.run.smithi001.stdout:Downloading Packages: 2024-03-25T18:23:29.622 INFO:teuthology.orchestra.run.smithi001.stdout:python3-cephfs-debuginfo-19.0.0-1439.g5dcb190d. 356 kB/s | 615 kB 00:01 2024-03-25T18:23:29.623 INFO:teuthology.orchestra.run.smithi001.stdout:-------------------------------------------------------------------------------- 2024-03-25T18:23:29.623 INFO:teuthology.orchestra.run.smithi001.stdout:Total 355 kB/s | 615 kB 00:01 2024-03-25T18:23:29.623 INFO:teuthology.orchestra.run.smithi001.stdout:Running transaction check 2024-03-25T18:23:29.627 INFO:teuthology.orchestra.run.smithi001.stdout:Transaction check succeeded. 2024-03-25T18:23:29.627 INFO:teuthology.orchestra.run.smithi001.stdout:Running transaction test 2024-03-25T18:23:29.671 INFO:teuthology.orchestra.run.smithi001.stdout:Transaction test succeeded. 2024-03-25T18:23:29.672 INFO:teuthology.orchestra.run.smithi001.stdout:Running transaction 2024-03-25T18:23:29.866 INFO:teuthology.orchestra.run.smithi001.stdout: Preparing : 1/1 2024-03-25T18:23:29.957 INFO:teuthology.orchestra.run.smithi001.stdout: Installing : python3-cephfs-debuginfo-2:19.0.0-1439.g5dcb190d.el9 1/1 2024-03-25T18:23:30.330 INFO:teuthology.orchestra.run.smithi001.stdout: Running scriptlet: python3-cephfs-debuginfo-2:19.0.0-1439.g5dcb190d.el9 1/1 2024-03-25T18:23:30.591 INFO:teuthology.orchestra.run.smithi001.stdout: Verifying : python3-cephfs-debuginfo-2:19.0.0-1439.g5dcb190d.el9 1/1 2024-03-25T18:23:30.591 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-25T18:23:30.591 INFO:teuthology.orchestra.run.smithi001.stdout:Installed: 2024-03-25T18:23:30.591 INFO:teuthology.orchestra.run.smithi001.stdout: python3-cephfs-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T18:23:30.591 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-25T18:23:30.591 INFO:teuthology.orchestra.run.smithi001.stdout:Complete! 2024-03-25T18:23:30.851 DEBUG:teuthology.orchestra.run.smithi001:> sudo yum -y install python3-rados-debuginfo 2024-03-25T18:23:31.442 INFO:teuthology.orchestra.run.smithi001.stdout:Last metadata expiration check: 1:50:42 ago on Mon 25 Mar 2024 04:32:49 PM UTC. 2024-03-25T18:23:31.600 INFO:teuthology.orchestra.run.smithi001.stdout:Dependencies resolved. 2024-03-25T18:23:31.600 INFO:teuthology.orchestra.run.smithi001.stdout:================================================================================ 2024-03-25T18:23:31.600 INFO:teuthology.orchestra.run.smithi001.stdout: Package Arch Version Repo Size 2024-03-25T18:23:31.600 INFO:teuthology.orchestra.run.smithi001.stdout:================================================================================ 2024-03-25T18:23:31.600 INFO:teuthology.orchestra.run.smithi001.stdout:Installing: 2024-03-25T18:23:31.600 INFO:teuthology.orchestra.run.smithi001.stdout: python3-rados-debuginfo x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 771 k 2024-03-25T18:23:31.601 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-25T18:23:31.601 INFO:teuthology.orchestra.run.smithi001.stdout:Transaction Summary 2024-03-25T18:23:31.601 INFO:teuthology.orchestra.run.smithi001.stdout:================================================================================ 2024-03-25T18:23:31.601 INFO:teuthology.orchestra.run.smithi001.stdout:Install 1 Package 2024-03-25T18:23:31.601 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-25T18:23:31.601 INFO:teuthology.orchestra.run.smithi001.stdout:Total download size: 771 k 2024-03-25T18:23:31.602 INFO:teuthology.orchestra.run.smithi001.stdout:Installed size: 2.7 M 2024-03-25T18:23:31.602 INFO:teuthology.orchestra.run.smithi001.stdout:Downloading Packages: 2024-03-25T18:23:33.983 INFO:teuthology.orchestra.run.smithi001.stdout:python3-rados-debuginfo-19.0.0-1439.g5dcb190d.e 324 kB/s | 771 kB 00:02 2024-03-25T18:23:33.984 INFO:teuthology.orchestra.run.smithi001.stdout:-------------------------------------------------------------------------------- 2024-03-25T18:23:33.984 INFO:teuthology.orchestra.run.smithi001.stdout:Total 323 kB/s | 771 kB 00:02 2024-03-25T18:23:33.985 INFO:teuthology.orchestra.run.smithi001.stdout:Running transaction check 2024-03-25T18:23:33.989 INFO:teuthology.orchestra.run.smithi001.stdout:Transaction check succeeded. 2024-03-25T18:23:33.989 INFO:teuthology.orchestra.run.smithi001.stdout:Running transaction test 2024-03-25T18:23:34.033 INFO:teuthology.orchestra.run.smithi001.stdout:Transaction test succeeded. 2024-03-25T18:23:34.033 INFO:teuthology.orchestra.run.smithi001.stdout:Running transaction 2024-03-25T18:23:34.224 INFO:teuthology.orchestra.run.smithi001.stdout: Preparing : 1/1 2024-03-25T18:23:34.327 INFO:teuthology.orchestra.run.smithi001.stdout: Installing : python3-rados-debuginfo-2:19.0.0-1439.g5dcb190d.el9. 1/1 2024-03-25T18:23:34.692 INFO:teuthology.orchestra.run.smithi001.stdout: Running scriptlet: python3-rados-debuginfo-2:19.0.0-1439.g5dcb190d.el9. 1/1 2024-03-25T18:23:34.944 INFO:teuthology.orchestra.run.smithi001.stdout: Verifying : python3-rados-debuginfo-2:19.0.0-1439.g5dcb190d.el9. 1/1 2024-03-25T18:23:34.944 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-25T18:23:34.944 INFO:teuthology.orchestra.run.smithi001.stdout:Installed: 2024-03-25T18:23:34.944 INFO:teuthology.orchestra.run.smithi001.stdout: python3-rados-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T18:23:34.944 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-25T18:23:34.944 INFO:teuthology.orchestra.run.smithi001.stdout:Complete! 2024-03-25T18:23:35.173 DEBUG:teuthology.orchestra.run.smithi001:> sudo yum -y install python3-rbd-debuginfo 2024-03-25T18:23:35.747 INFO:teuthology.orchestra.run.smithi001.stdout:Last metadata expiration check: 1:50:46 ago on Mon 25 Mar 2024 04:32:49 PM UTC. 2024-03-25T18:23:35.902 INFO:teuthology.orchestra.run.smithi001.stdout:Dependencies resolved. 2024-03-25T18:23:35.903 INFO:teuthology.orchestra.run.smithi001.stdout:================================================================================ 2024-03-25T18:23:35.903 INFO:teuthology.orchestra.run.smithi001.stdout: Package Arch Version Repo Size 2024-03-25T18:23:35.903 INFO:teuthology.orchestra.run.smithi001.stdout:================================================================================ 2024-03-25T18:23:35.903 INFO:teuthology.orchestra.run.smithi001.stdout:Installing: 2024-03-25T18:23:35.903 INFO:teuthology.orchestra.run.smithi001.stdout: python3-rbd-debuginfo x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 1.1 M 2024-03-25T18:23:35.903 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-25T18:23:35.903 INFO:teuthology.orchestra.run.smithi001.stdout:Transaction Summary 2024-03-25T18:23:35.904 INFO:teuthology.orchestra.run.smithi001.stdout:================================================================================ 2024-03-25T18:23:35.904 INFO:teuthology.orchestra.run.smithi001.stdout:Install 1 Package 2024-03-25T18:23:35.904 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-25T18:23:35.904 INFO:teuthology.orchestra.run.smithi001.stdout:Total download size: 1.1 M 2024-03-25T18:23:35.904 INFO:teuthology.orchestra.run.smithi001.stdout:Installed size: 4.3 M 2024-03-25T18:23:35.904 INFO:teuthology.orchestra.run.smithi001.stdout:Downloading Packages: 2024-03-25T18:23:38.628 INFO:teuthology.orchestra.run.smithi001.stdout:python3-rbd-debuginfo-19.0.0-1439.g5dcb190d.el9 422 kB/s | 1.1 MB 00:02 2024-03-25T18:23:38.629 INFO:teuthology.orchestra.run.smithi001.stdout:-------------------------------------------------------------------------------- 2024-03-25T18:23:38.629 INFO:teuthology.orchestra.run.smithi001.stdout:Total 422 kB/s | 1.1 MB 00:02 2024-03-25T18:23:38.629 INFO:teuthology.orchestra.run.smithi001.stdout:Running transaction check 2024-03-25T18:23:38.633 INFO:teuthology.orchestra.run.smithi001.stdout:Transaction check succeeded. 2024-03-25T18:23:38.634 INFO:teuthology.orchestra.run.smithi001.stdout:Running transaction test 2024-03-25T18:23:38.679 INFO:teuthology.orchestra.run.smithi001.stdout:Transaction test succeeded. 2024-03-25T18:23:38.679 INFO:teuthology.orchestra.run.smithi001.stdout:Running transaction 2024-03-25T18:23:38.913 INFO:teuthology.orchestra.run.smithi001.stdout: Preparing : 1/1 2024-03-25T18:23:39.013 INFO:teuthology.orchestra.run.smithi001.stdout: Installing : python3-rbd-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x8 1/1 2024-03-25T18:23:39.396 INFO:teuthology.orchestra.run.smithi001.stdout: Running scriptlet: python3-rbd-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x8 1/1 2024-03-25T18:23:39.674 INFO:teuthology.orchestra.run.smithi001.stdout: Verifying : python3-rbd-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x8 1/1 2024-03-25T18:23:39.675 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-25T18:23:39.675 INFO:teuthology.orchestra.run.smithi001.stdout:Installed: 2024-03-25T18:23:39.675 INFO:teuthology.orchestra.run.smithi001.stdout: python3-rbd-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T18:23:39.675 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-25T18:23:39.675 INFO:teuthology.orchestra.run.smithi001.stdout:Complete! 2024-03-25T18:23:39.904 DEBUG:teuthology.orchestra.run.smithi001:> sudo yum -y install python3-rgw-debuginfo 2024-03-25T18:23:40.478 INFO:teuthology.orchestra.run.smithi001.stdout:Last metadata expiration check: 1:50:51 ago on Mon 25 Mar 2024 04:32:49 PM UTC. 2024-03-25T18:23:40.634 INFO:teuthology.orchestra.run.smithi001.stdout:Dependencies resolved. 2024-03-25T18:23:40.634 INFO:teuthology.orchestra.run.smithi001.stdout:================================================================================ 2024-03-25T18:23:40.635 INFO:teuthology.orchestra.run.smithi001.stdout: Package Arch Version Repo Size 2024-03-25T18:23:40.635 INFO:teuthology.orchestra.run.smithi001.stdout:================================================================================ 2024-03-25T18:23:40.635 INFO:teuthology.orchestra.run.smithi001.stdout:Installing: 2024-03-25T18:23:40.635 INFO:teuthology.orchestra.run.smithi001.stdout: python3-rgw-debuginfo x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 337 k 2024-03-25T18:23:40.635 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-25T18:23:40.635 INFO:teuthology.orchestra.run.smithi001.stdout:Transaction Summary 2024-03-25T18:23:40.635 INFO:teuthology.orchestra.run.smithi001.stdout:================================================================================ 2024-03-25T18:23:40.635 INFO:teuthology.orchestra.run.smithi001.stdout:Install 1 Package 2024-03-25T18:23:40.635 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-25T18:23:40.635 INFO:teuthology.orchestra.run.smithi001.stdout:Total download size: 337 k 2024-03-25T18:23:40.635 INFO:teuthology.orchestra.run.smithi001.stdout:Installed size: 1.0 M 2024-03-25T18:23:40.635 INFO:teuthology.orchestra.run.smithi001.stdout:Downloading Packages: 2024-03-25T18:23:41.732 INFO:teuthology.orchestra.run.smithi001.stdout:python3-rgw-debuginfo-19.0.0-1439.g5dcb190d.el9 308 kB/s | 337 kB 00:01 2024-03-25T18:23:41.732 INFO:teuthology.orchestra.run.smithi001.stdout:-------------------------------------------------------------------------------- 2024-03-25T18:23:41.732 INFO:teuthology.orchestra.run.smithi001.stdout:Total 307 kB/s | 337 kB 00:01 2024-03-25T18:23:41.733 INFO:teuthology.orchestra.run.smithi001.stdout:Running transaction check 2024-03-25T18:23:41.737 INFO:teuthology.orchestra.run.smithi001.stdout:Transaction check succeeded. 2024-03-25T18:23:41.737 INFO:teuthology.orchestra.run.smithi001.stdout:Running transaction test 2024-03-25T18:23:41.778 INFO:teuthology.orchestra.run.smithi001.stdout:Transaction test succeeded. 2024-03-25T18:23:41.778 INFO:teuthology.orchestra.run.smithi001.stdout:Running transaction 2024-03-25T18:23:41.919 INFO:teuthology.orchestra.run.smithi001.stdout: Preparing : 1/1 2024-03-25T18:23:42.167 INFO:teuthology.orchestra.run.smithi001.stdout: Installing : python3-rgw-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x8 1/1 2024-03-25T18:23:42.543 INFO:teuthology.orchestra.run.smithi001.stdout: Running scriptlet: python3-rgw-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x8 1/1 2024-03-25T18:23:42.805 INFO:teuthology.orchestra.run.smithi001.stdout: Verifying : python3-rgw-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x8 1/1 2024-03-25T18:23:42.805 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-25T18:23:42.805 INFO:teuthology.orchestra.run.smithi001.stdout:Installed: 2024-03-25T18:23:42.805 INFO:teuthology.orchestra.run.smithi001.stdout: python3-rgw-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T18:23:42.805 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-25T18:23:42.805 INFO:teuthology.orchestra.run.smithi001.stdout:Complete! 2024-03-25T18:23:43.024 DEBUG:teuthology.orchestra.run.smithi001:> sudo yum -y install rbd-fuse-debuginfo 2024-03-25T18:23:43.598 INFO:teuthology.orchestra.run.smithi001.stdout:Last metadata expiration check: 1:50:54 ago on Mon 25 Mar 2024 04:32:49 PM UTC. 2024-03-25T18:23:43.756 INFO:teuthology.orchestra.run.smithi001.stdout:Dependencies resolved. 2024-03-25T18:23:43.757 INFO:teuthology.orchestra.run.smithi001.stdout:================================================================================ 2024-03-25T18:23:43.757 INFO:teuthology.orchestra.run.smithi001.stdout: Package Arch Version Repo Size 2024-03-25T18:23:43.757 INFO:teuthology.orchestra.run.smithi001.stdout:================================================================================ 2024-03-25T18:23:43.757 INFO:teuthology.orchestra.run.smithi001.stdout:Installing: 2024-03-25T18:23:43.757 INFO:teuthology.orchestra.run.smithi001.stdout: rbd-fuse-debuginfo x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 509 k 2024-03-25T18:23:43.757 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-25T18:23:43.757 INFO:teuthology.orchestra.run.smithi001.stdout:Transaction Summary 2024-03-25T18:23:43.757 INFO:teuthology.orchestra.run.smithi001.stdout:================================================================================ 2024-03-25T18:23:43.757 INFO:teuthology.orchestra.run.smithi001.stdout:Install 1 Package 2024-03-25T18:23:43.758 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-25T18:23:43.758 INFO:teuthology.orchestra.run.smithi001.stdout:Total download size: 509 k 2024-03-25T18:23:43.758 INFO:teuthology.orchestra.run.smithi001.stdout:Installed size: 2.5 M 2024-03-25T18:23:43.758 INFO:teuthology.orchestra.run.smithi001.stdout:Downloading Packages: 2024-03-25T18:23:45.184 INFO:teuthology.orchestra.run.smithi001.stdout:rbd-fuse-debuginfo-19.0.0-1439.g5dcb190d.el9.x8 357 kB/s | 509 kB 00:01 2024-03-25T18:23:45.185 INFO:teuthology.orchestra.run.smithi001.stdout:-------------------------------------------------------------------------------- 2024-03-25T18:23:45.185 INFO:teuthology.orchestra.run.smithi001.stdout:Total 357 kB/s | 509 kB 00:01 2024-03-25T18:23:45.185 INFO:teuthology.orchestra.run.smithi001.stdout:Running transaction check 2024-03-25T18:23:45.190 INFO:teuthology.orchestra.run.smithi001.stdout:Transaction check succeeded. 2024-03-25T18:23:45.190 INFO:teuthology.orchestra.run.smithi001.stdout:Running transaction test 2024-03-25T18:23:45.231 INFO:teuthology.orchestra.run.smithi001.stdout:Transaction test succeeded. 2024-03-25T18:23:45.231 INFO:teuthology.orchestra.run.smithi001.stdout:Running transaction 2024-03-25T18:23:45.401 INFO:teuthology.orchestra.run.smithi001.stdout: Preparing : 1/1 2024-03-25T18:23:45.512 INFO:teuthology.orchestra.run.smithi001.stdout: Installing : rbd-fuse-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_6 1/1 2024-03-25T18:23:45.898 INFO:teuthology.orchestra.run.smithi001.stdout: Running scriptlet: rbd-fuse-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_6 1/1 2024-03-25T18:23:46.157 INFO:teuthology.orchestra.run.smithi001.stdout: Verifying : rbd-fuse-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_6 1/1 2024-03-25T18:23:46.157 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-25T18:23:46.157 INFO:teuthology.orchestra.run.smithi001.stdout:Installed: 2024-03-25T18:23:46.157 INFO:teuthology.orchestra.run.smithi001.stdout: rbd-fuse-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T18:23:46.157 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-25T18:23:46.157 INFO:teuthology.orchestra.run.smithi001.stdout:Complete! 2024-03-25T18:23:46.384 DEBUG:teuthology.orchestra.run.smithi001:> sudo yum -y install rbd-mirror-debuginfo 2024-03-25T18:23:46.955 INFO:teuthology.orchestra.run.smithi001.stdout:Last metadata expiration check: 1:50:57 ago on Mon 25 Mar 2024 04:32:49 PM UTC. 2024-03-25T18:23:47.111 INFO:teuthology.orchestra.run.smithi001.stdout:Dependencies resolved. 2024-03-25T18:23:47.112 INFO:teuthology.orchestra.run.smithi001.stdout:================================================================================ 2024-03-25T18:23:47.112 INFO:teuthology.orchestra.run.smithi001.stdout: Package Arch Version Repo Size 2024-03-25T18:23:47.112 INFO:teuthology.orchestra.run.smithi001.stdout:================================================================================ 2024-03-25T18:23:47.112 INFO:teuthology.orchestra.run.smithi001.stdout:Installing: 2024-03-25T18:23:47.112 INFO:teuthology.orchestra.run.smithi001.stdout: rbd-mirror-debuginfo x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 52 M 2024-03-25T18:23:47.112 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-25T18:23:47.112 INFO:teuthology.orchestra.run.smithi001.stdout:Transaction Summary 2024-03-25T18:23:47.112 INFO:teuthology.orchestra.run.smithi001.stdout:================================================================================ 2024-03-25T18:23:47.112 INFO:teuthology.orchestra.run.smithi001.stdout:Install 1 Package 2024-03-25T18:23:47.112 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-25T18:23:47.112 INFO:teuthology.orchestra.run.smithi001.stdout:Total download size: 52 M 2024-03-25T18:23:47.112 INFO:teuthology.orchestra.run.smithi001.stdout:Installed size: 298 M 2024-03-25T18:23:47.112 INFO:teuthology.orchestra.run.smithi001.stdout:Downloading Packages: 2024-03-25T18:25:34.069 INFO:teuthology.orchestra.run.smithi001.stdout:rbd-mirror-debuginfo-19.0.0-1439.g5dcb190d.el9. 493 kB/s | 52 MB 01:46 2024-03-25T18:25:34.069 INFO:teuthology.orchestra.run.smithi001.stdout:-------------------------------------------------------------------------------- 2024-03-25T18:25:34.070 INFO:teuthology.orchestra.run.smithi001.stdout:Total 493 kB/s | 52 MB 01:46 2024-03-25T18:25:34.070 INFO:teuthology.orchestra.run.smithi001.stdout:Running transaction check 2024-03-25T18:25:34.073 INFO:teuthology.orchestra.run.smithi001.stdout:Transaction check succeeded. 2024-03-25T18:25:34.073 INFO:teuthology.orchestra.run.smithi001.stdout:Running transaction test 2024-03-25T18:25:34.423 INFO:teuthology.orchestra.run.smithi001.stdout:Transaction test succeeded. 2024-03-25T18:25:34.423 INFO:teuthology.orchestra.run.smithi001.stdout:Running transaction 2024-03-25T18:25:40.693 INFO:teuthology.orchestra.run.smithi001.stdout: Preparing : 1/1 2024-03-25T18:25:40.803 INFO:teuthology.orchestra.run.smithi001.stdout: Installing : rbd-mirror-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86 1/1 2024-03-25T18:25:42.832 INFO:teuthology.orchestra.run.smithi001.stdout: Running scriptlet: rbd-mirror-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86 1/1 2024-03-25T18:25:43.100 INFO:teuthology.orchestra.run.smithi001.stdout: Verifying : rbd-mirror-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86 1/1 2024-03-25T18:25:43.100 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-25T18:25:43.100 INFO:teuthology.orchestra.run.smithi001.stdout:Installed: 2024-03-25T18:25:43.100 INFO:teuthology.orchestra.run.smithi001.stdout: rbd-mirror-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T18:25:43.100 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-25T18:25:43.101 INFO:teuthology.orchestra.run.smithi001.stdout:Complete! 2024-03-25T18:25:43.331 DEBUG:teuthology.orchestra.run.smithi001:> sudo yum -y install rbd-nbd-debuginfo 2024-03-25T18:25:43.906 INFO:teuthology.orchestra.run.smithi001.stdout:Last metadata expiration check: 1:52:54 ago on Mon 25 Mar 2024 04:32:49 PM UTC. 2024-03-25T18:25:44.060 INFO:teuthology.orchestra.run.smithi001.stdout:Dependencies resolved. 2024-03-25T18:25:44.061 INFO:teuthology.orchestra.run.smithi001.stdout:================================================================================ 2024-03-25T18:25:44.061 INFO:teuthology.orchestra.run.smithi001.stdout: Package Arch Version Repo Size 2024-03-25T18:25:44.061 INFO:teuthology.orchestra.run.smithi001.stdout:================================================================================ 2024-03-25T18:25:44.062 INFO:teuthology.orchestra.run.smithi001.stdout:Installing: 2024-03-25T18:25:44.062 INFO:teuthology.orchestra.run.smithi001.stdout: rbd-nbd-debuginfo x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 1.3 M 2024-03-25T18:25:44.062 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-25T18:25:44.062 INFO:teuthology.orchestra.run.smithi001.stdout:Transaction Summary 2024-03-25T18:25:44.062 INFO:teuthology.orchestra.run.smithi001.stdout:================================================================================ 2024-03-25T18:25:44.062 INFO:teuthology.orchestra.run.smithi001.stdout:Install 1 Package 2024-03-25T18:25:44.062 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-25T18:25:44.062 INFO:teuthology.orchestra.run.smithi001.stdout:Total download size: 1.3 M 2024-03-25T18:25:44.062 INFO:teuthology.orchestra.run.smithi001.stdout:Installed size: 5.5 M 2024-03-25T18:25:44.062 INFO:teuthology.orchestra.run.smithi001.stdout:Downloading Packages: 2024-03-25T18:25:45.954 INFO:teuthology.orchestra.run.smithi001.stdout:rbd-nbd-debuginfo-19.0.0-1439.g5dcb190d.el9.x86 678 kB/s | 1.3 MB 00:01 2024-03-25T18:25:45.955 INFO:teuthology.orchestra.run.smithi001.stdout:-------------------------------------------------------------------------------- 2024-03-25T18:25:45.955 INFO:teuthology.orchestra.run.smithi001.stdout:Total 677 kB/s | 1.3 MB 00:01 2024-03-25T18:25:45.955 INFO:teuthology.orchestra.run.smithi001.stdout:Running transaction check 2024-03-25T18:25:45.959 INFO:teuthology.orchestra.run.smithi001.stdout:Transaction check succeeded. 2024-03-25T18:25:45.959 INFO:teuthology.orchestra.run.smithi001.stdout:Running transaction test 2024-03-25T18:25:46.007 INFO:teuthology.orchestra.run.smithi001.stdout:Transaction test succeeded. 2024-03-25T18:25:46.007 INFO:teuthology.orchestra.run.smithi001.stdout:Running transaction 2024-03-25T18:25:46.258 INFO:teuthology.orchestra.run.smithi001.stdout: Preparing : 1/1 2024-03-25T18:25:46.364 INFO:teuthology.orchestra.run.smithi001.stdout: Installing : rbd-nbd-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/1 2024-03-25T18:25:46.780 INFO:teuthology.orchestra.run.smithi001.stdout: Running scriptlet: rbd-nbd-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/1 2024-03-25T18:25:47.043 INFO:teuthology.orchestra.run.smithi001.stdout: Verifying : rbd-nbd-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/1 2024-03-25T18:25:47.044 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-25T18:25:47.044 INFO:teuthology.orchestra.run.smithi001.stdout:Installed: 2024-03-25T18:25:47.044 INFO:teuthology.orchestra.run.smithi001.stdout: rbd-nbd-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T18:25:47.044 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-25T18:25:47.044 INFO:teuthology.orchestra.run.smithi001.stdout:Complete! 2024-03-25T18:25:47.271 DEBUG:teuthology.parallel:result is None 2024-03-25T18:25:47.272 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F9%2Fx86_64&sha1=5dcb190da270440ef141d088eb27a9fdb8afc5f2 2024-03-25T18:25:47.492 DEBUG:teuthology.orchestra.run.smithi001:> rpm -q ceph --qf '%{VERSION}-%{RELEASE}' 2024-03-25T18:25:47.512 INFO:teuthology.orchestra.run.smithi001.stdout:19.0.0-1439.g5dcb190d.el9 2024-03-25T18:25:47.512 INFO:teuthology.packaging:The installed version of ceph is 19.0.0-1439.g5dcb190d.el9 2024-03-25T18:25:47.513 INFO:teuthology.task.install:The correct ceph version 19.0.0-1439.g5dcb190d is installed. 2024-03-25T18:25:47.514 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F9%2Fx86_64&sha1=5dcb190da270440ef141d088eb27a9fdb8afc5f2 2024-03-25T18:25:47.653 DEBUG:teuthology.orchestra.run.smithi115:> rpm -q ceph --qf '%{VERSION}-%{RELEASE}' 2024-03-25T18:25:47.678 INFO:teuthology.orchestra.run.smithi115.stdout:19.0.0-1439.g5dcb190d.el9 2024-03-25T18:25:47.678 INFO:teuthology.packaging:The installed version of ceph is 19.0.0-1439.g5dcb190d.el9 2024-03-25T18:25:47.678 INFO:teuthology.task.install:The correct ceph version 19.0.0-1439.g5dcb190d is installed. 2024-03-25T18:25:47.680 INFO:teuthology.task.install.util:Shipping valgrind.supp... 2024-03-25T18:25:47.680 DEBUG:teuthology.orchestra.run.smithi001:> set -ex 2024-03-25T18:25:47.680 DEBUG:teuthology.orchestra.run.smithi001:> sudo dd of=/home/ubuntu/cephtest/valgrind.supp 2024-03-25T18:25:47.707 DEBUG:teuthology.orchestra.run.smithi115:> set -ex 2024-03-25T18:25:47.707 DEBUG:teuthology.orchestra.run.smithi115:> sudo dd of=/home/ubuntu/cephtest/valgrind.supp 2024-03-25T18:25:47.750 INFO:teuthology.task.install.util:Shipping 'daemon-helper'... 2024-03-25T18:25:47.750 DEBUG:teuthology.orchestra.run.smithi001:> set -ex 2024-03-25T18:25:47.750 DEBUG:teuthology.orchestra.run.smithi001:> sudo dd of=/usr/bin/daemon-helper 2024-03-25T18:25:47.777 DEBUG:teuthology.orchestra.run.smithi001:> sudo chmod a=rx -- /usr/bin/daemon-helper 2024-03-25T18:25:47.845 DEBUG:teuthology.orchestra.run.smithi115:> set -ex 2024-03-25T18:25:47.845 DEBUG:teuthology.orchestra.run.smithi115:> sudo dd of=/usr/bin/daemon-helper 2024-03-25T18:25:47.875 DEBUG:teuthology.orchestra.run.smithi115:> sudo chmod a=rx -- /usr/bin/daemon-helper 2024-03-25T18:25:47.945 INFO:teuthology.task.install.util:Shipping 'adjust-ulimits'... 2024-03-25T18:25:47.945 DEBUG:teuthology.orchestra.run.smithi001:> set -ex 2024-03-25T18:25:47.945 DEBUG:teuthology.orchestra.run.smithi001:> sudo dd of=/usr/bin/adjust-ulimits 2024-03-25T18:25:47.973 DEBUG:teuthology.orchestra.run.smithi001:> sudo chmod a=rx -- /usr/bin/adjust-ulimits 2024-03-25T18:25:48.040 DEBUG:teuthology.orchestra.run.smithi115:> set -ex 2024-03-25T18:25:48.040 DEBUG:teuthology.orchestra.run.smithi115:> sudo dd of=/usr/bin/adjust-ulimits 2024-03-25T18:25:48.069 DEBUG:teuthology.orchestra.run.smithi115:> sudo chmod a=rx -- /usr/bin/adjust-ulimits 2024-03-25T18:25:48.138 INFO:teuthology.task.install.util:Shipping 'stdin-killer'... 2024-03-25T18:25:48.138 DEBUG:teuthology.orchestra.run.smithi001:> set -ex 2024-03-25T18:25:48.138 DEBUG:teuthology.orchestra.run.smithi001:> sudo dd of=/usr/bin/stdin-killer 2024-03-25T18:25:48.165 DEBUG:teuthology.orchestra.run.smithi001:> sudo chmod a=rx -- /usr/bin/stdin-killer 2024-03-25T18:25:48.232 DEBUG:teuthology.orchestra.run.smithi115:> set -ex 2024-03-25T18:25:48.233 DEBUG:teuthology.orchestra.run.smithi115:> sudo dd of=/usr/bin/stdin-killer 2024-03-25T18:25:48.259 DEBUG:teuthology.orchestra.run.smithi115:> sudo chmod a=rx -- /usr/bin/stdin-killer 2024-03-25T18:25:48.327 INFO:teuthology.run_tasks:Running task ceph... 2024-03-25T18:25:48.415 INFO:tasks.ceph:Making ceph log dir writeable by non-root... 2024-03-25T18:25:48.415 DEBUG:teuthology.orchestra.run.smithi001:> sudo chmod 777 /var/log/ceph 2024-03-25T18:25:48.417 DEBUG:teuthology.orchestra.run.smithi115:> sudo chmod 777 /var/log/ceph 2024-03-25T18:25:48.444 INFO:tasks.ceph:Disabling ceph logrotate... 2024-03-25T18:25:48.445 DEBUG:teuthology.orchestra.run.smithi001:> sudo rm -f -- /etc/logrotate.d/ceph 2024-03-25T18:25:48.485 DEBUG:teuthology.orchestra.run.smithi115:> sudo rm -f -- /etc/logrotate.d/ceph 2024-03-25T18:25:48.512 INFO:tasks.ceph:Creating extra log directories... 2024-03-25T18:25:48.512 DEBUG:teuthology.orchestra.run.smithi001:> sudo install -d -m0777 -- /var/log/ceph/valgrind /var/log/ceph/profiling-logger 2024-03-25T18:25:48.553 DEBUG:teuthology.orchestra.run.smithi115:> sudo install -d -m0777 -- /var/log/ceph/valgrind /var/log/ceph/profiling-logger 2024-03-25T18:25:48.622 INFO:tasks.ceph:Creating ceph cluster ceph... 2024-03-25T18:25:48.622 INFO:tasks.ceph:config {'conf': {'client': {'debug monc': 20, 'debug ms': 1, 'debug objecter': 20, 'debug rados': 20}, 'global': {'mon client directed command retry': 5, 'mon election default strategy': 3, 'ms bind msgr2': False, 'ms inject socket failures': 5000, 'ms type': 'async', 'osd heartbeat grace': 80}, 'mgr': {'debug mgr': 20, 'debug ms': 1}, 'mon': {'debug mon': 20, 'debug ms': 1, 'debug paxos': 20, 'mon min osdmap epochs': 50, 'mon osd crush smoke test': False, 'mon osdmap full prune interval': 2, 'mon osdmap full prune min': 15, 'mon osdmap full prune txsize': 2, 'mon scrub interval': 300, 'mon warn on pool no app': False, 'paxos service trim min': 10}, 'osd': {'bdev async discard': True, 'bdev enable discard': True, 'bluestore allocator': 'stupid', 'bluestore block size': 96636764160, 'bluestore fsck on mount': True, 'bluestore zero block detection': True, 'debug bluefs': 1, 'debug bluestore': 1, 'debug monc': 20, 'debug ms': 1, 'debug osd': 20, 'debug rocksdb': 10, 'mon osd backfillfull_ratio': 0.85, 'mon osd full ratio': 0.9, 'mon osd nearfull ratio': 0.8, 'osd blocked scrub grace period': 3600, 'osd class default list': '*', 'osd class load list': '*', 'osd client watch timeout': 120, 'osd debug verify cached snaps': True, 'osd debug verify missing on start': True, 'osd failsafe full ratio': 0.95, 'osd fast shutdown': False, 'osd max markdown count': 1000, 'osd mclock override recovery settings': True, 'osd mclock profile': 'high_recovery_ops', 'osd objectstore': 'bluestore', 'osd op queue': 'debug_random', 'osd op queue cut off': 'debug_random', 'osd shutdown pgref assert': True}}, 'fs': 'xfs', 'mkfs_options': None, 'mount_options': None, 'skip_mgr_daemons': False, 'log_ignorelist': ['\\(MDS_ALL_DOWN\\)', '\\(MDS_UP_LESS_THAN_MAX\\)', 'but it is still running', 'objects unfound and apparently lost', 'overall HEALTH_', '\\(OSDMAP_FLAGS\\)', '\\(OSD_', '\\(PG_', '\\(POOL_', '\\(CACHE_POOL_', '\\(SMALLER_PGP_NUM\\)', '\\(OBJECT_', '\\(SLOW_OPS\\)', '\\(REQUEST_SLOW\\)', '\\(TOO_FEW_PGS\\)', 'slow request', 'timeout on replica', 'late reservation from', 'MON_DOWN', 'OSDMAP_FLAGS', 'OSD_DOWN', 'PG_DEGRADED', 'PG_AVAILABILITY', 'POOL_APP_NOT_ENABLED', '\\(OSD_SLOW_PING_TIME', 'reached quota', 'overall HEALTH_', '\\(CACHE_POOL_NO_HIT_SET\\)', '\\(POOL_FULL\\)', '\\(SMALLER_PGP_NUM\\)', '\\(SLOW_OPS\\)', '\\(CACHE_POOL_NEAR_FULL\\)', '\\(POOL_APP_NOT_ENABLED\\)', '\\(PG_AVAILABILITY\\)', '\\(OBJECT_MISPLACED\\)', 'slow request', 'overall HEALTH_', '\\(PG_', '\\(MON_DOWN\\)', '\\(SLOW_OPS\\)', 'slow request'], 'cpu_profile': set(), 'cluster': 'ceph', 'mon_bind_msgr2': False, 'mon_bind_addrvec': True} 2024-03-25T18:25:48.622 INFO:tasks.ceph:ctx.config {'archive_path': '/home/teuthworker/archive/amathuri-2024-03-25_13:28:33-rados:verify-main-distro-default-smithi/7621279', 'branch': 'main', 'description': 'rados:verify/{centos_latest ceph clusters/{fixed-2 openstack} d-thrash/default/{default thrashosds-health} mon_election/connectivity msgr-failures/few msgr/async-v1only objectstore/bluestore-stupid rados tasks/rados_api_tests validater/valgrind}', 'email': None, 'first_in_suite': False, 'job_id': '7621279', 'kernel': {'kdb': True, 'sha1': 'distro'}, 'last_in_suite': False, 'machine_type': 'smithi', 'name': 'amathuri-2024-03-25_13:28:33-rados:verify-main-distro-default-smithi', 'no_nested_subset': False, 'nuke-on-error': True, 'openstack': [{'volumes': {'count': 4, 'size': 10}}, {'volumes': {'count': 4, 'size': 10}}], 'os_type': 'centos', 'os_version': '9.stream', 'overrides': {'admin_socket': {'branch': 'main'}, 'ceph': {'conf': {'client': {'debug monc': 20, 'debug ms': 1, 'debug objecter': 20, 'debug rados': 20}, 'global': {'mon client directed command retry': 5, 'mon election default strategy': 3, 'ms bind msgr2': False, 'ms inject socket failures': 5000, 'ms type': 'async', 'osd heartbeat grace': 80}, 'mgr': {'debug mgr': 20, 'debug ms': 1}, 'mon': {'debug mon': 20, 'debug ms': 1, 'debug paxos': 20, 'mon min osdmap epochs': 50, 'mon osd crush smoke test': False, 'mon osdmap full prune interval': 2, 'mon osdmap full prune min': 15, 'mon osdmap full prune txsize': 2, 'mon scrub interval': 300, 'mon warn on pool no app': False, 'paxos service trim min': 10}, 'osd': {'bdev async discard': True, 'bdev enable discard': True, 'bluestore allocator': 'stupid', 'bluestore block size': 96636764160, 'bluestore fsck on mount': True, 'bluestore zero block detection': True, 'debug bluefs': 1, 'debug bluestore': 1, 'debug monc': 20, 'debug ms': 1, 'debug osd': 20, 'debug rocksdb': 10, 'mon osd backfillfull_ratio': 0.85, 'mon osd full ratio': 0.9, 'mon osd nearfull ratio': 0.8, 'osd blocked scrub grace period': 3600, 'osd class default list': '*', 'osd class load list': '*', 'osd client watch timeout': 120, 'osd debug verify cached snaps': True, 'osd debug verify missing on start': True, 'osd failsafe full ratio': 0.95, 'osd fast shutdown': False, 'osd max markdown count': 1000, 'osd mclock override recovery settings': True, 'osd mclock profile': 'high_recovery_ops', 'osd objectstore': 'bluestore', 'osd op queue': 'debug_random', 'osd op queue cut off': 'debug_random', 'osd shutdown pgref assert': True}}, 'flavor': 'default', 'fs': 'xfs', 'log-ignorelist': ['\\(MDS_ALL_DOWN\\)', '\\(MDS_UP_LESS_THAN_MAX\\)', 'but it is still running', 'objects unfound and apparently lost', 'overall HEALTH_', '\\(OSDMAP_FLAGS\\)', '\\(OSD_', '\\(PG_', '\\(POOL_', '\\(CACHE_POOL_', '\\(SMALLER_PGP_NUM\\)', '\\(OBJECT_', '\\(SLOW_OPS\\)', '\\(REQUEST_SLOW\\)', '\\(TOO_FEW_PGS\\)', 'slow request', 'timeout on replica', 'late reservation from', 'MON_DOWN', 'OSDMAP_FLAGS', 'OSD_DOWN', 'PG_DEGRADED', 'PG_AVAILABILITY', 'POOL_APP_NOT_ENABLED', '\\(OSD_SLOW_PING_TIME', 'reached quota', 'overall HEALTH_', '\\(CACHE_POOL_NO_HIT_SET\\)', '\\(POOL_FULL\\)', '\\(SMALLER_PGP_NUM\\)', '\\(SLOW_OPS\\)', '\\(CACHE_POOL_NEAR_FULL\\)', '\\(POOL_APP_NOT_ENABLED\\)', '\\(PG_AVAILABILITY\\)', '\\(OBJECT_MISPLACED\\)', 'slow request', 'overall HEALTH_', '\\(PG_', '\\(MON_DOWN\\)', '\\(SLOW_OPS\\)', 'slow request'], 'mon_bind_msgr2': False, 'sha1': '5dcb190da270440ef141d088eb27a9fdb8afc5f2', 'valgrind': {'mds': ['--tool=memcheck'], 'mon': ['--tool=memcheck', '--leak-check=full', '--show-reachable=yes'], 'osd': ['--tool=memcheck']}}, 'ceph-deploy': {'bluestore': True, 'conf': {'client': {'log file': '/var/log/ceph/ceph-$name.$pid.log'}, 'mon': {}, 'osd': {'bdev async discard': True, 'bdev enable discard': True, 'bluestore block size': 96636764160, 'bluestore fsck on mount': True, 'debug bluefs': 20, 'debug bluestore': 20, 'debug rocksdb': 10, 'mon osd backfillfull_ratio': 0.85, 'mon osd full ratio': 0.9, 'mon osd nearfull ratio': 0.8, 'osd failsafe full ratio': 0.95, 'osd objectstore': 'bluestore'}}, 'fs': 'xfs'}, 'install': {'ceph': {'debuginfo': True, 'flavor': 'default', 'sha1': '5dcb190da270440ef141d088eb27a9fdb8afc5f2'}}, 'thrashosds': {'bdev_inject_crash': 2, 'bdev_inject_crash_probability': 0.5}, 'workunit': {'branch': 'main', 'sha1': '1fb5d10608dccee73edd6ef8a2c7b804a097c324'}}, 'owner': 'scheduled_amathuri@teuthology', 'priority': 150, 'repo': 'https://git.ceph.com/ceph-ci.git', 'roles': [['mon.a', 'mon.c', 'mgr.y', 'osd.0', 'osd.1', 'osd.2', 'osd.3', 'client.0', 'node-exporter.a'], ['mon.b', 'mgr.x', 'osd.4', 'osd.5', 'osd.6', 'osd.7', 'client.1', 'prometheus.a', 'node-exporter.b']], 'seed': 6785, 'sha1': '5dcb190da270440ef141d088eb27a9fdb8afc5f2', 'sleep_before_teardown': 0, 'subset': '99/1000', 'suite': 'rados:verify', 'suite_branch': 'main', 'suite_path': '/home/teuthworker/src/git.ceph.com_ceph-c_1fb5d10608dccee73edd6ef8a2c7b804a097c324/qa', 'suite_relpath': 'qa', 'suite_repo': 'https://git.ceph.com/ceph-ci.git', 'suite_sha1': '1fb5d10608dccee73edd6ef8a2c7b804a097c324', 'targets': {'smithi001.front.sepia.ceph.com': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBJLX2RU8lPyaGpFOX5F2XcKa3PqVDhgC0eHcD7GRZ/nAnWQ6WgV2gQWu4nRS1XboDX2zvaMZYEkHmNn70BXWi7o=', 'smithi115.front.sepia.ceph.com': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBNV2pGoe90+assupoD4niScAzAzu4+FXUpDaFYAxOMgNegi3l1KLiZsa3tGXUVJrb6Id8T/FVoGK7g9F8xwHv9c='}, 'tasks': [{'internal.check_packages': None}, {'internal.buildpackages_prep': None}, {'internal.save_config': None}, {'internal.check_lock': None}, {'internal.add_remotes': None}, {'console_log': None}, {'internal.connect': None}, {'internal.push_inventory': None}, {'internal.serialize_remote_roles': None}, {'internal.check_conflict': None}, {'internal.check_ceph_data': None}, {'internal.vm_setup': None}, {'kernel': {'kdb': True, 'sha1': 'distro'}}, {'internal.base': None}, {'internal.archive_upload': None}, {'internal.archive': None}, {'internal.coredump': None}, {'internal.sudo': None}, {'internal.syslog': None}, {'internal.timer': None}, {'pcp': None}, {'selinux': None}, {'ansible.cephlab': None}, {'clock': None}, {'install': None}, {'ceph': None}, {'thrashosds': {'chance_pgnum_grow': 1, 'chance_pgnum_shrink': 1, 'chance_pgpnum_fix': 1, 'timeout': 1200}}, {'workunit': {'clients': {'client.0': ['rados/test.sh']}, 'env': {'ALLOW_TIMEOUTS': '1'}, 'timeout': '6h'}}], 'teuthology': {'fragments_dropped': [], 'meta': {}, 'postmerge': []}, 'teuthology_branch': 'main', 'teuthology_sha1': 'e691533f9cbb33d85b2187bba20d7102f098636d', 'timestamp': '2024-03-25_13:28:33', 'tube': 'smithi', 'user': 'amathuri', 'verbose': False, 'worker_log': '/home/teuthworker/archive/worker_logs/dispatcher.smithi.2226885'} 2024-03-25T18:25:48.623 DEBUG:teuthology.orchestra.run.smithi001:> install -d -m0755 -- /home/ubuntu/cephtest/ceph.data 2024-03-25T18:25:48.625 DEBUG:teuthology.orchestra.run.smithi115:> install -d -m0755 -- /home/ubuntu/cephtest/ceph.data 2024-03-25T18:25:48.639 DEBUG:teuthology.orchestra.run.smithi001:> sudo install -d -m0777 -- /var/run/ceph 2024-03-25T18:25:48.680 DEBUG:teuthology.orchestra.run.smithi115:> sudo install -d -m0777 -- /var/run/ceph 2024-03-25T18:25:48.706 DEBUG:teuthology.orchestra.run.smithi001:> set -ex 2024-03-25T18:25:48.706 DEBUG:teuthology.orchestra.run.smithi001:> dd if=/scratch_devs of=/dev/stdout 2024-03-25T18:25:48.760 DEBUG:teuthology.misc:devs=['/dev/vg_nvme/lv_1', '/dev/vg_nvme/lv_2', '/dev/vg_nvme/lv_3', '/dev/vg_nvme/lv_4'] 2024-03-25T18:25:48.761 DEBUG:teuthology.orchestra.run.smithi001:> stat /dev/vg_nvme/lv_1 2024-03-25T18:25:48.815 INFO:teuthology.orchestra.run.smithi001.stdout: File: /dev/vg_nvme/lv_1 -> ../dm-0 2024-03-25T18:25:48.815 INFO:teuthology.orchestra.run.smithi001.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-03-25T18:25:48.815 INFO:teuthology.orchestra.run.smithi001.stdout:Device: 5h/5d Inode: 667 Links: 1 2024-03-25T18:25:48.815 INFO:teuthology.orchestra.run.smithi001.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-03-25T18:25:48.815 INFO:teuthology.orchestra.run.smithi001.stdout:Context: system_u:object_r:device_t:s0 2024-03-25T18:25:48.815 INFO:teuthology.orchestra.run.smithi001.stdout:Access: 2024-03-25 16:38:56.331556930 +0000 2024-03-25T18:25:48.815 INFO:teuthology.orchestra.run.smithi001.stdout:Modify: 2024-03-25 16:31:48.948407875 +0000 2024-03-25T18:25:48.815 INFO:teuthology.orchestra.run.smithi001.stdout:Change: 2024-03-25 16:31:48.948407875 +0000 2024-03-25T18:25:48.815 INFO:teuthology.orchestra.run.smithi001.stdout: Birth: 2024-03-25 16:31:48.948407875 +0000 2024-03-25T18:25:48.816 DEBUG:teuthology.orchestra.run.smithi001:> sudo dd if=/dev/vg_nvme/lv_1 of=/dev/null count=1 2024-03-25T18:25:48.881 INFO:teuthology.orchestra.run.smithi001.stderr:1+0 records in 2024-03-25T18:25:48.881 INFO:teuthology.orchestra.run.smithi001.stderr:1+0 records out 2024-03-25T18:25:48.881 INFO:teuthology.orchestra.run.smithi001.stderr:512 bytes copied, 8.9136e-05 s, 5.7 MB/s 2024-03-25T18:25:48.882 DEBUG:teuthology.orchestra.run.smithi001:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_1 2024-03-25T18:25:48.938 DEBUG:teuthology.orchestra.run.smithi001:> stat /dev/vg_nvme/lv_2 2024-03-25T18:25:48.992 INFO:teuthology.orchestra.run.smithi001.stdout: File: /dev/vg_nvme/lv_2 -> ../dm-1 2024-03-25T18:25:48.993 INFO:teuthology.orchestra.run.smithi001.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-03-25T18:25:48.993 INFO:teuthology.orchestra.run.smithi001.stdout:Device: 5h/5d Inode: 687 Links: 1 2024-03-25T18:25:48.993 INFO:teuthology.orchestra.run.smithi001.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-03-25T18:25:48.993 INFO:teuthology.orchestra.run.smithi001.stdout:Context: system_u:object_r:device_t:s0 2024-03-25T18:25:48.993 INFO:teuthology.orchestra.run.smithi001.stdout:Access: 2024-03-25 16:38:56.332556916 +0000 2024-03-25T18:25:48.993 INFO:teuthology.orchestra.run.smithi001.stdout:Modify: 2024-03-25 16:31:49.276403606 +0000 2024-03-25T18:25:48.993 INFO:teuthology.orchestra.run.smithi001.stdout:Change: 2024-03-25 16:31:49.276403606 +0000 2024-03-25T18:25:48.993 INFO:teuthology.orchestra.run.smithi001.stdout: Birth: 2024-03-25 16:31:49.276403606 +0000 2024-03-25T18:25:48.993 DEBUG:teuthology.orchestra.run.smithi001:> sudo dd if=/dev/vg_nvme/lv_2 of=/dev/null count=1 2024-03-25T18:25:49.056 INFO:teuthology.orchestra.run.smithi001.stderr:1+0 records in 2024-03-25T18:25:49.057 INFO:teuthology.orchestra.run.smithi001.stderr:1+0 records out 2024-03-25T18:25:49.057 INFO:teuthology.orchestra.run.smithi001.stderr:512 bytes copied, 0.000126723 s, 4.0 MB/s 2024-03-25T18:25:49.058 DEBUG:teuthology.orchestra.run.smithi001:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_2 2024-03-25T18:25:49.113 DEBUG:teuthology.orchestra.run.smithi001:> stat /dev/vg_nvme/lv_3 2024-03-25T18:25:49.167 INFO:teuthology.orchestra.run.smithi001.stdout: File: /dev/vg_nvme/lv_3 -> ../dm-2 2024-03-25T18:25:49.167 INFO:teuthology.orchestra.run.smithi001.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-03-25T18:25:49.168 INFO:teuthology.orchestra.run.smithi001.stdout:Device: 5h/5d Inode: 705 Links: 1 2024-03-25T18:25:49.168 INFO:teuthology.orchestra.run.smithi001.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-03-25T18:25:49.168 INFO:teuthology.orchestra.run.smithi001.stdout:Context: system_u:object_r:device_t:s0 2024-03-25T18:25:49.168 INFO:teuthology.orchestra.run.smithi001.stdout:Access: 2024-03-25 16:38:56.332556916 +0000 2024-03-25T18:25:49.168 INFO:teuthology.orchestra.run.smithi001.stdout:Modify: 2024-03-25 16:31:49.613399221 +0000 2024-03-25T18:25:49.168 INFO:teuthology.orchestra.run.smithi001.stdout:Change: 2024-03-25 16:31:49.613399221 +0000 2024-03-25T18:25:49.168 INFO:teuthology.orchestra.run.smithi001.stdout: Birth: 2024-03-25 16:31:49.613399221 +0000 2024-03-25T18:25:49.168 DEBUG:teuthology.orchestra.run.smithi001:> sudo dd if=/dev/vg_nvme/lv_3 of=/dev/null count=1 2024-03-25T18:25:49.232 INFO:teuthology.orchestra.run.smithi001.stderr:1+0 records in 2024-03-25T18:25:49.232 INFO:teuthology.orchestra.run.smithi001.stderr:1+0 records out 2024-03-25T18:25:49.232 INFO:teuthology.orchestra.run.smithi001.stderr:512 bytes copied, 9.2825e-05 s, 5.5 MB/s 2024-03-25T18:25:49.233 DEBUG:teuthology.orchestra.run.smithi001:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_3 2024-03-25T18:25:49.288 DEBUG:teuthology.orchestra.run.smithi001:> stat /dev/vg_nvme/lv_4 2024-03-25T18:25:49.343 INFO:teuthology.orchestra.run.smithi001.stdout: File: /dev/vg_nvme/lv_4 -> ../dm-3 2024-03-25T18:25:49.343 INFO:teuthology.orchestra.run.smithi001.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-03-25T18:25:49.343 INFO:teuthology.orchestra.run.smithi001.stdout:Device: 5h/5d Inode: 723 Links: 1 2024-03-25T18:25:49.343 INFO:teuthology.orchestra.run.smithi001.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-03-25T18:25:49.343 INFO:teuthology.orchestra.run.smithi001.stdout:Context: system_u:object_r:device_t:s0 2024-03-25T18:25:49.343 INFO:teuthology.orchestra.run.smithi001.stdout:Access: 2024-03-25 16:38:56.333556904 +0000 2024-03-25T18:25:49.343 INFO:teuthology.orchestra.run.smithi001.stdout:Modify: 2024-03-25 16:31:49.944394913 +0000 2024-03-25T18:25:49.343 INFO:teuthology.orchestra.run.smithi001.stdout:Change: 2024-03-25 16:31:49.944394913 +0000 2024-03-25T18:25:49.343 INFO:teuthology.orchestra.run.smithi001.stdout: Birth: 2024-03-25 16:31:49.944394913 +0000 2024-03-25T18:25:49.344 DEBUG:teuthology.orchestra.run.smithi001:> sudo dd if=/dev/vg_nvme/lv_4 of=/dev/null count=1 2024-03-25T18:25:49.408 INFO:teuthology.orchestra.run.smithi001.stderr:1+0 records in 2024-03-25T18:25:49.408 INFO:teuthology.orchestra.run.smithi001.stderr:1+0 records out 2024-03-25T18:25:49.408 INFO:teuthology.orchestra.run.smithi001.stderr:512 bytes copied, 9.6498e-05 s, 5.3 MB/s 2024-03-25T18:25:49.409 DEBUG:teuthology.orchestra.run.smithi001:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_4 2024-03-25T18:25:49.464 INFO:tasks.ceph:osd dev map: {'osd.0': '/dev/vg_nvme/lv_1', 'osd.1': '/dev/vg_nvme/lv_2', 'osd.2': '/dev/vg_nvme/lv_3', 'osd.3': '/dev/vg_nvme/lv_4'} 2024-03-25T18:25:49.464 DEBUG:teuthology.orchestra.run.smithi115:> set -ex 2024-03-25T18:25:49.464 DEBUG:teuthology.orchestra.run.smithi115:> dd if=/scratch_devs of=/dev/stdout 2024-03-25T18:25:49.481 DEBUG:teuthology.misc:devs=['/dev/vg_nvme/lv_1', '/dev/vg_nvme/lv_2', '/dev/vg_nvme/lv_3', '/dev/vg_nvme/lv_4'] 2024-03-25T18:25:49.481 DEBUG:teuthology.orchestra.run.smithi115:> stat /dev/vg_nvme/lv_1 2024-03-25T18:25:49.538 INFO:teuthology.orchestra.run.smithi115.stdout: File: /dev/vg_nvme/lv_1 -> ../dm-0 2024-03-25T18:25:49.538 INFO:teuthology.orchestra.run.smithi115.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-03-25T18:25:49.538 INFO:teuthology.orchestra.run.smithi115.stdout:Device: 5h/5d Inode: 666 Links: 1 2024-03-25T18:25:49.538 INFO:teuthology.orchestra.run.smithi115.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-03-25T18:25:49.538 INFO:teuthology.orchestra.run.smithi115.stdout:Context: system_u:object_r:device_t:s0 2024-03-25T18:25:49.538 INFO:teuthology.orchestra.run.smithi115.stdout:Access: 2024-03-25 16:38:37.518709045 +0000 2024-03-25T18:25:49.539 INFO:teuthology.orchestra.run.smithi115.stdout:Modify: 2024-03-25 16:31:46.119390213 +0000 2024-03-25T18:25:49.539 INFO:teuthology.orchestra.run.smithi115.stdout:Change: 2024-03-25 16:31:46.119390213 +0000 2024-03-25T18:25:49.539 INFO:teuthology.orchestra.run.smithi115.stdout: Birth: 2024-03-25 16:31:46.119390213 +0000 2024-03-25T18:25:49.539 DEBUG:teuthology.orchestra.run.smithi115:> sudo dd if=/dev/vg_nvme/lv_1 of=/dev/null count=1 2024-03-25T18:25:49.606 INFO:teuthology.orchestra.run.smithi115.stderr:1+0 records in 2024-03-25T18:25:49.606 INFO:teuthology.orchestra.run.smithi115.stderr:1+0 records out 2024-03-25T18:25:49.606 INFO:teuthology.orchestra.run.smithi115.stderr:512 bytes copied, 0.000216325 s, 2.4 MB/s 2024-03-25T18:25:49.607 DEBUG:teuthology.orchestra.run.smithi115:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_1 2024-03-25T18:25:49.664 DEBUG:teuthology.orchestra.run.smithi115:> stat /dev/vg_nvme/lv_2 2024-03-25T18:25:49.720 INFO:teuthology.orchestra.run.smithi115.stdout: File: /dev/vg_nvme/lv_2 -> ../dm-1 2024-03-25T18:25:49.721 INFO:teuthology.orchestra.run.smithi115.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-03-25T18:25:49.721 INFO:teuthology.orchestra.run.smithi115.stdout:Device: 5h/5d Inode: 687 Links: 1 2024-03-25T18:25:49.721 INFO:teuthology.orchestra.run.smithi115.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-03-25T18:25:49.721 INFO:teuthology.orchestra.run.smithi115.stdout:Context: system_u:object_r:device_t:s0 2024-03-25T18:25:49.721 INFO:teuthology.orchestra.run.smithi115.stdout:Access: 2024-03-25 16:38:37.518709045 +0000 2024-03-25T18:25:49.721 INFO:teuthology.orchestra.run.smithi115.stdout:Modify: 2024-03-25 16:31:46.478383261 +0000 2024-03-25T18:25:49.721 INFO:teuthology.orchestra.run.smithi115.stdout:Change: 2024-03-25 16:31:46.478383261 +0000 2024-03-25T18:25:49.721 INFO:teuthology.orchestra.run.smithi115.stdout: Birth: 2024-03-25 16:31:46.478383261 +0000 2024-03-25T18:25:49.722 DEBUG:teuthology.orchestra.run.smithi115:> sudo dd if=/dev/vg_nvme/lv_2 of=/dev/null count=1 2024-03-25T18:25:49.786 INFO:teuthology.orchestra.run.smithi115.stderr:1+0 records in 2024-03-25T18:25:49.787 INFO:teuthology.orchestra.run.smithi115.stderr:1+0 records out 2024-03-25T18:25:49.787 INFO:teuthology.orchestra.run.smithi115.stderr:512 bytes copied, 0.000188412 s, 2.7 MB/s 2024-03-25T18:25:49.788 DEBUG:teuthology.orchestra.run.smithi115:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_2 2024-03-25T18:25:49.844 DEBUG:teuthology.orchestra.run.smithi115:> stat /dev/vg_nvme/lv_3 2024-03-25T18:25:49.899 INFO:teuthology.orchestra.run.smithi115.stdout: File: /dev/vg_nvme/lv_3 -> ../dm-2 2024-03-25T18:25:49.899 INFO:teuthology.orchestra.run.smithi115.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-03-25T18:25:49.899 INFO:teuthology.orchestra.run.smithi115.stdout:Device: 5h/5d Inode: 704 Links: 1 2024-03-25T18:25:49.899 INFO:teuthology.orchestra.run.smithi115.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-03-25T18:25:49.899 INFO:teuthology.orchestra.run.smithi115.stdout:Context: system_u:object_r:device_t:s0 2024-03-25T18:25:49.899 INFO:teuthology.orchestra.run.smithi115.stdout:Access: 2024-03-25 16:38:37.519709025 +0000 2024-03-25T18:25:49.899 INFO:teuthology.orchestra.run.smithi115.stdout:Modify: 2024-03-25 16:31:46.811376813 +0000 2024-03-25T18:25:49.900 INFO:teuthology.orchestra.run.smithi115.stdout:Change: 2024-03-25 16:31:46.811376813 +0000 2024-03-25T18:25:49.900 INFO:teuthology.orchestra.run.smithi115.stdout: Birth: 2024-03-25 16:31:46.811376813 +0000 2024-03-25T18:25:49.900 DEBUG:teuthology.orchestra.run.smithi115:> sudo dd if=/dev/vg_nvme/lv_3 of=/dev/null count=1 2024-03-25T18:25:49.965 INFO:teuthology.orchestra.run.smithi115.stderr:1+0 records in 2024-03-25T18:25:49.966 INFO:teuthology.orchestra.run.smithi115.stderr:1+0 records out 2024-03-25T18:25:49.966 INFO:teuthology.orchestra.run.smithi115.stderr:512 bytes copied, 0.000203351 s, 2.5 MB/s 2024-03-25T18:25:49.967 DEBUG:teuthology.orchestra.run.smithi115:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_3 2024-03-25T18:25:50.023 DEBUG:teuthology.orchestra.run.smithi115:> stat /dev/vg_nvme/lv_4 2024-03-25T18:25:50.078 INFO:teuthology.orchestra.run.smithi115.stdout: File: /dev/vg_nvme/lv_4 -> ../dm-3 2024-03-25T18:25:50.078 INFO:teuthology.orchestra.run.smithi115.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-03-25T18:25:50.078 INFO:teuthology.orchestra.run.smithi115.stdout:Device: 5h/5d Inode: 725 Links: 1 2024-03-25T18:25:50.079 INFO:teuthology.orchestra.run.smithi115.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-03-25T18:25:50.079 INFO:teuthology.orchestra.run.smithi115.stdout:Context: system_u:object_r:device_t:s0 2024-03-25T18:25:50.079 INFO:teuthology.orchestra.run.smithi115.stdout:Access: 2024-03-25 16:38:37.519709025 +0000 2024-03-25T18:25:50.079 INFO:teuthology.orchestra.run.smithi115.stdout:Modify: 2024-03-25 16:31:47.153370190 +0000 2024-03-25T18:25:50.079 INFO:teuthology.orchestra.run.smithi115.stdout:Change: 2024-03-25 16:31:47.153370190 +0000 2024-03-25T18:25:50.079 INFO:teuthology.orchestra.run.smithi115.stdout: Birth: 2024-03-25 16:31:47.153370190 +0000 2024-03-25T18:25:50.079 DEBUG:teuthology.orchestra.run.smithi115:> sudo dd if=/dev/vg_nvme/lv_4 of=/dev/null count=1 2024-03-25T18:25:50.144 INFO:teuthology.orchestra.run.smithi115.stderr:1+0 records in 2024-03-25T18:25:50.144 INFO:teuthology.orchestra.run.smithi115.stderr:1+0 records out 2024-03-25T18:25:50.144 INFO:teuthology.orchestra.run.smithi115.stderr:512 bytes copied, 0.00023342 s, 2.2 MB/s 2024-03-25T18:25:50.145 DEBUG:teuthology.orchestra.run.smithi115:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_4 2024-03-25T18:25:50.201 INFO:tasks.ceph:osd dev map: {'osd.4': '/dev/vg_nvme/lv_1', 'osd.5': '/dev/vg_nvme/lv_2', 'osd.6': '/dev/vg_nvme/lv_3', 'osd.7': '/dev/vg_nvme/lv_4'} 2024-03-25T18:25:50.201 INFO:tasks.ceph:remote_to_roles_to_devs: {Remote(name='ubuntu@smithi001.front.sepia.ceph.com'): {'osd.0': '/dev/vg_nvme/lv_1', 'osd.1': '/dev/vg_nvme/lv_2', 'osd.2': '/dev/vg_nvme/lv_3', 'osd.3': '/dev/vg_nvme/lv_4'}, Remote(name='ubuntu@smithi115.front.sepia.ceph.com'): {'osd.4': '/dev/vg_nvme/lv_1', 'osd.5': '/dev/vg_nvme/lv_2', 'osd.6': '/dev/vg_nvme/lv_3', 'osd.7': '/dev/vg_nvme/lv_4'}} 2024-03-25T18:25:50.201 INFO:tasks.ceph:Generating config... 2024-03-25T18:25:50.203 INFO:tasks.ceph:[client] debug monc = 20 2024-03-25T18:25:50.203 INFO:tasks.ceph:[client] debug ms = 1 2024-03-25T18:25:50.203 INFO:tasks.ceph:[client] debug objecter = 20 2024-03-25T18:25:50.203 INFO:tasks.ceph:[client] debug rados = 20 2024-03-25T18:25:50.203 INFO:tasks.ceph:[global] mon client directed command retry = 5 2024-03-25T18:25:50.203 INFO:tasks.ceph:[global] mon election default strategy = 3 2024-03-25T18:25:50.203 INFO:tasks.ceph:[global] ms bind msgr2 = False 2024-03-25T18:25:50.203 INFO:tasks.ceph:[global] ms inject socket failures = 5000 2024-03-25T18:25:50.203 INFO:tasks.ceph:[global] ms type = async 2024-03-25T18:25:50.203 INFO:tasks.ceph:[global] osd heartbeat grace = 80 2024-03-25T18:25:50.203 INFO:tasks.ceph:[mgr] debug mgr = 20 2024-03-25T18:25:50.203 INFO:tasks.ceph:[mgr] debug ms = 1 2024-03-25T18:25:50.203 INFO:tasks.ceph:[mon] debug mon = 20 2024-03-25T18:25:50.203 INFO:tasks.ceph:[mon] debug ms = 1 2024-03-25T18:25:50.204 INFO:tasks.ceph:[mon] debug paxos = 20 2024-03-25T18:25:50.204 INFO:tasks.ceph:[mon] mon min osdmap epochs = 50 2024-03-25T18:25:50.204 INFO:tasks.ceph:[mon] mon osd crush smoke test = False 2024-03-25T18:25:50.204 INFO:tasks.ceph:[mon] mon osdmap full prune interval = 2 2024-03-25T18:25:50.204 INFO:tasks.ceph:[mon] mon osdmap full prune min = 15 2024-03-25T18:25:50.204 INFO:tasks.ceph:[mon] mon osdmap full prune txsize = 2 2024-03-25T18:25:50.204 INFO:tasks.ceph:[mon] mon scrub interval = 300 2024-03-25T18:25:50.204 INFO:tasks.ceph:[mon] mon warn on pool no app = False 2024-03-25T18:25:50.204 INFO:tasks.ceph:[mon] paxos service trim min = 10 2024-03-25T18:25:50.204 INFO:tasks.ceph:[osd] bdev async discard = True 2024-03-25T18:25:50.204 INFO:tasks.ceph:[osd] bdev enable discard = True 2024-03-25T18:25:50.204 INFO:tasks.ceph:[osd] bluestore allocator = stupid 2024-03-25T18:25:50.204 INFO:tasks.ceph:[osd] bluestore block size = 96636764160 2024-03-25T18:25:50.204 INFO:tasks.ceph:[osd] bluestore fsck on mount = True 2024-03-25T18:25:50.204 INFO:tasks.ceph:[osd] bluestore zero block detection = True 2024-03-25T18:25:50.205 INFO:tasks.ceph:[osd] debug bluefs = 1 2024-03-25T18:25:50.205 INFO:tasks.ceph:[osd] debug bluestore = 1 2024-03-25T18:25:50.205 INFO:tasks.ceph:[osd] debug monc = 20 2024-03-25T18:25:50.205 INFO:tasks.ceph:[osd] debug ms = 1 2024-03-25T18:25:50.205 INFO:tasks.ceph:[osd] debug osd = 20 2024-03-25T18:25:50.205 INFO:tasks.ceph:[osd] debug rocksdb = 10 2024-03-25T18:25:50.205 INFO:tasks.ceph:[osd] mon osd backfillfull_ratio = 0.85 2024-03-25T18:25:50.205 INFO:tasks.ceph:[osd] mon osd full ratio = 0.9 2024-03-25T18:25:50.205 INFO:tasks.ceph:[osd] mon osd nearfull ratio = 0.8 2024-03-25T18:25:50.206 INFO:tasks.ceph:[osd] osd blocked scrub grace period = 3600 2024-03-25T18:25:50.206 INFO:tasks.ceph:[osd] osd class default list = * 2024-03-25T18:25:50.206 INFO:tasks.ceph:[osd] osd class load list = * 2024-03-25T18:25:50.206 INFO:tasks.ceph:[osd] osd client watch timeout = 120 2024-03-25T18:25:50.206 INFO:tasks.ceph:[osd] osd debug verify cached snaps = True 2024-03-25T18:25:50.206 INFO:tasks.ceph:[osd] osd debug verify missing on start = True 2024-03-25T18:25:50.206 INFO:tasks.ceph:[osd] osd failsafe full ratio = 0.95 2024-03-25T18:25:50.206 INFO:tasks.ceph:[osd] osd fast shutdown = False 2024-03-25T18:25:50.206 INFO:tasks.ceph:[osd] osd max markdown count = 1000 2024-03-25T18:25:50.206 INFO:tasks.ceph:[osd] osd mclock override recovery settings = True 2024-03-25T18:25:50.206 INFO:tasks.ceph:[osd] osd mclock profile = high_recovery_ops 2024-03-25T18:25:50.206 INFO:tasks.ceph:[osd] osd objectstore = bluestore 2024-03-25T18:25:50.206 INFO:tasks.ceph:[osd] osd op queue = debug_random 2024-03-25T18:25:50.206 INFO:tasks.ceph:[osd] osd op queue cut off = debug_random 2024-03-25T18:25:50.207 INFO:tasks.ceph:[osd] osd shutdown pgref assert = True 2024-03-25T18:25:50.207 INFO:tasks.ceph:Setting up mon.a... 2024-03-25T18:25:50.207 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring /etc/ceph/ceph.keyring 2024-03-25T18:25:50.256 INFO:teuthology.orchestra.run.smithi001.stdout:creating /etc/ceph/ceph.keyring 2024-03-25T18:25:50.259 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --gen-key --name=mon. /etc/ceph/ceph.keyring 2024-03-25T18:25:50.303 DEBUG:teuthology.orchestra.run.smithi001:> sudo chmod 0644 /etc/ceph/ceph.keyring 2024-03-25T18:25:50.370 DEBUG:tasks.ceph:Ceph mon addresses: [('mon.a', '[v1:172.21.15.1:6789]'), ('mon.c', '[v1:172.21.15.1:6790]'), ('mon.b', '[v1:172.21.15.115:6789]')] 2024-03-25T18:25:50.371 DEBUG:tasks.ceph:writing out conf {'global': {'chdir': '', 'pid file': '/var/run/ceph/$cluster-$name.pid', 'auth supported': 'cephx', 'filestore xattr use omap': 'true', 'mon clock drift allowed': '1.000', 'osd crush chooseleaf type': '0', 'auth debug': 'true', 'ms die on old message': 'true', 'ms die on bug': 'true', 'mon max pg per osd': '10000', 'mon pg warn max object skew': '0', 'osd_pool_default_pg_autoscale_mode': 'off', 'osd pool default size': '2', 'mon osd allow primary affinity': 'true', 'mon osd allow pg remap': 'true', 'mon warn on legacy crush tunables': 'false', 'mon warn on crush straw calc version zero': 'false', 'mon warn on no sortbitwise': 'false', 'mon warn on osd down out interval zero': 'false', 'mon warn on too few osds': 'false', 'mon_warn_on_pool_pg_num_not_power_of_two': 'false', 'mon_warn_on_pool_no_redundancy': 'false', 'mon_allow_pool_size_one': 'true', 'osd pool default erasure code profile': 'plugin=jerasure technique=reed_sol_van k=2 m=1 crush-failure-domain=osd', 'osd default data pool replay window': '5', 'mon allow pool delete': 'true', 'mon cluster log file level': 'debug', 'debug asserts on shutdown': 'true', 'mon health detail to clog': 'false', 'mon host': '[v1:172.21.15.1:6789],[v1:172.21.15.1:6790],[v1:172.21.15.115:6789]', 'mon client directed command retry': 5, 'mon election default strategy': 3, 'ms bind msgr2': False, 'ms inject socket failures': 5000, 'ms type': 'async', 'osd heartbeat grace': 80}, 'osd': {'osd journal size': '100', 'osd scrub load threshold': '5.0', 'osd scrub max interval': '600', 'osd mclock profile': 'high_recovery_ops', 'osd recover clone overlap': 'true', 'osd recovery max chunk': '1048576', 'osd debug shutdown': 'true', 'osd debug op order': 'true', 'osd debug verify stray on activate': 'true', 'osd debug trim objects': 'true', 'osd open classes on start': 'true', 'osd debug pg log writeout': 'true', 'osd deep scrub update digest min age': '30', 'osd map max advance': '10', 'journal zero on create': 'true', 'filestore ondisk finisher threads': '3', 'filestore apply finisher threads': '3', 'bdev debug aio': 'true', 'osd debug misdirected ops': 'true', 'bdev async discard': True, 'bdev enable discard': True, 'bluestore allocator': 'stupid', 'bluestore block size': 96636764160, 'bluestore fsck on mount': True, 'bluestore zero block detection': True, 'debug bluefs': 1, 'debug bluestore': 1, 'debug monc': 20, 'debug ms': 1, 'debug osd': 20, 'debug rocksdb': 10, 'mon osd backfillfull_ratio': 0.85, 'mon osd full ratio': 0.9, 'mon osd nearfull ratio': 0.8, 'osd blocked scrub grace period': 3600, 'osd class default list': '*', 'osd class load list': '*', 'osd client watch timeout': 120, 'osd debug verify cached snaps': True, 'osd debug verify missing on start': True, 'osd failsafe full ratio': 0.95, 'osd fast shutdown': False, 'osd max markdown count': 1000, 'osd mclock override recovery settings': True, 'osd objectstore': 'bluestore', 'osd op queue': 'debug_random', 'osd op queue cut off': 'debug_random', 'osd shutdown pgref assert': True}, 'mgr': {'debug ms': 1, 'debug mgr': 20, 'debug mon': '20', 'debug auth': '20', 'mon reweight min pgs per osd': '4', 'mon reweight min bytes per osd': '10', 'mgr/telemetry/nag': 'false'}, 'mon': {'debug ms': 1, 'debug mon': 20, 'debug paxos': 20, 'debug auth': '20', 'mon data avail warn': '5', 'mon mgr mkfs grace': '240', 'mon reweight min pgs per osd': '4', 'mon osd reporter subtree level': 'osd', 'mon osd prime pg temp': 'true', 'mon reweight min bytes per osd': '10', 'auth mon ticket ttl': '660', 'auth service ticket ttl': '240', 'mon_warn_on_insecure_global_id_reclaim': 'false', 'mon_warn_on_insecure_global_id_reclaim_allowed': 'false', 'mon_down_mkfs_grace': '2m', 'mon_warn_on_filestore_osds': 'false', 'mon min osdmap epochs': 50, 'mon osd crush smoke test': False, 'mon osdmap full prune interval': 2, 'mon osdmap full prune min': 15, 'mon osdmap full prune txsize': 2, 'mon scrub interval': 300, 'mon warn on pool no app': False, 'paxos service trim min': 10}, 'client': {'rgw cache enabled': 'true', 'rgw enable ops log': 'true', 'rgw enable usage log': 'true', 'log file': '/var/log/ceph/$cluster-$name.$pid.log', 'admin socket': '/var/run/ceph/$cluster-$name.$pid.asok', 'debug monc': 20, 'debug ms': 1, 'debug objecter': 20, 'debug rados': 20}, 'mon.a': {}, 'mon.c': {}, 'mon.b': {}} 2024-03-25T18:25:50.372 DEBUG:teuthology.orchestra.run.smithi001:> set -ex 2024-03-25T18:25:50.372 DEBUG:teuthology.orchestra.run.smithi001:> dd of=/home/ubuntu/cephtest/ceph.tmp.conf 2024-03-25T18:25:50.425 DEBUG:teuthology.orchestra.run.smithi001:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage monmaptool -c /home/ubuntu/cephtest/ceph.tmp.conf --create --clobber --enable-all-features --addv a '[v1:172.21.15.1:6789]' --addv c '[v1:172.21.15.1:6790]' --addv b '[v1:172.21.15.115:6789]' --print /home/ubuntu/cephtest/ceph.monmap 2024-03-25T18:25:50.510 INFO:teuthology.orchestra.run.smithi001.stdout:monmaptool: monmap file /home/ubuntu/cephtest/ceph.monmap 2024-03-25T18:25:50.510 INFO:teuthology.orchestra.run.smithi001.stdout:monmaptool: generated fsid c8ea72cf-9e99-4383-9b52-6c23d365fdfc 2024-03-25T18:25:50.510 INFO:teuthology.orchestra.run.smithi001.stdout:setting min_mon_release = quincy 2024-03-25T18:25:50.510 INFO:teuthology.orchestra.run.smithi001.stdout:epoch 0 2024-03-25T18:25:50.510 INFO:teuthology.orchestra.run.smithi001.stdout:fsid c8ea72cf-9e99-4383-9b52-6c23d365fdfc 2024-03-25T18:25:50.511 INFO:teuthology.orchestra.run.smithi001.stdout:last_changed 2024-03-25T18:25:50.510519+0000 2024-03-25T18:25:50.511 INFO:teuthology.orchestra.run.smithi001.stdout:created 2024-03-25T18:25:50.510519+0000 2024-03-25T18:25:50.511 INFO:teuthology.orchestra.run.smithi001.stdout:min_mon_release 17 (quincy) 2024-03-25T18:25:50.511 INFO:teuthology.orchestra.run.smithi001.stdout:election_strategy: 3 2024-03-25T18:25:50.511 INFO:teuthology.orchestra.run.smithi001.stdout:0: v1:172.21.15.1:6789/0 mon.a 2024-03-25T18:25:50.511 INFO:teuthology.orchestra.run.smithi001.stdout:1: v1:172.21.15.115:6789/0 mon.b 2024-03-25T18:25:50.511 INFO:teuthology.orchestra.run.smithi001.stdout:2: v1:172.21.15.1:6790/0 mon.c 2024-03-25T18:25:50.511 INFO:teuthology.orchestra.run.smithi001.stdout:monmaptool: writing epoch 0 to /home/ubuntu/cephtest/ceph.monmap (3 monitors) 2024-03-25T18:25:50.513 DEBUG:teuthology.orchestra.run.smithi001:> rm -- /home/ubuntu/cephtest/ceph.tmp.conf 2024-03-25T18:25:50.527 INFO:tasks.ceph:Writing /etc/ceph/ceph.conf for FSID c8ea72cf-9e99-4383-9b52-6c23d365fdfc... 2024-03-25T18:25:50.528 DEBUG:teuthology.orchestra.run.smithi001:> sudo mkdir -p /etc/ceph && sudo chmod 0755 /etc/ceph && sudo tee /etc/ceph/ceph.conf && sudo chmod 0644 /etc/ceph/ceph.conf > /dev/null 2024-03-25T18:25:50.570 DEBUG:teuthology.orchestra.run.smithi115:> sudo mkdir -p /etc/ceph && sudo chmod 0755 /etc/ceph && sudo tee /etc/ceph/ceph.conf && sudo chmod 0644 /etc/ceph/ceph.conf > /dev/null 2024-03-25T18:25:50.617 INFO:teuthology.orchestra.run.smithi001.stdout:[global] 2024-03-25T18:25:50.617 INFO:teuthology.orchestra.run.smithi001.stdout: chdir = "" 2024-03-25T18:25:50.617 INFO:teuthology.orchestra.run.smithi001.stdout: pid file = /var/run/ceph/$cluster-$name.pid 2024-03-25T18:25:50.617 INFO:teuthology.orchestra.run.smithi001.stdout: auth supported = cephx 2024-03-25T18:25:50.617 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-25T18:25:50.617 INFO:teuthology.orchestra.run.smithi001.stdout: filestore xattr use omap = true 2024-03-25T18:25:50.617 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-25T18:25:50.617 INFO:teuthology.orchestra.run.smithi001.stdout: mon clock drift allowed = 1.000 2024-03-25T18:25:50.617 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-25T18:25:50.617 INFO:teuthology.orchestra.run.smithi001.stdout: osd crush chooseleaf type = 0 2024-03-25T18:25:50.617 INFO:teuthology.orchestra.run.smithi001.stdout: auth debug = true 2024-03-25T18:25:50.617 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-25T18:25:50.618 INFO:teuthology.orchestra.run.smithi001.stdout: ms die on old message = true 2024-03-25T18:25:50.618 INFO:teuthology.orchestra.run.smithi001.stdout: ms die on bug = true 2024-03-25T18:25:50.618 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-25T18:25:50.618 INFO:teuthology.orchestra.run.smithi001.stdout: mon max pg per osd = 10000 # >= luminous 2024-03-25T18:25:50.618 INFO:teuthology.orchestra.run.smithi001.stdout: mon pg warn max object skew = 0 2024-03-25T18:25:50.618 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-25T18:25:50.618 INFO:teuthology.orchestra.run.smithi001.stdout: # disable pg_autoscaler by default for new pools 2024-03-25T18:25:50.618 INFO:teuthology.orchestra.run.smithi001.stdout: osd_pool_default_pg_autoscale_mode = off 2024-03-25T18:25:50.618 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-25T18:25:50.618 INFO:teuthology.orchestra.run.smithi001.stdout: osd pool default size = 2 2024-03-25T18:25:50.618 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-25T18:25:50.618 INFO:teuthology.orchestra.run.smithi001.stdout: mon osd allow primary affinity = true 2024-03-25T18:25:50.618 INFO:teuthology.orchestra.run.smithi001.stdout: mon osd allow pg remap = true 2024-03-25T18:25:50.619 INFO:teuthology.orchestra.run.smithi001.stdout: mon warn on legacy crush tunables = false 2024-03-25T18:25:50.619 INFO:teuthology.orchestra.run.smithi001.stdout: mon warn on crush straw calc version zero = false 2024-03-25T18:25:50.619 INFO:teuthology.orchestra.run.smithi001.stdout: mon warn on no sortbitwise = false 2024-03-25T18:25:50.619 INFO:teuthology.orchestra.run.smithi001.stdout: mon warn on osd down out interval zero = false 2024-03-25T18:25:50.619 INFO:teuthology.orchestra.run.smithi001.stdout: mon warn on too few osds = false 2024-03-25T18:25:50.619 INFO:teuthology.orchestra.run.smithi001.stdout: mon_warn_on_pool_pg_num_not_power_of_two = false 2024-03-25T18:25:50.619 INFO:teuthology.orchestra.run.smithi001.stdout: mon_warn_on_pool_no_redundancy = false 2024-03-25T18:25:50.619 INFO:teuthology.orchestra.run.smithi001.stdout: mon_allow_pool_size_one = true 2024-03-25T18:25:50.619 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-25T18:25:50.619 INFO:teuthology.orchestra.run.smithi001.stdout: osd pool default erasure code profile = plugin=jerasure technique=reed_sol_van k=2 m=1 crush-failure-domain=osd 2024-03-25T18:25:50.619 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-25T18:25:50.619 INFO:teuthology.orchestra.run.smithi001.stdout: osd default data pool replay window = 5 2024-03-25T18:25:50.619 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-25T18:25:50.619 INFO:teuthology.orchestra.run.smithi001.stdout: mon allow pool delete = true 2024-03-25T18:25:50.619 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-25T18:25:50.620 INFO:teuthology.orchestra.run.smithi001.stdout: mon cluster log file level = debug 2024-03-25T18:25:50.620 INFO:teuthology.orchestra.run.smithi001.stdout: debug asserts on shutdown = true 2024-03-25T18:25:50.620 INFO:teuthology.orchestra.run.smithi001.stdout: mon health detail to clog = false 2024-03-25T18:25:50.620 INFO:teuthology.orchestra.run.smithi001.stdout: mon host = "[v1:172.21.15.1:6789],[v1:172.21.15.1:6790],[v1:172.21.15.115:6789]" 2024-03-25T18:25:50.620 INFO:teuthology.orchestra.run.smithi001.stdout: mon client directed command retry = 5 2024-03-25T18:25:50.620 INFO:teuthology.orchestra.run.smithi001.stdout: mon election default strategy = 3 2024-03-25T18:25:50.620 INFO:teuthology.orchestra.run.smithi001.stdout: ms bind msgr2 = False 2024-03-25T18:25:50.620 INFO:teuthology.orchestra.run.smithi001.stdout: ms inject socket failures = 5000 2024-03-25T18:25:50.620 INFO:teuthology.orchestra.run.smithi001.stdout: ms type = async 2024-03-25T18:25:50.620 INFO:teuthology.orchestra.run.smithi001.stdout: osd heartbeat grace = 80 2024-03-25T18:25:50.620 INFO:teuthology.orchestra.run.smithi001.stdout: fsid = c8ea72cf-9e99-4383-9b52-6c23d365fdfc 2024-03-25T18:25:50.620 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-25T18:25:50.620 INFO:teuthology.orchestra.run.smithi001.stdout:[osd] 2024-03-25T18:25:50.620 INFO:teuthology.orchestra.run.smithi001.stdout: osd journal size = 100 2024-03-25T18:25:50.621 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-25T18:25:50.621 INFO:teuthology.orchestra.run.smithi001.stdout: osd scrub load threshold = 5.0 2024-03-25T18:25:50.621 INFO:teuthology.orchestra.run.smithi001.stdout: osd scrub max interval = 600 2024-03-25T18:25:50.621 INFO:teuthology.orchestra.run.smithi001.stdout: osd mclock profile = high_recovery_ops 2024-03-25T18:25:50.621 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-25T18:25:50.621 INFO:teuthology.orchestra.run.smithi001.stdout: osd recover clone overlap = true 2024-03-25T18:25:50.621 INFO:teuthology.orchestra.run.smithi001.stdout: osd recovery max chunk = 1048576 2024-03-25T18:25:50.621 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-25T18:25:50.621 INFO:teuthology.orchestra.run.smithi001.stdout: osd debug shutdown = true 2024-03-25T18:25:50.621 INFO:teuthology.orchestra.run.smithi001.stdout: osd debug op order = true 2024-03-25T18:25:50.621 INFO:teuthology.orchestra.run.smithi001.stdout: osd debug verify stray on activate = true 2024-03-25T18:25:50.621 INFO:teuthology.orchestra.run.smithi001.stdout: osd debug trim objects = true 2024-03-25T18:25:50.621 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-25T18:25:50.621 INFO:teuthology.orchestra.run.smithi001.stdout: osd open classes on start = true 2024-03-25T18:25:50.621 INFO:teuthology.orchestra.run.smithi001.stdout: osd debug pg log writeout = true 2024-03-25T18:25:50.622 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-25T18:25:50.622 INFO:teuthology.orchestra.run.smithi001.stdout: osd deep scrub update digest min age = 30 2024-03-25T18:25:50.622 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-25T18:25:50.622 INFO:teuthology.orchestra.run.smithi001.stdout: osd map max advance = 10 2024-03-25T18:25:50.622 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-25T18:25:50.622 INFO:teuthology.orchestra.run.smithi001.stdout: journal zero on create = true 2024-03-25T18:25:50.622 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-25T18:25:50.622 INFO:teuthology.orchestra.run.smithi001.stdout: filestore ondisk finisher threads = 3 2024-03-25T18:25:50.622 INFO:teuthology.orchestra.run.smithi001.stdout: filestore apply finisher threads = 3 2024-03-25T18:25:50.622 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-25T18:25:50.622 INFO:teuthology.orchestra.run.smithi001.stdout: bdev debug aio = true 2024-03-25T18:25:50.622 INFO:teuthology.orchestra.run.smithi001.stdout: osd debug misdirected ops = true 2024-03-25T18:25:50.622 INFO:teuthology.orchestra.run.smithi001.stdout: bdev async discard = True 2024-03-25T18:25:50.623 INFO:teuthology.orchestra.run.smithi001.stdout: bdev enable discard = True 2024-03-25T18:25:50.623 INFO:teuthology.orchestra.run.smithi001.stdout: bluestore allocator = stupid 2024-03-25T18:25:50.623 INFO:teuthology.orchestra.run.smithi001.stdout: bluestore block size = 96636764160 2024-03-25T18:25:50.623 INFO:teuthology.orchestra.run.smithi001.stdout: bluestore fsck on mount = True 2024-03-25T18:25:50.623 INFO:teuthology.orchestra.run.smithi001.stdout: bluestore zero block detection = True 2024-03-25T18:25:50.623 INFO:teuthology.orchestra.run.smithi001.stdout: debug bluefs = 1 2024-03-25T18:25:50.623 INFO:teuthology.orchestra.run.smithi001.stdout: debug bluestore = 1 2024-03-25T18:25:50.623 INFO:teuthology.orchestra.run.smithi001.stdout: debug monc = 20 2024-03-25T18:25:50.623 INFO:teuthology.orchestra.run.smithi001.stdout: debug ms = 1 2024-03-25T18:25:50.623 INFO:teuthology.orchestra.run.smithi001.stdout: debug osd = 20 2024-03-25T18:25:50.623 INFO:teuthology.orchestra.run.smithi001.stdout: debug rocksdb = 10 2024-03-25T18:25:50.623 INFO:teuthology.orchestra.run.smithi001.stdout: mon osd backfillfull_ratio = 0.85 2024-03-25T18:25:50.623 INFO:teuthology.orchestra.run.smithi001.stdout: mon osd full ratio = 0.9 2024-03-25T18:25:50.623 INFO:teuthology.orchestra.run.smithi001.stdout: mon osd nearfull ratio = 0.8 2024-03-25T18:25:50.623 INFO:teuthology.orchestra.run.smithi001.stdout: osd blocked scrub grace period = 3600 2024-03-25T18:25:50.623 INFO:teuthology.orchestra.run.smithi001.stdout: osd class default list = * 2024-03-25T18:25:50.624 INFO:teuthology.orchestra.run.smithi001.stdout: osd class load list = * 2024-03-25T18:25:50.624 INFO:teuthology.orchestra.run.smithi001.stdout: osd client watch timeout = 120 2024-03-25T18:25:50.624 INFO:teuthology.orchestra.run.smithi001.stdout: osd debug verify cached snaps = True 2024-03-25T18:25:50.624 INFO:teuthology.orchestra.run.smithi001.stdout: osd debug verify missing on start = True 2024-03-25T18:25:50.624 INFO:teuthology.orchestra.run.smithi001.stdout: osd failsafe full ratio = 0.95 2024-03-25T18:25:50.624 INFO:teuthology.orchestra.run.smithi001.stdout: osd fast shutdown = False 2024-03-25T18:25:50.624 INFO:teuthology.orchestra.run.smithi001.stdout: osd max markdown count = 1000 2024-03-25T18:25:50.624 INFO:teuthology.orchestra.run.smithi001.stdout: osd mclock override recovery settings = True 2024-03-25T18:25:50.624 INFO:teuthology.orchestra.run.smithi001.stdout: osd objectstore = bluestore 2024-03-25T18:25:50.624 INFO:teuthology.orchestra.run.smithi001.stdout: osd op queue = debug_random 2024-03-25T18:25:50.624 INFO:teuthology.orchestra.run.smithi001.stdout: osd op queue cut off = debug_random 2024-03-25T18:25:50.624 INFO:teuthology.orchestra.run.smithi001.stdout: osd shutdown pgref assert = True 2024-03-25T18:25:50.624 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-25T18:25:50.624 INFO:teuthology.orchestra.run.smithi001.stdout:[mgr] 2024-03-25T18:25:50.625 INFO:teuthology.orchestra.run.smithi001.stdout: debug ms = 1 2024-03-25T18:25:50.625 INFO:teuthology.orchestra.run.smithi001.stdout: debug mgr = 20 2024-03-25T18:25:50.625 INFO:teuthology.orchestra.run.smithi001.stdout: debug mon = 20 2024-03-25T18:25:50.625 INFO:teuthology.orchestra.run.smithi001.stdout: debug auth = 20 2024-03-25T18:25:50.625 INFO:teuthology.orchestra.run.smithi001.stdout: mon reweight min pgs per osd = 4 2024-03-25T18:25:50.625 INFO:teuthology.orchestra.run.smithi001.stdout: mon reweight min bytes per osd = 10 2024-03-25T18:25:50.625 INFO:teuthology.orchestra.run.smithi001.stdout: mgr/telemetry/nag = false 2024-03-25T18:25:50.625 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-25T18:25:50.625 INFO:teuthology.orchestra.run.smithi001.stdout:[mon] 2024-03-25T18:25:50.625 INFO:teuthology.orchestra.run.smithi001.stdout: debug ms = 1 2024-03-25T18:25:50.625 INFO:teuthology.orchestra.run.smithi001.stdout: debug mon = 20 2024-03-25T18:25:50.625 INFO:teuthology.orchestra.run.smithi001.stdout: debug paxos = 20 2024-03-25T18:25:50.625 INFO:teuthology.orchestra.run.smithi001.stdout: debug auth = 20 2024-03-25T18:25:50.625 INFO:teuthology.orchestra.run.smithi001.stdout: mon data avail warn = 5 2024-03-25T18:25:50.626 INFO:teuthology.orchestra.run.smithi001.stdout: mon mgr mkfs grace = 240 2024-03-25T18:25:50.626 INFO:teuthology.orchestra.run.smithi001.stdout: mon reweight min pgs per osd = 4 2024-03-25T18:25:50.626 INFO:teuthology.orchestra.run.smithi001.stdout: mon osd reporter subtree level = osd 2024-03-25T18:25:50.626 INFO:teuthology.orchestra.run.smithi001.stdout: mon osd prime pg temp = true 2024-03-25T18:25:50.626 INFO:teuthology.orchestra.run.smithi001.stdout: mon reweight min bytes per osd = 10 2024-03-25T18:25:50.626 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-25T18:25:50.626 INFO:teuthology.orchestra.run.smithi001.stdout: # rotate auth tickets quickly to exercise renewal paths 2024-03-25T18:25:50.626 INFO:teuthology.orchestra.run.smithi001.stdout: auth mon ticket ttl = 660 # 11m 2024-03-25T18:25:50.626 INFO:teuthology.orchestra.run.smithi001.stdout: auth service ticket ttl = 240 # 4m 2024-03-25T18:25:50.626 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-25T18:25:50.626 INFO:teuthology.orchestra.run.smithi001.stdout: # don't complain about insecure global_id in the test suite 2024-03-25T18:25:50.626 INFO:teuthology.orchestra.run.smithi001.stdout: mon_warn_on_insecure_global_id_reclaim = false 2024-03-25T18:25:50.626 INFO:teuthology.orchestra.run.smithi001.stdout: mon_warn_on_insecure_global_id_reclaim_allowed = false 2024-03-25T18:25:50.626 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-25T18:25:50.626 INFO:teuthology.orchestra.run.smithi001.stdout: # 1m isn't quite enough 2024-03-25T18:25:50.627 INFO:teuthology.orchestra.run.smithi001.stdout: mon_down_mkfs_grace = 2m 2024-03-25T18:25:50.627 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-25T18:25:50.627 INFO:teuthology.orchestra.run.smithi001.stdout: mon_warn_on_filestore_osds = false 2024-03-25T18:25:50.627 INFO:teuthology.orchestra.run.smithi001.stdout: mon min osdmap epochs = 50 2024-03-25T18:25:50.627 INFO:teuthology.orchestra.run.smithi001.stdout: mon osd crush smoke test = False 2024-03-25T18:25:50.627 INFO:teuthology.orchestra.run.smithi001.stdout: mon osdmap full prune interval = 2 2024-03-25T18:25:50.627 INFO:teuthology.orchestra.run.smithi001.stdout: mon osdmap full prune min = 15 2024-03-25T18:25:50.627 INFO:teuthology.orchestra.run.smithi001.stdout: mon osdmap full prune txsize = 2 2024-03-25T18:25:50.627 INFO:teuthology.orchestra.run.smithi001.stdout: mon scrub interval = 300 2024-03-25T18:25:50.627 INFO:teuthology.orchestra.run.smithi001.stdout: mon warn on pool no app = False 2024-03-25T18:25:50.627 INFO:teuthology.orchestra.run.smithi001.stdout: paxos service trim min = 10 2024-03-25T18:25:50.627 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-25T18:25:50.627 INFO:teuthology.orchestra.run.smithi001.stdout:[client] 2024-03-25T18:25:50.627 INFO:teuthology.orchestra.run.smithi001.stdout: rgw cache enabled = true 2024-03-25T18:25:50.628 INFO:teuthology.orchestra.run.smithi001.stdout: rgw enable ops log = true 2024-03-25T18:25:50.628 INFO:teuthology.orchestra.run.smithi001.stdout: rgw enable usage log = true 2024-03-25T18:25:50.628 INFO:teuthology.orchestra.run.smithi001.stdout: log file = /var/log/ceph/$cluster-$name.$pid.log 2024-03-25T18:25:50.628 INFO:teuthology.orchestra.run.smithi001.stdout: admin socket = /var/run/ceph/$cluster-$name.$pid.asok 2024-03-25T18:25:50.628 INFO:teuthology.orchestra.run.smithi001.stdout: debug monc = 20 2024-03-25T18:25:50.628 INFO:teuthology.orchestra.run.smithi001.stdout: debug ms = 1 2024-03-25T18:25:50.628 INFO:teuthology.orchestra.run.smithi001.stdout: debug objecter = 20 2024-03-25T18:25:50.628 INFO:teuthology.orchestra.run.smithi001.stdout: debug rados = 20 2024-03-25T18:25:50.628 INFO:teuthology.orchestra.run.smithi001.stdout:[mon.a] 2024-03-25T18:25:50.628 INFO:teuthology.orchestra.run.smithi001.stdout:[mon.c] 2024-03-25T18:25:50.628 INFO:teuthology.orchestra.run.smithi001.stdout:[mon.b] 2024-03-25T18:25:50.629 INFO:teuthology.orchestra.run.smithi115.stdout:[global] 2024-03-25T18:25:50.629 INFO:teuthology.orchestra.run.smithi115.stdout: chdir = "" 2024-03-25T18:25:50.629 INFO:teuthology.orchestra.run.smithi115.stdout: pid file = /var/run/ceph/$cluster-$name.pid 2024-03-25T18:25:50.629 INFO:teuthology.orchestra.run.smithi115.stdout: auth supported = cephx 2024-03-25T18:25:50.629 INFO:teuthology.orchestra.run.smithi115.stdout: 2024-03-25T18:25:50.629 INFO:teuthology.orchestra.run.smithi115.stdout: filestore xattr use omap = true 2024-03-25T18:25:50.629 INFO:teuthology.orchestra.run.smithi115.stdout: 2024-03-25T18:25:50.629 INFO:teuthology.orchestra.run.smithi115.stdout: mon clock drift allowed = 1.000 2024-03-25T18:25:50.629 INFO:teuthology.orchestra.run.smithi115.stdout: 2024-03-25T18:25:50.629 INFO:teuthology.orchestra.run.smithi115.stdout: osd crush chooseleaf type = 0 2024-03-25T18:25:50.629 INFO:teuthology.orchestra.run.smithi115.stdout: auth debug = true 2024-03-25T18:25:50.629 INFO:teuthology.orchestra.run.smithi115.stdout: 2024-03-25T18:25:50.630 INFO:teuthology.orchestra.run.smithi115.stdout: ms die on old message = true 2024-03-25T18:25:50.630 INFO:teuthology.orchestra.run.smithi115.stdout: ms die on bug = true 2024-03-25T18:25:50.630 INFO:teuthology.orchestra.run.smithi115.stdout: 2024-03-25T18:25:50.630 INFO:teuthology.orchestra.run.smithi115.stdout: mon max pg per osd = 10000 # >= luminous 2024-03-25T18:25:50.630 INFO:teuthology.orchestra.run.smithi115.stdout: mon pg warn max object skew = 0 2024-03-25T18:25:50.630 INFO:teuthology.orchestra.run.smithi115.stdout: 2024-03-25T18:25:50.630 INFO:teuthology.orchestra.run.smithi115.stdout: # disable pg_autoscaler by default for new pools 2024-03-25T18:25:50.630 INFO:teuthology.orchestra.run.smithi115.stdout: osd_pool_default_pg_autoscale_mode = off 2024-03-25T18:25:50.630 INFO:teuthology.orchestra.run.smithi115.stdout: 2024-03-25T18:25:50.630 INFO:teuthology.orchestra.run.smithi115.stdout: osd pool default size = 2 2024-03-25T18:25:50.630 INFO:teuthology.orchestra.run.smithi115.stdout: 2024-03-25T18:25:50.630 INFO:teuthology.orchestra.run.smithi115.stdout: mon osd allow primary affinity = true 2024-03-25T18:25:50.630 INFO:teuthology.orchestra.run.smithi115.stdout: mon osd allow pg remap = true 2024-03-25T18:25:50.630 INFO:teuthology.orchestra.run.smithi115.stdout: mon warn on legacy crush tunables = false 2024-03-25T18:25:50.631 INFO:teuthology.orchestra.run.smithi115.stdout: mon warn on crush straw calc version zero = false 2024-03-25T18:25:50.631 INFO:teuthology.orchestra.run.smithi115.stdout: mon warn on no sortbitwise = false 2024-03-25T18:25:50.631 INFO:teuthology.orchestra.run.smithi115.stdout: mon warn on osd down out interval zero = false 2024-03-25T18:25:50.631 INFO:teuthology.orchestra.run.smithi115.stdout: mon warn on too few osds = false 2024-03-25T18:25:50.631 INFO:teuthology.orchestra.run.smithi115.stdout: mon_warn_on_pool_pg_num_not_power_of_two = false 2024-03-25T18:25:50.631 INFO:teuthology.orchestra.run.smithi115.stdout: mon_warn_on_pool_no_redundancy = false 2024-03-25T18:25:50.631 INFO:teuthology.orchestra.run.smithi115.stdout: mon_allow_pool_size_one = true 2024-03-25T18:25:50.631 INFO:teuthology.orchestra.run.smithi115.stdout: 2024-03-25T18:25:50.631 INFO:teuthology.orchestra.run.smithi115.stdout: osd pool default erasure code profile = plugin=jerasure technique=reed_sol_van k=2 m=1 crush-failure-domain=osd 2024-03-25T18:25:50.631 INFO:teuthology.orchestra.run.smithi115.stdout: 2024-03-25T18:25:50.631 INFO:teuthology.orchestra.run.smithi115.stdout: osd default data pool replay window = 5 2024-03-25T18:25:50.631 INFO:teuthology.orchestra.run.smithi115.stdout: 2024-03-25T18:25:50.631 INFO:teuthology.orchestra.run.smithi115.stdout: mon allow pool delete = true 2024-03-25T18:25:50.631 INFO:teuthology.orchestra.run.smithi115.stdout: 2024-03-25T18:25:50.632 INFO:teuthology.orchestra.run.smithi115.stdout: mon cluster log file level = debug 2024-03-25T18:25:50.632 INFO:teuthology.orchestra.run.smithi115.stdout: debug asserts on shutdown = true 2024-03-25T18:25:50.632 INFO:teuthology.orchestra.run.smithi115.stdout: mon health detail to clog = false 2024-03-25T18:25:50.632 INFO:teuthology.orchestra.run.smithi115.stdout: mon host = "[v1:172.21.15.1:6789],[v1:172.21.15.1:6790],[v1:172.21.15.115:6789]" 2024-03-25T18:25:50.632 INFO:teuthology.orchestra.run.smithi115.stdout: mon client directed command retry = 5 2024-03-25T18:25:50.632 INFO:teuthology.orchestra.run.smithi115.stdout: mon election default strategy = 3 2024-03-25T18:25:50.632 INFO:teuthology.orchestra.run.smithi115.stdout: ms bind msgr2 = False 2024-03-25T18:25:50.632 INFO:teuthology.orchestra.run.smithi115.stdout: ms inject socket failures = 5000 2024-03-25T18:25:50.632 INFO:teuthology.orchestra.run.smithi115.stdout: ms type = async 2024-03-25T18:25:50.632 INFO:teuthology.orchestra.run.smithi115.stdout: osd heartbeat grace = 80 2024-03-25T18:25:50.632 INFO:teuthology.orchestra.run.smithi115.stdout: fsid = c8ea72cf-9e99-4383-9b52-6c23d365fdfc 2024-03-25T18:25:50.632 INFO:teuthology.orchestra.run.smithi115.stdout: 2024-03-25T18:25:50.632 INFO:teuthology.orchestra.run.smithi115.stdout:[osd] 2024-03-25T18:25:50.632 INFO:teuthology.orchestra.run.smithi115.stdout: osd journal size = 100 2024-03-25T18:25:50.632 INFO:teuthology.orchestra.run.smithi115.stdout: 2024-03-25T18:25:50.632 INFO:teuthology.orchestra.run.smithi115.stdout: osd scrub load threshold = 5.0 2024-03-25T18:25:50.633 INFO:teuthology.orchestra.run.smithi115.stdout: osd scrub max interval = 600 2024-03-25T18:25:50.633 INFO:teuthology.orchestra.run.smithi115.stdout: osd mclock profile = high_recovery_ops 2024-03-25T18:25:50.633 INFO:teuthology.orchestra.run.smithi115.stdout: 2024-03-25T18:25:50.633 INFO:teuthology.orchestra.run.smithi115.stdout: osd recover clone overlap = true 2024-03-25T18:25:50.633 INFO:teuthology.orchestra.run.smithi115.stdout: osd recovery max chunk = 1048576 2024-03-25T18:25:50.633 INFO:teuthology.orchestra.run.smithi115.stdout: 2024-03-25T18:25:50.633 INFO:teuthology.orchestra.run.smithi115.stdout: osd debug shutdown = true 2024-03-25T18:25:50.633 INFO:teuthology.orchestra.run.smithi115.stdout: osd debug op order = true 2024-03-25T18:25:50.633 INFO:teuthology.orchestra.run.smithi115.stdout: osd debug verify stray on activate = true 2024-03-25T18:25:50.633 INFO:teuthology.orchestra.run.smithi115.stdout: osd debug trim objects = true 2024-03-25T18:25:50.633 INFO:teuthology.orchestra.run.smithi115.stdout: 2024-03-25T18:25:50.633 INFO:teuthology.orchestra.run.smithi115.stdout: osd open classes on start = true 2024-03-25T18:25:50.633 INFO:teuthology.orchestra.run.smithi115.stdout: osd debug pg log writeout = true 2024-03-25T18:25:50.633 INFO:teuthology.orchestra.run.smithi115.stdout: 2024-03-25T18:25:50.633 INFO:teuthology.orchestra.run.smithi115.stdout: osd deep scrub update digest min age = 30 2024-03-25T18:25:50.633 INFO:teuthology.orchestra.run.smithi115.stdout: 2024-03-25T18:25:50.633 INFO:teuthology.orchestra.run.smithi115.stdout: osd map max advance = 10 2024-03-25T18:25:50.634 INFO:teuthology.orchestra.run.smithi115.stdout: 2024-03-25T18:25:50.634 INFO:teuthology.orchestra.run.smithi115.stdout: journal zero on create = true 2024-03-25T18:25:50.634 INFO:teuthology.orchestra.run.smithi115.stdout: 2024-03-25T18:25:50.634 INFO:teuthology.orchestra.run.smithi115.stdout: filestore ondisk finisher threads = 3 2024-03-25T18:25:50.634 INFO:teuthology.orchestra.run.smithi115.stdout: filestore apply finisher threads = 3 2024-03-25T18:25:50.634 INFO:teuthology.orchestra.run.smithi115.stdout: 2024-03-25T18:25:50.634 INFO:teuthology.orchestra.run.smithi115.stdout: bdev debug aio = true 2024-03-25T18:25:50.634 INFO:teuthology.orchestra.run.smithi115.stdout: osd debug misdirected ops = true 2024-03-25T18:25:50.634 INFO:teuthology.orchestra.run.smithi115.stdout: bdev async discard = True 2024-03-25T18:25:50.634 INFO:teuthology.orchestra.run.smithi115.stdout: bdev enable discard = True 2024-03-25T18:25:50.634 INFO:teuthology.orchestra.run.smithi115.stdout: bluestore allocator = stupid 2024-03-25T18:25:50.634 INFO:teuthology.orchestra.run.smithi115.stdout: bluestore block size = 96636764160 2024-03-25T18:25:50.634 INFO:teuthology.orchestra.run.smithi115.stdout: bluestore fsck on mount = True 2024-03-25T18:25:50.634 INFO:teuthology.orchestra.run.smithi115.stdout: bluestore zero block detection = True 2024-03-25T18:25:50.635 INFO:teuthology.orchestra.run.smithi115.stdout: debug bluefs = 1 2024-03-25T18:25:50.635 INFO:teuthology.orchestra.run.smithi115.stdout: debug bluestore = 1 2024-03-25T18:25:50.635 INFO:teuthology.orchestra.run.smithi115.stdout: debug monc = 20 2024-03-25T18:25:50.635 INFO:teuthology.orchestra.run.smithi115.stdout: debug ms = 1 2024-03-25T18:25:50.635 INFO:teuthology.orchestra.run.smithi115.stdout: debug osd = 20 2024-03-25T18:25:50.635 INFO:teuthology.orchestra.run.smithi115.stdout: debug rocksdb = 10 2024-03-25T18:25:50.635 INFO:teuthology.orchestra.run.smithi115.stdout: mon osd backfillfull_ratio = 0.85 2024-03-25T18:25:50.635 INFO:teuthology.orchestra.run.smithi115.stdout: mon osd full ratio = 0.9 2024-03-25T18:25:50.635 INFO:teuthology.orchestra.run.smithi115.stdout: mon osd nearfull ratio = 0.8 2024-03-25T18:25:50.635 INFO:teuthology.orchestra.run.smithi115.stdout: osd blocked scrub grace period = 3600 2024-03-25T18:25:50.635 INFO:teuthology.orchestra.run.smithi115.stdout: osd class default list = * 2024-03-25T18:25:50.635 INFO:teuthology.orchestra.run.smithi115.stdout: osd class load list = * 2024-03-25T18:25:50.635 INFO:teuthology.orchestra.run.smithi115.stdout: osd client watch timeout = 120 2024-03-25T18:25:50.635 INFO:teuthology.orchestra.run.smithi115.stdout: osd debug verify cached snaps = True 2024-03-25T18:25:50.635 INFO:teuthology.orchestra.run.smithi115.stdout: osd debug verify missing on start = True 2024-03-25T18:25:50.636 INFO:teuthology.orchestra.run.smithi115.stdout: osd failsafe full ratio = 0.95 2024-03-25T18:25:50.636 INFO:teuthology.orchestra.run.smithi115.stdout: osd fast shutdown = False 2024-03-25T18:25:50.636 INFO:teuthology.orchestra.run.smithi115.stdout: osd max markdown count = 1000 2024-03-25T18:25:50.636 INFO:teuthology.orchestra.run.smithi115.stdout: osd mclock override recovery settings = True 2024-03-25T18:25:50.636 INFO:teuthology.orchestra.run.smithi115.stdout: osd objectstore = bluestore 2024-03-25T18:25:50.636 INFO:teuthology.orchestra.run.smithi115.stdout: osd op queue = debug_random 2024-03-25T18:25:50.636 INFO:teuthology.orchestra.run.smithi115.stdout: osd op queue cut off = debug_random 2024-03-25T18:25:50.636 INFO:teuthology.orchestra.run.smithi115.stdout: osd shutdown pgref assert = True 2024-03-25T18:25:50.636 INFO:teuthology.orchestra.run.smithi115.stdout: 2024-03-25T18:25:50.636 INFO:teuthology.orchestra.run.smithi115.stdout:[mgr] 2024-03-25T18:25:50.636 INFO:teuthology.orchestra.run.smithi115.stdout: debug ms = 1 2024-03-25T18:25:50.636 INFO:teuthology.orchestra.run.smithi115.stdout: debug mgr = 20 2024-03-25T18:25:50.636 INFO:teuthology.orchestra.run.smithi115.stdout: debug mon = 20 2024-03-25T18:25:50.636 INFO:teuthology.orchestra.run.smithi115.stdout: debug auth = 20 2024-03-25T18:25:50.636 INFO:teuthology.orchestra.run.smithi115.stdout: mon reweight min pgs per osd = 4 2024-03-25T18:25:50.637 INFO:teuthology.orchestra.run.smithi115.stdout: mon reweight min bytes per osd = 10 2024-03-25T18:25:50.637 INFO:teuthology.orchestra.run.smithi115.stdout: mgr/telemetry/nag = false 2024-03-25T18:25:50.637 INFO:teuthology.orchestra.run.smithi115.stdout: 2024-03-25T18:25:50.637 INFO:teuthology.orchestra.run.smithi115.stdout:[mon] 2024-03-25T18:25:50.637 INFO:teuthology.orchestra.run.smithi115.stdout: debug ms = 1 2024-03-25T18:25:50.637 INFO:teuthology.orchestra.run.smithi115.stdout: debug mon = 20 2024-03-25T18:25:50.637 INFO:teuthology.orchestra.run.smithi115.stdout: debug paxos = 20 2024-03-25T18:25:50.637 INFO:teuthology.orchestra.run.smithi115.stdout: debug auth = 20 2024-03-25T18:25:50.637 INFO:teuthology.orchestra.run.smithi115.stdout: mon data avail warn = 5 2024-03-25T18:25:50.637 INFO:teuthology.orchestra.run.smithi115.stdout: mon mgr mkfs grace = 240 2024-03-25T18:25:50.637 INFO:teuthology.orchestra.run.smithi115.stdout: mon reweight min pgs per osd = 4 2024-03-25T18:25:50.637 INFO:teuthology.orchestra.run.smithi115.stdout: mon osd reporter subtree level = osd 2024-03-25T18:25:50.637 INFO:teuthology.orchestra.run.smithi115.stdout: mon osd prime pg temp = true 2024-03-25T18:25:50.637 INFO:teuthology.orchestra.run.smithi115.stdout: mon reweight min bytes per osd = 10 2024-03-25T18:25:50.637 INFO:teuthology.orchestra.run.smithi115.stdout: 2024-03-25T18:25:50.637 INFO:teuthology.orchestra.run.smithi115.stdout: # rotate auth tickets quickly to exercise renewal paths 2024-03-25T18:25:50.638 INFO:teuthology.orchestra.run.smithi115.stdout: auth mon ticket ttl = 660 # 11m 2024-03-25T18:25:50.638 INFO:teuthology.orchestra.run.smithi115.stdout: auth service ticket ttl = 240 # 4m 2024-03-25T18:25:50.638 INFO:teuthology.orchestra.run.smithi115.stdout: 2024-03-25T18:25:50.638 INFO:teuthology.orchestra.run.smithi115.stdout: # don't complain about insecure global_id in the test suite 2024-03-25T18:25:50.638 INFO:teuthology.orchestra.run.smithi115.stdout: mon_warn_on_insecure_global_id_reclaim = false 2024-03-25T18:25:50.638 INFO:teuthology.orchestra.run.smithi115.stdout: mon_warn_on_insecure_global_id_reclaim_allowed = false 2024-03-25T18:25:50.638 INFO:teuthology.orchestra.run.smithi115.stdout: 2024-03-25T18:25:50.638 INFO:teuthology.orchestra.run.smithi115.stdout: # 1m isn't quite enough 2024-03-25T18:25:50.638 INFO:teuthology.orchestra.run.smithi115.stdout: mon_down_mkfs_grace = 2m 2024-03-25T18:25:50.638 INFO:teuthology.orchestra.run.smithi115.stdout: 2024-03-25T18:25:50.638 INFO:teuthology.orchestra.run.smithi115.stdout: mon_warn_on_filestore_osds = false 2024-03-25T18:25:50.638 INFO:teuthology.orchestra.run.smithi115.stdout: mon min osdmap epochs = 50 2024-03-25T18:25:50.638 INFO:teuthology.orchestra.run.smithi115.stdout: mon osd crush smoke test = False 2024-03-25T18:25:50.639 INFO:teuthology.orchestra.run.smithi115.stdout: mon osdmap full prune interval = 2 2024-03-25T18:25:50.639 INFO:teuthology.orchestra.run.smithi115.stdout: mon osdmap full prune min = 15 2024-03-25T18:25:50.639 INFO:teuthology.orchestra.run.smithi115.stdout: mon osdmap full prune txsize = 2 2024-03-25T18:25:50.639 INFO:teuthology.orchestra.run.smithi115.stdout: mon scrub interval = 300 2024-03-25T18:25:50.639 INFO:teuthology.orchestra.run.smithi115.stdout: mon warn on pool no app = False 2024-03-25T18:25:50.639 INFO:teuthology.orchestra.run.smithi115.stdout: paxos service trim min = 10 2024-03-25T18:25:50.639 INFO:teuthology.orchestra.run.smithi115.stdout: 2024-03-25T18:25:50.639 INFO:teuthology.orchestra.run.smithi115.stdout:[client] 2024-03-25T18:25:50.639 INFO:teuthology.orchestra.run.smithi115.stdout: rgw cache enabled = true 2024-03-25T18:25:50.639 INFO:teuthology.orchestra.run.smithi115.stdout: rgw enable ops log = true 2024-03-25T18:25:50.639 INFO:teuthology.orchestra.run.smithi115.stdout: rgw enable usage log = true 2024-03-25T18:25:50.639 INFO:teuthology.orchestra.run.smithi115.stdout: log file = /var/log/ceph/$cluster-$name.$pid.log 2024-03-25T18:25:50.639 INFO:teuthology.orchestra.run.smithi115.stdout: admin socket = /var/run/ceph/$cluster-$name.$pid.asok 2024-03-25T18:25:50.639 INFO:teuthology.orchestra.run.smithi115.stdout: debug monc = 20 2024-03-25T18:25:50.640 INFO:teuthology.orchestra.run.smithi115.stdout: debug ms = 1 2024-03-25T18:25:50.640 INFO:teuthology.orchestra.run.smithi115.stdout: debug objecter = 20 2024-03-25T18:25:50.640 INFO:teuthology.orchestra.run.smithi115.stdout: debug rados = 20 2024-03-25T18:25:50.640 INFO:teuthology.orchestra.run.smithi115.stdout:[mon.a] 2024-03-25T18:25:50.640 INFO:teuthology.orchestra.run.smithi115.stdout:[mon.c] 2024-03-25T18:25:50.640 INFO:teuthology.orchestra.run.smithi115.stdout:[mon.b] 2024-03-25T18:25:50.641 INFO:tasks.ceph:Creating admin key on mon.a... 2024-03-25T18:25:50.641 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --gen-key --name=client.admin --cap mon 'allow *' --cap osd 'allow *' --cap mds 'allow *' --cap mgr 'allow *' /etc/ceph/ceph.keyring 2024-03-25T18:25:50.696 INFO:tasks.ceph:Copying monmap to all nodes... 2024-03-25T18:25:50.696 DEBUG:teuthology.orchestra.run.smithi001:> set -ex 2024-03-25T18:25:50.696 DEBUG:teuthology.orchestra.run.smithi001:> dd if=/etc/ceph/ceph.keyring of=/dev/stdout 2024-03-25T18:25:50.710 DEBUG:teuthology.orchestra.run.smithi001:> set -ex 2024-03-25T18:25:50.710 DEBUG:teuthology.orchestra.run.smithi001:> dd if=/home/ubuntu/cephtest/ceph.monmap of=/dev/stdout 2024-03-25T18:25:50.764 INFO:tasks.ceph:Sending monmap to node ubuntu@smithi001.front.sepia.ceph.com 2024-03-25T18:25:50.764 DEBUG:teuthology.orchestra.run.smithi001:> set -ex 2024-03-25T18:25:50.764 DEBUG:teuthology.orchestra.run.smithi001:> sudo dd of=/etc/ceph/ceph.keyring 2024-03-25T18:25:50.764 DEBUG:teuthology.orchestra.run.smithi001:> sudo chmod 0644 /etc/ceph/ceph.keyring 2024-03-25T18:25:50.843 DEBUG:teuthology.orchestra.run.smithi001:> set -ex 2024-03-25T18:25:50.843 DEBUG:teuthology.orchestra.run.smithi001:> dd of=/home/ubuntu/cephtest/ceph.monmap 2024-03-25T18:25:50.898 INFO:tasks.ceph:Sending monmap to node ubuntu@smithi115.front.sepia.ceph.com 2024-03-25T18:25:50.898 DEBUG:teuthology.orchestra.run.smithi115:> set -ex 2024-03-25T18:25:50.898 DEBUG:teuthology.orchestra.run.smithi115:> sudo dd of=/etc/ceph/ceph.keyring 2024-03-25T18:25:50.898 DEBUG:teuthology.orchestra.run.smithi115:> sudo chmod 0644 /etc/ceph/ceph.keyring 2024-03-25T18:25:50.937 DEBUG:teuthology.orchestra.run.smithi115:> set -ex 2024-03-25T18:25:50.937 DEBUG:teuthology.orchestra.run.smithi115:> dd of=/home/ubuntu/cephtest/ceph.monmap 2024-03-25T18:25:50.992 INFO:tasks.ceph:Setting up mon nodes... 2024-03-25T18:25:50.992 INFO:tasks.ceph:Setting up mgr nodes... 2024-03-25T18:25:50.992 DEBUG:teuthology.orchestra.run.smithi001:> sudo mkdir -p /var/lib/ceph/mgr/ceph-y && sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=mgr.y /var/lib/ceph/mgr/ceph-y/keyring 2024-03-25T18:25:51.046 INFO:teuthology.orchestra.run.smithi001.stdout:creating /var/lib/ceph/mgr/ceph-y/keyring 2024-03-25T18:25:51.049 DEBUG:teuthology.orchestra.run.smithi115:> sudo mkdir -p /var/lib/ceph/mgr/ceph-x && sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=mgr.x /var/lib/ceph/mgr/ceph-x/keyring 2024-03-25T18:25:51.115 INFO:teuthology.orchestra.run.smithi115.stdout:creating /var/lib/ceph/mgr/ceph-x/keyring 2024-03-25T18:25:51.117 INFO:tasks.ceph:Setting up mds nodes... 2024-03-25T18:25:51.117 INFO:tasks.ceph_client:Setting up client nodes... 2024-03-25T18:25:51.118 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=client.0 /etc/ceph/ceph.client.0.keyring && sudo chmod 0644 /etc/ceph/ceph.client.0.keyring 2024-03-25T18:25:51.159 INFO:teuthology.orchestra.run.smithi001.stdout:creating /etc/ceph/ceph.client.0.keyring 2024-03-25T18:25:51.174 DEBUG:teuthology.orchestra.run.smithi115:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=client.1 /etc/ceph/ceph.client.1.keyring && sudo chmod 0644 /etc/ceph/ceph.client.1.keyring 2024-03-25T18:25:51.216 INFO:teuthology.orchestra.run.smithi115.stdout:creating /etc/ceph/ceph.client.1.keyring 2024-03-25T18:25:51.231 INFO:tasks.ceph:Running mkfs on osd nodes... 2024-03-25T18:25:51.232 INFO:tasks.ceph:ctx.disk_config.remote_to_roles_to_dev: {Remote(name='ubuntu@smithi001.front.sepia.ceph.com'): {'osd.0': '/dev/vg_nvme/lv_1', 'osd.1': '/dev/vg_nvme/lv_2', 'osd.2': '/dev/vg_nvme/lv_3', 'osd.3': '/dev/vg_nvme/lv_4'}, Remote(name='ubuntu@smithi115.front.sepia.ceph.com'): {'osd.4': '/dev/vg_nvme/lv_1', 'osd.5': '/dev/vg_nvme/lv_2', 'osd.6': '/dev/vg_nvme/lv_3', 'osd.7': '/dev/vg_nvme/lv_4'}} 2024-03-25T18:25:51.232 DEBUG:teuthology.orchestra.run.smithi001:> sudo mkdir -p /var/lib/ceph/osd/ceph-0 2024-03-25T18:25:51.258 INFO:tasks.ceph:roles_to_devs: {'osd.0': '/dev/vg_nvme/lv_1', 'osd.1': '/dev/vg_nvme/lv_2', 'osd.2': '/dev/vg_nvme/lv_3', 'osd.3': '/dev/vg_nvme/lv_4'} 2024-03-25T18:25:51.258 INFO:tasks.ceph:role: osd.0 2024-03-25T18:25:51.258 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_1 on ubuntu@smithi001.front.sepia.ceph.com 2024-03-25T18:25:51.259 DEBUG:teuthology.orchestra.run.smithi001:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_1 2024-03-25T18:25:51.324 INFO:teuthology.orchestra.run.smithi001.stdout:meta-data=/dev/vg_nvme/lv_1 isize=2048 agcount=4, agsize=5859072 blks 2024-03-25T18:25:51.324 INFO:teuthology.orchestra.run.smithi001.stdout: = sectsz=512 attr=2, projid32bit=1 2024-03-25T18:25:51.324 INFO:teuthology.orchestra.run.smithi001.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2024-03-25T18:25:51.324 INFO:teuthology.orchestra.run.smithi001.stdout: = reflink=1 bigtime=1 inobtcount=1 nrext64=0 2024-03-25T18:25:51.324 INFO:teuthology.orchestra.run.smithi001.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2024-03-25T18:25:51.324 INFO:teuthology.orchestra.run.smithi001.stdout: = sunit=0 swidth=0 blks 2024-03-25T18:25:51.324 INFO:teuthology.orchestra.run.smithi001.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2024-03-25T18:25:51.325 INFO:teuthology.orchestra.run.smithi001.stdout:log =internal log bsize=4096 blocks=16384, version=2 2024-03-25T18:25:51.325 INFO:teuthology.orchestra.run.smithi001.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2024-03-25T18:25:51.325 INFO:teuthology.orchestra.run.smithi001.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2024-03-25T18:25:51.356 INFO:teuthology.orchestra.run.smithi001.stdout:Discarding blocks...Done. 2024-03-25T18:25:51.358 INFO:tasks.ceph:mount /dev/vg_nvme/lv_1 on ubuntu@smithi001.front.sepia.ceph.com -o noatime 2024-03-25T18:25:51.358 DEBUG:teuthology.orchestra.run.smithi001:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_1 /var/lib/ceph/osd/ceph-0 2024-03-25T18:25:51.429 DEBUG:teuthology.orchestra.run.smithi001:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-0 2024-03-25T18:25:51.503 DEBUG:teuthology.orchestra.run.smithi001:> sudo mkdir -p /var/lib/ceph/osd/ceph-1 2024-03-25T18:25:51.570 INFO:tasks.ceph:roles_to_devs: {'osd.0': '/dev/vg_nvme/lv_1', 'osd.1': '/dev/vg_nvme/lv_2', 'osd.2': '/dev/vg_nvme/lv_3', 'osd.3': '/dev/vg_nvme/lv_4'} 2024-03-25T18:25:51.570 INFO:tasks.ceph:role: osd.1 2024-03-25T18:25:51.570 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_2 on ubuntu@smithi001.front.sepia.ceph.com 2024-03-25T18:25:51.570 DEBUG:teuthology.orchestra.run.smithi001:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_2 2024-03-25T18:25:51.636 INFO:teuthology.orchestra.run.smithi001.stdout:meta-data=/dev/vg_nvme/lv_2 isize=2048 agcount=4, agsize=5859072 blks 2024-03-25T18:25:51.636 INFO:teuthology.orchestra.run.smithi001.stdout: = sectsz=512 attr=2, projid32bit=1 2024-03-25T18:25:51.636 INFO:teuthology.orchestra.run.smithi001.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2024-03-25T18:25:51.636 INFO:teuthology.orchestra.run.smithi001.stdout: = reflink=1 bigtime=1 inobtcount=1 nrext64=0 2024-03-25T18:25:51.636 INFO:teuthology.orchestra.run.smithi001.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2024-03-25T18:25:51.636 INFO:teuthology.orchestra.run.smithi001.stdout: = sunit=0 swidth=0 blks 2024-03-25T18:25:51.636 INFO:teuthology.orchestra.run.smithi001.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2024-03-25T18:25:51.636 INFO:teuthology.orchestra.run.smithi001.stdout:log =internal log bsize=4096 blocks=16384, version=2 2024-03-25T18:25:51.636 INFO:teuthology.orchestra.run.smithi001.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2024-03-25T18:25:51.637 INFO:teuthology.orchestra.run.smithi001.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2024-03-25T18:25:51.668 INFO:teuthology.orchestra.run.smithi001.stdout:Discarding blocks...Done. 2024-03-25T18:25:51.670 INFO:tasks.ceph:mount /dev/vg_nvme/lv_2 on ubuntu@smithi001.front.sepia.ceph.com -o noatime 2024-03-25T18:25:51.670 DEBUG:teuthology.orchestra.run.smithi001:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_2 /var/lib/ceph/osd/ceph-1 2024-03-25T18:25:51.751 DEBUG:teuthology.orchestra.run.smithi001:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-1 2024-03-25T18:25:51.821 DEBUG:teuthology.orchestra.run.smithi001:> sudo mkdir -p /var/lib/ceph/osd/ceph-2 2024-03-25T18:25:51.887 INFO:tasks.ceph:roles_to_devs: {'osd.0': '/dev/vg_nvme/lv_1', 'osd.1': '/dev/vg_nvme/lv_2', 'osd.2': '/dev/vg_nvme/lv_3', 'osd.3': '/dev/vg_nvme/lv_4'} 2024-03-25T18:25:51.888 INFO:tasks.ceph:role: osd.2 2024-03-25T18:25:51.888 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_3 on ubuntu@smithi001.front.sepia.ceph.com 2024-03-25T18:25:51.888 DEBUG:teuthology.orchestra.run.smithi001:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_3 2024-03-25T18:25:51.953 INFO:teuthology.orchestra.run.smithi001.stdout:meta-data=/dev/vg_nvme/lv_3 isize=2048 agcount=4, agsize=5859072 blks 2024-03-25T18:25:51.953 INFO:teuthology.orchestra.run.smithi001.stdout: = sectsz=512 attr=2, projid32bit=1 2024-03-25T18:25:51.953 INFO:teuthology.orchestra.run.smithi001.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2024-03-25T18:25:51.953 INFO:teuthology.orchestra.run.smithi001.stdout: = reflink=1 bigtime=1 inobtcount=1 nrext64=0 2024-03-25T18:25:51.953 INFO:teuthology.orchestra.run.smithi001.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2024-03-25T18:25:51.954 INFO:teuthology.orchestra.run.smithi001.stdout: = sunit=0 swidth=0 blks 2024-03-25T18:25:51.954 INFO:teuthology.orchestra.run.smithi001.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2024-03-25T18:25:51.954 INFO:teuthology.orchestra.run.smithi001.stdout:log =internal log bsize=4096 blocks=16384, version=2 2024-03-25T18:25:51.954 INFO:teuthology.orchestra.run.smithi001.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2024-03-25T18:25:51.954 INFO:teuthology.orchestra.run.smithi001.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2024-03-25T18:25:51.986 INFO:teuthology.orchestra.run.smithi001.stdout:Discarding blocks...Done. 2024-03-25T18:25:51.987 INFO:tasks.ceph:mount /dev/vg_nvme/lv_3 on ubuntu@smithi001.front.sepia.ceph.com -o noatime 2024-03-25T18:25:51.988 DEBUG:teuthology.orchestra.run.smithi001:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_3 /var/lib/ceph/osd/ceph-2 2024-03-25T18:25:52.058 DEBUG:teuthology.orchestra.run.smithi001:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-2 2024-03-25T18:25:52.131 DEBUG:teuthology.orchestra.run.smithi001:> sudo mkdir -p /var/lib/ceph/osd/ceph-3 2024-03-25T18:25:52.198 INFO:tasks.ceph:roles_to_devs: {'osd.0': '/dev/vg_nvme/lv_1', 'osd.1': '/dev/vg_nvme/lv_2', 'osd.2': '/dev/vg_nvme/lv_3', 'osd.3': '/dev/vg_nvme/lv_4'} 2024-03-25T18:25:52.198 INFO:tasks.ceph:role: osd.3 2024-03-25T18:25:52.198 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_4 on ubuntu@smithi001.front.sepia.ceph.com 2024-03-25T18:25:52.198 DEBUG:teuthology.orchestra.run.smithi001:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_4 2024-03-25T18:25:52.264 INFO:teuthology.orchestra.run.smithi001.stdout:meta-data=/dev/vg_nvme/lv_4 isize=2048 agcount=4, agsize=5859072 blks 2024-03-25T18:25:52.265 INFO:teuthology.orchestra.run.smithi001.stdout: = sectsz=512 attr=2, projid32bit=1 2024-03-25T18:25:52.265 INFO:teuthology.orchestra.run.smithi001.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2024-03-25T18:25:52.265 INFO:teuthology.orchestra.run.smithi001.stdout: = reflink=1 bigtime=1 inobtcount=1 nrext64=0 2024-03-25T18:25:52.265 INFO:teuthology.orchestra.run.smithi001.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2024-03-25T18:25:52.265 INFO:teuthology.orchestra.run.smithi001.stdout: = sunit=0 swidth=0 blks 2024-03-25T18:25:52.265 INFO:teuthology.orchestra.run.smithi001.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2024-03-25T18:25:52.265 INFO:teuthology.orchestra.run.smithi001.stdout:log =internal log bsize=4096 blocks=16384, version=2 2024-03-25T18:25:52.265 INFO:teuthology.orchestra.run.smithi001.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2024-03-25T18:25:52.265 INFO:teuthology.orchestra.run.smithi001.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2024-03-25T18:25:52.297 INFO:teuthology.orchestra.run.smithi001.stdout:Discarding blocks...Done. 2024-03-25T18:25:52.298 INFO:tasks.ceph:mount /dev/vg_nvme/lv_4 on ubuntu@smithi001.front.sepia.ceph.com -o noatime 2024-03-25T18:25:52.298 DEBUG:teuthology.orchestra.run.smithi001:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_4 /var/lib/ceph/osd/ceph-3 2024-03-25T18:25:52.368 DEBUG:teuthology.orchestra.run.smithi001:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-3 2024-03-25T18:25:52.439 DEBUG:teuthology.orchestra.run.smithi001:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 0 --monmap /home/ubuntu/cephtest/ceph.monmap 2024-03-25T18:25:52.532 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-25T18:25:52.531+0000 7f80a98c4640 -1 auth: error reading file: /var/lib/ceph/osd/ceph-0/keyring: can't open /var/lib/ceph/osd/ceph-0/keyring: (2) No such file or directory 2024-03-25T18:25:52.532 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-25T18:25:52.531+0000 7f80a98c4640 -1 created new key in keyring /var/lib/ceph/osd/ceph-0/keyring 2024-03-25T18:25:52.532 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-25T18:25:52.531+0000 7f80a98c4640 -1 bluestore(/var/lib/ceph/osd/ceph-0/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-0/block: (2) No such file or directory 2024-03-25T18:25:52.532 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-25T18:25:52.531+0000 7f80a98c4640 -1 bluestore(/var/lib/ceph/osd/ceph-0/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-0/block: (2) No such file or directory 2024-03-25T18:25:52.532 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-25T18:25:52.531+0000 7f80a98c4640 -1 bluestore(/var/lib/ceph/osd/ceph-0/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-0/block: (2) No such file or directory 2024-03-25T18:25:52.532 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-25T18:25:52.531+0000 7f80a98c4640 -1 bluestore(/var/lib/ceph/osd/ceph-0) _read_fsid unparsable uuid 2024-03-25T18:25:55.375 DEBUG:teuthology.orchestra.run.smithi001:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-0 2024-03-25T18:25:55.406 DEBUG:teuthology.orchestra.run.smithi001:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 1 --monmap /home/ubuntu/cephtest/ceph.monmap 2024-03-25T18:25:55.497 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-25T18:25:55.496+0000 7f2861c13640 -1 auth: error reading file: /var/lib/ceph/osd/ceph-1/keyring: can't open /var/lib/ceph/osd/ceph-1/keyring: (2) No such file or directory 2024-03-25T18:25:55.497 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-25T18:25:55.496+0000 7f2861c13640 -1 created new key in keyring /var/lib/ceph/osd/ceph-1/keyring 2024-03-25T18:25:55.497 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-25T18:25:55.496+0000 7f2861c13640 -1 bluestore(/var/lib/ceph/osd/ceph-1/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-1/block: (2) No such file or directory 2024-03-25T18:25:55.497 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-25T18:25:55.496+0000 7f2861c13640 -1 bluestore(/var/lib/ceph/osd/ceph-1/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-1/block: (2) No such file or directory 2024-03-25T18:25:55.497 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-25T18:25:55.496+0000 7f2861c13640 -1 bluestore(/var/lib/ceph/osd/ceph-1/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-1/block: (2) No such file or directory 2024-03-25T18:25:55.498 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-25T18:25:55.497+0000 7f2861c13640 -1 bluestore(/var/lib/ceph/osd/ceph-1) _read_fsid unparsable uuid 2024-03-25T18:25:58.344 DEBUG:teuthology.orchestra.run.smithi001:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-1 2024-03-25T18:25:58.375 DEBUG:teuthology.orchestra.run.smithi001:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 2 --monmap /home/ubuntu/cephtest/ceph.monmap 2024-03-25T18:25:58.469 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-25T18:25:58.468+0000 7fbe398d5640 -1 auth: error reading file: /var/lib/ceph/osd/ceph-2/keyring: can't open /var/lib/ceph/osd/ceph-2/keyring: (2) No such file or directory 2024-03-25T18:25:58.469 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-25T18:25:58.468+0000 7fbe398d5640 -1 created new key in keyring /var/lib/ceph/osd/ceph-2/keyring 2024-03-25T18:25:58.469 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-25T18:25:58.468+0000 7fbe398d5640 -1 bluestore(/var/lib/ceph/osd/ceph-2/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-2/block: (2) No such file or directory 2024-03-25T18:25:58.469 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-25T18:25:58.468+0000 7fbe398d5640 -1 bluestore(/var/lib/ceph/osd/ceph-2/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-2/block: (2) No such file or directory 2024-03-25T18:25:58.469 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-25T18:25:58.468+0000 7fbe398d5640 -1 bluestore(/var/lib/ceph/osd/ceph-2/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-2/block: (2) No such file or directory 2024-03-25T18:25:58.469 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-25T18:25:58.468+0000 7fbe398d5640 -1 bluestore(/var/lib/ceph/osd/ceph-2) _read_fsid unparsable uuid 2024-03-25T18:26:01.307 DEBUG:teuthology.orchestra.run.smithi001:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-2 2024-03-25T18:26:01.338 DEBUG:teuthology.orchestra.run.smithi001:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 3 --monmap /home/ubuntu/cephtest/ceph.monmap 2024-03-25T18:26:01.428 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-25T18:26:01.427+0000 7f13c278f640 -1 auth: error reading file: /var/lib/ceph/osd/ceph-3/keyring: can't open /var/lib/ceph/osd/ceph-3/keyring: (2) No such file or directory 2024-03-25T18:26:01.429 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-25T18:26:01.428+0000 7f13c278f640 -1 created new key in keyring /var/lib/ceph/osd/ceph-3/keyring 2024-03-25T18:26:01.429 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-25T18:26:01.428+0000 7f13c278f640 -1 bluestore(/var/lib/ceph/osd/ceph-3/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-3/block: (2) No such file or directory 2024-03-25T18:26:01.429 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-25T18:26:01.428+0000 7f13c278f640 -1 bluestore(/var/lib/ceph/osd/ceph-3/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-3/block: (2) No such file or directory 2024-03-25T18:26:01.429 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-25T18:26:01.428+0000 7f13c278f640 -1 bluestore(/var/lib/ceph/osd/ceph-3/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-3/block: (2) No such file or directory 2024-03-25T18:26:01.429 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-25T18:26:01.428+0000 7f13c278f640 -1 bluestore(/var/lib/ceph/osd/ceph-3) _read_fsid unparsable uuid 2024-03-25T18:26:04.268 DEBUG:teuthology.orchestra.run.smithi001:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-3 2024-03-25T18:26:04.299 DEBUG:teuthology.orchestra.run.smithi115:> sudo mkdir -p /var/lib/ceph/osd/ceph-4 2024-03-25T18:26:04.330 INFO:tasks.ceph:roles_to_devs: {'osd.4': '/dev/vg_nvme/lv_1', 'osd.5': '/dev/vg_nvme/lv_2', 'osd.6': '/dev/vg_nvme/lv_3', 'osd.7': '/dev/vg_nvme/lv_4'} 2024-03-25T18:26:04.330 INFO:tasks.ceph:role: osd.4 2024-03-25T18:26:04.330 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_1 on ubuntu@smithi115.front.sepia.ceph.com 2024-03-25T18:26:04.331 DEBUG:teuthology.orchestra.run.smithi115:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_1 2024-03-25T18:26:04.409 INFO:teuthology.orchestra.run.smithi115.stdout:meta-data=/dev/vg_nvme/lv_1 isize=2048 agcount=4, agsize=5859072 blks 2024-03-25T18:26:04.409 INFO:teuthology.orchestra.run.smithi115.stdout: = sectsz=512 attr=2, projid32bit=1 2024-03-25T18:26:04.409 INFO:teuthology.orchestra.run.smithi115.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2024-03-25T18:26:04.409 INFO:teuthology.orchestra.run.smithi115.stdout: = reflink=1 bigtime=1 inobtcount=1 nrext64=0 2024-03-25T18:26:04.409 INFO:teuthology.orchestra.run.smithi115.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2024-03-25T18:26:04.409 INFO:teuthology.orchestra.run.smithi115.stdout: = sunit=0 swidth=0 blks 2024-03-25T18:26:04.410 INFO:teuthology.orchestra.run.smithi115.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2024-03-25T18:26:04.410 INFO:teuthology.orchestra.run.smithi115.stdout:log =internal log bsize=4096 blocks=16384, version=2 2024-03-25T18:26:04.410 INFO:teuthology.orchestra.run.smithi115.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2024-03-25T18:26:04.410 INFO:teuthology.orchestra.run.smithi115.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2024-03-25T18:26:04.929 INFO:teuthology.orchestra.run.smithi115.stdout:Discarding blocks...Done. 2024-03-25T18:26:04.931 INFO:tasks.ceph:mount /dev/vg_nvme/lv_1 on ubuntu@smithi115.front.sepia.ceph.com -o noatime 2024-03-25T18:26:04.931 DEBUG:teuthology.orchestra.run.smithi115:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_1 /var/lib/ceph/osd/ceph-4 2024-03-25T18:26:05.081 DEBUG:teuthology.orchestra.run.smithi115:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-4 2024-03-25T18:26:05.117 DEBUG:teuthology.orchestra.run.smithi115:> sudo mkdir -p /var/lib/ceph/osd/ceph-5 2024-03-25T18:26:05.187 INFO:tasks.ceph:roles_to_devs: {'osd.4': '/dev/vg_nvme/lv_1', 'osd.5': '/dev/vg_nvme/lv_2', 'osd.6': '/dev/vg_nvme/lv_3', 'osd.7': '/dev/vg_nvme/lv_4'} 2024-03-25T18:26:05.188 INFO:tasks.ceph:role: osd.5 2024-03-25T18:26:05.188 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_2 on ubuntu@smithi115.front.sepia.ceph.com 2024-03-25T18:26:05.188 DEBUG:teuthology.orchestra.run.smithi115:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_2 2024-03-25T18:26:05.263 INFO:teuthology.orchestra.run.smithi115.stdout:meta-data=/dev/vg_nvme/lv_2 isize=2048 agcount=4, agsize=5859072 blks 2024-03-25T18:26:05.263 INFO:teuthology.orchestra.run.smithi115.stdout: = sectsz=512 attr=2, projid32bit=1 2024-03-25T18:26:05.263 INFO:teuthology.orchestra.run.smithi115.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2024-03-25T18:26:05.263 INFO:teuthology.orchestra.run.smithi115.stdout: = reflink=1 bigtime=1 inobtcount=1 nrext64=0 2024-03-25T18:26:05.263 INFO:teuthology.orchestra.run.smithi115.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2024-03-25T18:26:05.264 INFO:teuthology.orchestra.run.smithi115.stdout: = sunit=0 swidth=0 blks 2024-03-25T18:26:05.264 INFO:teuthology.orchestra.run.smithi115.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2024-03-25T18:26:05.264 INFO:teuthology.orchestra.run.smithi115.stdout:log =internal log bsize=4096 blocks=16384, version=2 2024-03-25T18:26:05.264 INFO:teuthology.orchestra.run.smithi115.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2024-03-25T18:26:05.264 INFO:teuthology.orchestra.run.smithi115.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2024-03-25T18:26:05.707 INFO:teuthology.orchestra.run.smithi115.stdout:Discarding blocks...Done. 2024-03-25T18:26:05.709 INFO:tasks.ceph:mount /dev/vg_nvme/lv_2 on ubuntu@smithi115.front.sepia.ceph.com -o noatime 2024-03-25T18:26:05.709 DEBUG:teuthology.orchestra.run.smithi115:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_2 /var/lib/ceph/osd/ceph-5 2024-03-25T18:26:05.806 DEBUG:teuthology.orchestra.run.smithi115:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-5 2024-03-25T18:26:05.839 DEBUG:teuthology.orchestra.run.smithi115:> sudo mkdir -p /var/lib/ceph/osd/ceph-6 2024-03-25T18:26:05.906 INFO:tasks.ceph:roles_to_devs: {'osd.4': '/dev/vg_nvme/lv_1', 'osd.5': '/dev/vg_nvme/lv_2', 'osd.6': '/dev/vg_nvme/lv_3', 'osd.7': '/dev/vg_nvme/lv_4'} 2024-03-25T18:26:05.906 INFO:tasks.ceph:role: osd.6 2024-03-25T18:26:05.907 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_3 on ubuntu@smithi115.front.sepia.ceph.com 2024-03-25T18:26:05.907 DEBUG:teuthology.orchestra.run.smithi115:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_3 2024-03-25T18:26:05.980 INFO:teuthology.orchestra.run.smithi115.stdout:meta-data=/dev/vg_nvme/lv_3 isize=2048 agcount=4, agsize=5859072 blks 2024-03-25T18:26:05.981 INFO:teuthology.orchestra.run.smithi115.stdout: = sectsz=512 attr=2, projid32bit=1 2024-03-25T18:26:05.981 INFO:teuthology.orchestra.run.smithi115.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2024-03-25T18:26:05.981 INFO:teuthology.orchestra.run.smithi115.stdout: = reflink=1 bigtime=1 inobtcount=1 nrext64=0 2024-03-25T18:26:05.981 INFO:teuthology.orchestra.run.smithi115.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2024-03-25T18:26:05.981 INFO:teuthology.orchestra.run.smithi115.stdout: = sunit=0 swidth=0 blks 2024-03-25T18:26:05.981 INFO:teuthology.orchestra.run.smithi115.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2024-03-25T18:26:05.981 INFO:teuthology.orchestra.run.smithi115.stdout:log =internal log bsize=4096 blocks=16384, version=2 2024-03-25T18:26:05.981 INFO:teuthology.orchestra.run.smithi115.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2024-03-25T18:26:05.981 INFO:teuthology.orchestra.run.smithi115.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2024-03-25T18:26:06.471 INFO:teuthology.orchestra.run.smithi115.stdout:Discarding blocks...Done. 2024-03-25T18:26:06.473 INFO:tasks.ceph:mount /dev/vg_nvme/lv_3 on ubuntu@smithi115.front.sepia.ceph.com -o noatime 2024-03-25T18:26:06.473 DEBUG:teuthology.orchestra.run.smithi115:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_3 /var/lib/ceph/osd/ceph-6 2024-03-25T18:26:06.624 DEBUG:teuthology.orchestra.run.smithi115:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-6 2024-03-25T18:26:06.658 DEBUG:teuthology.orchestra.run.smithi115:> sudo mkdir -p /var/lib/ceph/osd/ceph-7 2024-03-25T18:26:06.725 INFO:tasks.ceph:roles_to_devs: {'osd.4': '/dev/vg_nvme/lv_1', 'osd.5': '/dev/vg_nvme/lv_2', 'osd.6': '/dev/vg_nvme/lv_3', 'osd.7': '/dev/vg_nvme/lv_4'} 2024-03-25T18:26:06.725 INFO:tasks.ceph:role: osd.7 2024-03-25T18:26:06.725 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_4 on ubuntu@smithi115.front.sepia.ceph.com 2024-03-25T18:26:06.725 DEBUG:teuthology.orchestra.run.smithi115:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_4 2024-03-25T18:26:06.799 INFO:teuthology.orchestra.run.smithi115.stdout:meta-data=/dev/vg_nvme/lv_4 isize=2048 agcount=4, agsize=5859072 blks 2024-03-25T18:26:06.799 INFO:teuthology.orchestra.run.smithi115.stdout: = sectsz=512 attr=2, projid32bit=1 2024-03-25T18:26:06.799 INFO:teuthology.orchestra.run.smithi115.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2024-03-25T18:26:06.799 INFO:teuthology.orchestra.run.smithi115.stdout: = reflink=1 bigtime=1 inobtcount=1 nrext64=0 2024-03-25T18:26:06.799 INFO:teuthology.orchestra.run.smithi115.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2024-03-25T18:26:06.799 INFO:teuthology.orchestra.run.smithi115.stdout: = sunit=0 swidth=0 blks 2024-03-25T18:26:06.800 INFO:teuthology.orchestra.run.smithi115.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2024-03-25T18:26:06.800 INFO:teuthology.orchestra.run.smithi115.stdout:log =internal log bsize=4096 blocks=16384, version=2 2024-03-25T18:26:06.800 INFO:teuthology.orchestra.run.smithi115.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2024-03-25T18:26:06.800 INFO:teuthology.orchestra.run.smithi115.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2024-03-25T18:26:07.303 INFO:teuthology.orchestra.run.smithi115.stdout:Discarding blocks...Done. 2024-03-25T18:26:07.305 INFO:tasks.ceph:mount /dev/vg_nvme/lv_4 on ubuntu@smithi115.front.sepia.ceph.com -o noatime 2024-03-25T18:26:07.305 DEBUG:teuthology.orchestra.run.smithi115:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_4 /var/lib/ceph/osd/ceph-7 2024-03-25T18:26:07.451 DEBUG:teuthology.orchestra.run.smithi115:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-7 2024-03-25T18:26:07.487 DEBUG:teuthology.orchestra.run.smithi115:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 4 --monmap /home/ubuntu/cephtest/ceph.monmap 2024-03-25T18:26:07.588 INFO:teuthology.orchestra.run.smithi115.stderr:2024-03-25T18:26:07.587+0000 7f3807524640 -1 auth: error reading file: /var/lib/ceph/osd/ceph-4/keyring: can't open /var/lib/ceph/osd/ceph-4/keyring: (2) No such file or directory 2024-03-25T18:26:07.588 INFO:teuthology.orchestra.run.smithi115.stderr:2024-03-25T18:26:07.587+0000 7f3807524640 -1 created new key in keyring /var/lib/ceph/osd/ceph-4/keyring 2024-03-25T18:26:07.589 INFO:teuthology.orchestra.run.smithi115.stderr:2024-03-25T18:26:07.587+0000 7f3807524640 -1 bluestore(/var/lib/ceph/osd/ceph-4/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-4/block: (2) No such file or directory 2024-03-25T18:26:07.589 INFO:teuthology.orchestra.run.smithi115.stderr:2024-03-25T18:26:07.587+0000 7f3807524640 -1 bluestore(/var/lib/ceph/osd/ceph-4/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-4/block: (2) No such file or directory 2024-03-25T18:26:07.589 INFO:teuthology.orchestra.run.smithi115.stderr:2024-03-25T18:26:07.587+0000 7f3807524640 -1 bluestore(/var/lib/ceph/osd/ceph-4/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-4/block: (2) No such file or directory 2024-03-25T18:26:07.589 INFO:teuthology.orchestra.run.smithi115.stderr:2024-03-25T18:26:07.587+0000 7f3807524640 -1 bluestore(/var/lib/ceph/osd/ceph-4) _read_fsid unparsable uuid 2024-03-25T18:26:10.435 DEBUG:teuthology.orchestra.run.smithi115:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-4 2024-03-25T18:26:10.467 DEBUG:teuthology.orchestra.run.smithi115:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 5 --monmap /home/ubuntu/cephtest/ceph.monmap 2024-03-25T18:26:10.557 INFO:teuthology.orchestra.run.smithi115.stderr:2024-03-25T18:26:10.556+0000 7faf4c624640 -1 auth: error reading file: /var/lib/ceph/osd/ceph-5/keyring: can't open /var/lib/ceph/osd/ceph-5/keyring: (2) No such file or directory 2024-03-25T18:26:10.557 INFO:teuthology.orchestra.run.smithi115.stderr:2024-03-25T18:26:10.556+0000 7faf4c624640 -1 created new key in keyring /var/lib/ceph/osd/ceph-5/keyring 2024-03-25T18:26:10.557 INFO:teuthology.orchestra.run.smithi115.stderr:2024-03-25T18:26:10.556+0000 7faf4c624640 -1 bluestore(/var/lib/ceph/osd/ceph-5/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-5/block: (2) No such file or directory 2024-03-25T18:26:10.557 INFO:teuthology.orchestra.run.smithi115.stderr:2024-03-25T18:26:10.556+0000 7faf4c624640 -1 bluestore(/var/lib/ceph/osd/ceph-5/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-5/block: (2) No such file or directory 2024-03-25T18:26:10.557 INFO:teuthology.orchestra.run.smithi115.stderr:2024-03-25T18:26:10.556+0000 7faf4c624640 -1 bluestore(/var/lib/ceph/osd/ceph-5/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-5/block: (2) No such file or directory 2024-03-25T18:26:10.557 INFO:teuthology.orchestra.run.smithi115.stderr:2024-03-25T18:26:10.557+0000 7faf4c624640 -1 bluestore(/var/lib/ceph/osd/ceph-5) _read_fsid unparsable uuid 2024-03-25T18:26:13.395 DEBUG:teuthology.orchestra.run.smithi115:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-5 2024-03-25T18:26:13.426 DEBUG:teuthology.orchestra.run.smithi115:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 6 --monmap /home/ubuntu/cephtest/ceph.monmap 2024-03-25T18:26:13.518 INFO:teuthology.orchestra.run.smithi115.stderr:2024-03-25T18:26:13.516+0000 7f716ac11640 -1 auth: error reading file: /var/lib/ceph/osd/ceph-6/keyring: can't open /var/lib/ceph/osd/ceph-6/keyring: (2) No such file or directory 2024-03-25T18:26:13.518 INFO:teuthology.orchestra.run.smithi115.stderr:2024-03-25T18:26:13.516+0000 7f716ac11640 -1 created new key in keyring /var/lib/ceph/osd/ceph-6/keyring 2024-03-25T18:26:13.518 INFO:teuthology.orchestra.run.smithi115.stderr:2024-03-25T18:26:13.516+0000 7f716ac11640 -1 bluestore(/var/lib/ceph/osd/ceph-6/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-6/block: (2) No such file or directory 2024-03-25T18:26:13.518 INFO:teuthology.orchestra.run.smithi115.stderr:2024-03-25T18:26:13.516+0000 7f716ac11640 -1 bluestore(/var/lib/ceph/osd/ceph-6/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-6/block: (2) No such file or directory 2024-03-25T18:26:13.518 INFO:teuthology.orchestra.run.smithi115.stderr:2024-03-25T18:26:13.516+0000 7f716ac11640 -1 bluestore(/var/lib/ceph/osd/ceph-6/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-6/block: (2) No such file or directory 2024-03-25T18:26:13.518 INFO:teuthology.orchestra.run.smithi115.stderr:2024-03-25T18:26:13.517+0000 7f716ac11640 -1 bluestore(/var/lib/ceph/osd/ceph-6) _read_fsid unparsable uuid 2024-03-25T18:26:16.365 DEBUG:teuthology.orchestra.run.smithi115:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-6 2024-03-25T18:26:16.396 DEBUG:teuthology.orchestra.run.smithi115:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 7 --monmap /home/ubuntu/cephtest/ceph.monmap 2024-03-25T18:26:16.486 INFO:teuthology.orchestra.run.smithi115.stderr:2024-03-25T18:26:16.484+0000 7fd17bb9a640 -1 auth: error reading file: /var/lib/ceph/osd/ceph-7/keyring: can't open /var/lib/ceph/osd/ceph-7/keyring: (2) No such file or directory 2024-03-25T18:26:16.486 INFO:teuthology.orchestra.run.smithi115.stderr:2024-03-25T18:26:16.485+0000 7fd17bb9a640 -1 created new key in keyring /var/lib/ceph/osd/ceph-7/keyring 2024-03-25T18:26:16.486 INFO:teuthology.orchestra.run.smithi115.stderr:2024-03-25T18:26:16.485+0000 7fd17bb9a640 -1 bluestore(/var/lib/ceph/osd/ceph-7/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-7/block: (2) No such file or directory 2024-03-25T18:26:16.486 INFO:teuthology.orchestra.run.smithi115.stderr:2024-03-25T18:26:16.485+0000 7fd17bb9a640 -1 bluestore(/var/lib/ceph/osd/ceph-7/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-7/block: (2) No such file or directory 2024-03-25T18:26:16.486 INFO:teuthology.orchestra.run.smithi115.stderr:2024-03-25T18:26:16.485+0000 7fd17bb9a640 -1 bluestore(/var/lib/ceph/osd/ceph-7/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-7/block: (2) No such file or directory 2024-03-25T18:26:16.486 INFO:teuthology.orchestra.run.smithi115.stderr:2024-03-25T18:26:16.485+0000 7fd17bb9a640 -1 bluestore(/var/lib/ceph/osd/ceph-7) _read_fsid unparsable uuid 2024-03-25T18:26:19.326 DEBUG:teuthology.orchestra.run.smithi115:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-7 2024-03-25T18:26:19.356 INFO:tasks.ceph:Reading keys from all nodes... 2024-03-25T18:26:19.357 DEBUG:teuthology.orchestra.run.smithi001:> set -ex 2024-03-25T18:26:19.357 DEBUG:teuthology.orchestra.run.smithi001:> sudo dd if=/var/lib/ceph/mgr/ceph-y/keyring of=/dev/stdout 2024-03-25T18:26:19.389 DEBUG:teuthology.orchestra.run.smithi001:> set -ex 2024-03-25T18:26:19.389 DEBUG:teuthology.orchestra.run.smithi001:> sudo dd if=/var/lib/ceph/osd/ceph-0/keyring of=/dev/stdout 2024-03-25T18:26:19.458 DEBUG:teuthology.orchestra.run.smithi001:> set -ex 2024-03-25T18:26:19.458 DEBUG:teuthology.orchestra.run.smithi001:> sudo dd if=/var/lib/ceph/osd/ceph-1/keyring of=/dev/stdout 2024-03-25T18:26:19.528 DEBUG:teuthology.orchestra.run.smithi001:> set -ex 2024-03-25T18:26:19.528 DEBUG:teuthology.orchestra.run.smithi001:> sudo dd if=/var/lib/ceph/osd/ceph-2/keyring of=/dev/stdout 2024-03-25T18:26:19.597 DEBUG:teuthology.orchestra.run.smithi001:> set -ex 2024-03-25T18:26:19.597 DEBUG:teuthology.orchestra.run.smithi001:> sudo dd if=/var/lib/ceph/osd/ceph-3/keyring of=/dev/stdout 2024-03-25T18:26:19.666 DEBUG:teuthology.orchestra.run.smithi115:> set -ex 2024-03-25T18:26:19.666 DEBUG:teuthology.orchestra.run.smithi115:> sudo dd if=/var/lib/ceph/mgr/ceph-x/keyring of=/dev/stdout 2024-03-25T18:26:19.698 DEBUG:teuthology.orchestra.run.smithi115:> set -ex 2024-03-25T18:26:19.698 DEBUG:teuthology.orchestra.run.smithi115:> sudo dd if=/var/lib/ceph/osd/ceph-4/keyring of=/dev/stdout 2024-03-25T18:26:19.769 DEBUG:teuthology.orchestra.run.smithi115:> set -ex 2024-03-25T18:26:19.769 DEBUG:teuthology.orchestra.run.smithi115:> sudo dd if=/var/lib/ceph/osd/ceph-5/keyring of=/dev/stdout 2024-03-25T18:26:19.840 DEBUG:teuthology.orchestra.run.smithi115:> set -ex 2024-03-25T18:26:19.840 DEBUG:teuthology.orchestra.run.smithi115:> sudo dd if=/var/lib/ceph/osd/ceph-6/keyring of=/dev/stdout 2024-03-25T18:26:19.910 DEBUG:teuthology.orchestra.run.smithi115:> set -ex 2024-03-25T18:26:19.910 DEBUG:teuthology.orchestra.run.smithi115:> sudo dd if=/var/lib/ceph/osd/ceph-7/keyring of=/dev/stdout 2024-03-25T18:26:19.979 DEBUG:teuthology.orchestra.run.smithi001:> set -ex 2024-03-25T18:26:19.979 DEBUG:teuthology.orchestra.run.smithi001:> dd if=/etc/ceph/ceph.client.0.keyring of=/dev/stdout 2024-03-25T18:26:19.996 DEBUG:teuthology.orchestra.run.smithi115:> set -ex 2024-03-25T18:26:19.996 DEBUG:teuthology.orchestra.run.smithi115:> dd if=/etc/ceph/ceph.client.1.keyring of=/dev/stdout 2024-03-25T18:26:20.037 INFO:tasks.ceph:Adding keys to all mons... 2024-03-25T18:26:20.037 DEBUG:teuthology.orchestra.run.smithi001:> sudo tee -a /etc/ceph/ceph.keyring 2024-03-25T18:26:20.040 DEBUG:teuthology.orchestra.run.smithi115:> sudo tee -a /etc/ceph/ceph.keyring 2024-03-25T18:26:20.082 INFO:teuthology.orchestra.run.smithi001.stdout:[mgr.y] 2024-03-25T18:26:20.082 INFO:teuthology.orchestra.run.smithi001.stdout: key = AQCvwQFmo1nGAhAAGd/ZvXGi7DdfLTPr3Tv3gQ== 2024-03-25T18:26:20.082 INFO:teuthology.orchestra.run.smithi001.stdout:[osd.0] 2024-03-25T18:26:20.082 INFO:teuthology.orchestra.run.smithi001.stdout: key = AQCwwQFmpby0HxAAktVpBDFwl/WTwmjjWPqOMQ== 2024-03-25T18:26:20.082 INFO:teuthology.orchestra.run.smithi001.stdout:[osd.1] 2024-03-25T18:26:20.083 INFO:teuthology.orchestra.run.smithi001.stdout: key = AQCzwQFmPtigHRAAigViHj7VrCkA/5/Y8DGEig== 2024-03-25T18:26:20.083 INFO:teuthology.orchestra.run.smithi001.stdout:[osd.2] 2024-03-25T18:26:20.083 INFO:teuthology.orchestra.run.smithi001.stdout: key = AQC2wQFmD0PzGxAAYph5m5v/GuJ4qVT5exhNDQ== 2024-03-25T18:26:20.083 INFO:teuthology.orchestra.run.smithi001.stdout:[osd.3] 2024-03-25T18:26:20.083 INFO:teuthology.orchestra.run.smithi001.stdout: key = AQC5wQFmU++KGRAAOxG55d4G635Y0BcRt06txw== 2024-03-25T18:26:20.083 INFO:teuthology.orchestra.run.smithi001.stdout:[mgr.x] 2024-03-25T18:26:20.083 INFO:teuthology.orchestra.run.smithi001.stdout: key = AQCvwQFm/IDhBhAAqjNYg79yA56YOJDxtfNE3A== 2024-03-25T18:26:20.083 INFO:teuthology.orchestra.run.smithi001.stdout:[osd.4] 2024-03-25T18:26:20.084 INFO:teuthology.orchestra.run.smithi001.stdout: key = AQC/wQFmiyENIxAAmgntEsBisiKnLbGvWdRFfg== 2024-03-25T18:26:20.084 INFO:teuthology.orchestra.run.smithi001.stdout:[osd.5] 2024-03-25T18:26:20.084 INFO:teuthology.orchestra.run.smithi001.stdout: key = AQDCwQFmm4I0IRAADwRWiRjU88OVW1t3vcvXWA== 2024-03-25T18:26:20.084 INFO:teuthology.orchestra.run.smithi001.stdout:[osd.6] 2024-03-25T18:26:20.084 INFO:teuthology.orchestra.run.smithi001.stdout: key = AQDFwQFmwrzXHhAAVlDBPBtVH2/wCOU1laMwhw== 2024-03-25T18:26:20.084 INFO:teuthology.orchestra.run.smithi001.stdout:[osd.7] 2024-03-25T18:26:20.084 INFO:teuthology.orchestra.run.smithi001.stdout: key = AQDIwQFmsdLyHBAAgNTrGt2vjIfbz7Ou98f7Mg== 2024-03-25T18:26:20.085 INFO:teuthology.orchestra.run.smithi001.stdout:[client.0] 2024-03-25T18:26:20.085 INFO:teuthology.orchestra.run.smithi001.stdout: key = AQCvwQFma8GFCRAA1upolLydy9rUe3yYZ6KVIg== 2024-03-25T18:26:20.085 INFO:teuthology.orchestra.run.smithi001.stdout:[client.1] 2024-03-25T18:26:20.085 INFO:teuthology.orchestra.run.smithi001.stdout: key = AQCvwQFm7rDwDBAApwhfWFYvNXc3UXSHoqWKZg== 2024-03-25T18:26:20.104 INFO:teuthology.orchestra.run.smithi115.stdout:[mgr.y] 2024-03-25T18:26:20.105 INFO:teuthology.orchestra.run.smithi115.stdout: key = AQCvwQFmo1nGAhAAGd/ZvXGi7DdfLTPr3Tv3gQ== 2024-03-25T18:26:20.105 INFO:teuthology.orchestra.run.smithi115.stdout:[osd.0] 2024-03-25T18:26:20.105 INFO:teuthology.orchestra.run.smithi115.stdout: key = AQCwwQFmpby0HxAAktVpBDFwl/WTwmjjWPqOMQ== 2024-03-25T18:26:20.105 INFO:teuthology.orchestra.run.smithi115.stdout:[osd.1] 2024-03-25T18:26:20.105 INFO:teuthology.orchestra.run.smithi115.stdout: key = AQCzwQFmPtigHRAAigViHj7VrCkA/5/Y8DGEig== 2024-03-25T18:26:20.105 INFO:teuthology.orchestra.run.smithi115.stdout:[osd.2] 2024-03-25T18:26:20.105 INFO:teuthology.orchestra.run.smithi115.stdout: key = AQC2wQFmD0PzGxAAYph5m5v/GuJ4qVT5exhNDQ== 2024-03-25T18:26:20.105 INFO:teuthology.orchestra.run.smithi115.stdout:[osd.3] 2024-03-25T18:26:20.105 INFO:teuthology.orchestra.run.smithi115.stdout: key = AQC5wQFmU++KGRAAOxG55d4G635Y0BcRt06txw== 2024-03-25T18:26:20.105 INFO:teuthology.orchestra.run.smithi115.stdout:[mgr.x] 2024-03-25T18:26:20.105 INFO:teuthology.orchestra.run.smithi115.stdout: key = AQCvwQFm/IDhBhAAqjNYg79yA56YOJDxtfNE3A== 2024-03-25T18:26:20.105 INFO:teuthology.orchestra.run.smithi115.stdout:[osd.4] 2024-03-25T18:26:20.106 INFO:teuthology.orchestra.run.smithi115.stdout: key = AQC/wQFmiyENIxAAmgntEsBisiKnLbGvWdRFfg== 2024-03-25T18:26:20.106 INFO:teuthology.orchestra.run.smithi115.stdout:[osd.5] 2024-03-25T18:26:20.106 INFO:teuthology.orchestra.run.smithi115.stdout: key = AQDCwQFmm4I0IRAADwRWiRjU88OVW1t3vcvXWA== 2024-03-25T18:26:20.106 INFO:teuthology.orchestra.run.smithi115.stdout:[osd.6] 2024-03-25T18:26:20.106 INFO:teuthology.orchestra.run.smithi115.stdout: key = AQDFwQFmwrzXHhAAVlDBPBtVH2/wCOU1laMwhw== 2024-03-25T18:26:20.106 INFO:teuthology.orchestra.run.smithi115.stdout:[osd.7] 2024-03-25T18:26:20.106 INFO:teuthology.orchestra.run.smithi115.stdout: key = AQDIwQFmsdLyHBAAgNTrGt2vjIfbz7Ou98f7Mg== 2024-03-25T18:26:20.106 INFO:teuthology.orchestra.run.smithi115.stdout:[client.0] 2024-03-25T18:26:20.106 INFO:teuthology.orchestra.run.smithi115.stdout: key = AQCvwQFma8GFCRAA1upolLydy9rUe3yYZ6KVIg== 2024-03-25T18:26:20.106 INFO:teuthology.orchestra.run.smithi115.stdout:[client.1] 2024-03-25T18:26:20.106 INFO:teuthology.orchestra.run.smithi115.stdout: key = AQCvwQFm7rDwDBAApwhfWFYvNXc3UXSHoqWKZg== 2024-03-25T18:26:20.107 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=mgr.y --cap mon 'allow profile mgr' --cap osd 'allow *' --cap mds 'allow *' 2024-03-25T18:26:20.129 DEBUG:teuthology.orchestra.run.smithi115:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=mgr.y --cap mon 'allow profile mgr' --cap osd 'allow *' --cap mds 'allow *' 2024-03-25T18:26:20.200 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.0 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-03-25T18:26:20.202 DEBUG:teuthology.orchestra.run.smithi115:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.0 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-03-25T18:26:20.254 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.1 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-03-25T18:26:20.256 DEBUG:teuthology.orchestra.run.smithi115:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.1 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-03-25T18:26:20.307 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.2 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-03-25T18:26:20.309 DEBUG:teuthology.orchestra.run.smithi115:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.2 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-03-25T18:26:20.360 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.3 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-03-25T18:26:20.362 DEBUG:teuthology.orchestra.run.smithi115:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.3 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-03-25T18:26:20.413 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=mgr.x --cap mon 'allow profile mgr' --cap osd 'allow *' --cap mds 'allow *' 2024-03-25T18:26:20.416 DEBUG:teuthology.orchestra.run.smithi115:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=mgr.x --cap mon 'allow profile mgr' --cap osd 'allow *' --cap mds 'allow *' 2024-03-25T18:26:20.467 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.4 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-03-25T18:26:20.469 DEBUG:teuthology.orchestra.run.smithi115:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.4 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-03-25T18:26:20.520 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.5 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-03-25T18:26:20.522 DEBUG:teuthology.orchestra.run.smithi115:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.5 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-03-25T18:26:20.573 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.6 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-03-25T18:26:20.575 DEBUG:teuthology.orchestra.run.smithi115:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.6 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-03-25T18:26:20.626 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.7 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-03-25T18:26:20.628 DEBUG:teuthology.orchestra.run.smithi115:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.7 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-03-25T18:26:20.680 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=client.0 --cap mon 'allow rw' --cap mgr 'allow r' --cap osd 'allow rwx' --cap mds allow 2024-03-25T18:26:20.683 DEBUG:teuthology.orchestra.run.smithi115:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=client.0 --cap mon 'allow rw' --cap mgr 'allow r' --cap osd 'allow rwx' --cap mds allow 2024-03-25T18:26:20.737 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=client.1 --cap mon 'allow rw' --cap mgr 'allow r' --cap osd 'allow rwx' --cap mds allow 2024-03-25T18:26:20.739 DEBUG:teuthology.orchestra.run.smithi115:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=client.1 --cap mon 'allow rw' --cap mgr 'allow r' --cap osd 'allow rwx' --cap mds allow 2024-03-25T18:26:20.794 INFO:tasks.ceph:Running mkfs on mon nodes... 2024-03-25T18:26:20.794 DEBUG:teuthology.orchestra.run.smithi001:> sudo mkdir -p /var/lib/ceph/mon/ceph-a 2024-03-25T18:26:20.820 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-mon --cluster ceph --mkfs -i a --monmap /home/ubuntu/cephtest/ceph.monmap --keyring /etc/ceph/ceph.keyring 2024-03-25T18:26:20.916 DEBUG:teuthology.orchestra.run.smithi001:> sudo chown -R ceph:ceph /var/lib/ceph/mon/ceph-a 2024-03-25T18:26:20.942 DEBUG:teuthology.orchestra.run.smithi001:> sudo mkdir -p /var/lib/ceph/mon/ceph-c 2024-03-25T18:26:21.008 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-mon --cluster ceph --mkfs -i c --monmap /home/ubuntu/cephtest/ceph.monmap --keyring /etc/ceph/ceph.keyring 2024-03-25T18:26:21.104 DEBUG:teuthology.orchestra.run.smithi001:> sudo chown -R ceph:ceph /var/lib/ceph/mon/ceph-c 2024-03-25T18:26:21.131 DEBUG:teuthology.orchestra.run.smithi115:> sudo mkdir -p /var/lib/ceph/mon/ceph-b 2024-03-25T18:26:21.159 DEBUG:teuthology.orchestra.run.smithi115:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-mon --cluster ceph --mkfs -i b --monmap /home/ubuntu/cephtest/ceph.monmap --keyring /etc/ceph/ceph.keyring 2024-03-25T18:26:21.257 DEBUG:teuthology.orchestra.run.smithi115:> sudo chown -R ceph:ceph /var/lib/ceph/mon/ceph-b 2024-03-25T18:26:21.283 DEBUG:teuthology.orchestra.run.smithi001:> rm -- /home/ubuntu/cephtest/ceph.monmap 2024-03-25T18:26:21.286 DEBUG:teuthology.orchestra.run.smithi115:> rm -- /home/ubuntu/cephtest/ceph.monmap 2024-03-25T18:26:21.338 INFO:tasks.ceph:Starting mon daemons in cluster ceph... 2024-03-25T18:26:21.338 DEBUG:tasks.ceph_manager:running mon.a under valgrind with args ['cd', '/home/ubuntu/cephtest', Raw('&&'), 'sudo', 'adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'daemon-helper', 'term', 'env', 'OPENSSL_ia32cap=~0x1000000000000000', 'valgrind', '--trace-children=no', '--child-silent-after-fork=yes', '--soname-synonyms=somalloc=*tcmalloc*', '--num-callers=50', '--suppressions=/home/ubuntu/cephtest/valgrind.supp', '--xml=yes', '--xml-file=/var/log/ceph/valgrind/mon.a.log', '--time-stamp=yes', '--vgdb=yes', '--exit-on-first-error=yes', '--error-exitcode=42', '--tool=memcheck', '--leak-check=full', '--show-reachable=yes'] 2024-03-25T18:26:21.339 INFO:tasks.ceph.mon.a:Restarting daemon 2024-03-25T18:26:21.339 DEBUG:teuthology.orchestra.run.smithi001:> cd /home/ubuntu/cephtest && sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term env 'OPENSSL_ia32cap=~0x1000000000000000' valgrind --trace-children=no --child-silent-after-fork=yes '--soname-synonyms=somalloc=*tcmalloc*' --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/mon.a.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck --leak-check=full --show-reachable=yes ceph-mon -f --cluster ceph -i a 2024-03-25T18:26:21.341 INFO:tasks.ceph.mon.a:Started 2024-03-25T18:26:21.341 DEBUG:tasks.ceph_manager:running mon.c under valgrind with args ['cd', '/home/ubuntu/cephtest', Raw('&&'), 'sudo', 'adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'daemon-helper', 'term', 'env', 'OPENSSL_ia32cap=~0x1000000000000000', 'valgrind', '--trace-children=no', '--child-silent-after-fork=yes', '--soname-synonyms=somalloc=*tcmalloc*', '--num-callers=50', '--suppressions=/home/ubuntu/cephtest/valgrind.supp', '--xml=yes', '--xml-file=/var/log/ceph/valgrind/mon.c.log', '--time-stamp=yes', '--vgdb=yes', '--exit-on-first-error=yes', '--error-exitcode=42', '--tool=memcheck', '--leak-check=full', '--show-reachable=yes'] 2024-03-25T18:26:21.341 INFO:tasks.ceph.mon.c:Restarting daemon 2024-03-25T18:26:21.341 DEBUG:teuthology.orchestra.run.smithi001:> cd /home/ubuntu/cephtest && sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term env 'OPENSSL_ia32cap=~0x1000000000000000' valgrind --trace-children=no --child-silent-after-fork=yes '--soname-synonyms=somalloc=*tcmalloc*' --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/mon.c.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck --leak-check=full --show-reachable=yes ceph-mon -f --cluster ceph -i c 2024-03-25T18:26:21.343 INFO:tasks.ceph.mon.c:Started 2024-03-25T18:26:21.343 DEBUG:tasks.ceph_manager:running mon.b under valgrind with args ['cd', '/home/ubuntu/cephtest', Raw('&&'), 'sudo', 'adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'daemon-helper', 'term', 'env', 'OPENSSL_ia32cap=~0x1000000000000000', 'valgrind', '--trace-children=no', '--child-silent-after-fork=yes', '--soname-synonyms=somalloc=*tcmalloc*', '--num-callers=50', '--suppressions=/home/ubuntu/cephtest/valgrind.supp', '--xml=yes', '--xml-file=/var/log/ceph/valgrind/mon.b.log', '--time-stamp=yes', '--vgdb=yes', '--exit-on-first-error=yes', '--error-exitcode=42', '--tool=memcheck', '--leak-check=full', '--show-reachable=yes'] 2024-03-25T18:26:21.343 INFO:tasks.ceph.mon.b:Restarting daemon 2024-03-25T18:26:21.343 DEBUG:teuthology.orchestra.run.smithi115:> cd /home/ubuntu/cephtest && sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term env 'OPENSSL_ia32cap=~0x1000000000000000' valgrind --trace-children=no --child-silent-after-fork=yes '--soname-synonyms=somalloc=*tcmalloc*' --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/mon.b.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck --leak-check=full --show-reachable=yes ceph-mon -f --cluster ceph -i b 2024-03-25T18:26:21.381 INFO:tasks.ceph.mon.b:Started 2024-03-25T18:26:21.381 INFO:tasks.ceph:Starting mgr daemons in cluster ceph... 2024-03-25T18:26:21.381 INFO:tasks.ceph.mgr.y:Restarting daemon 2024-03-25T18:26:21.382 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term ceph-mgr -f --cluster ceph -i y 2024-03-25T18:26:21.384 INFO:tasks.ceph.mgr.y:Started 2024-03-25T18:26:21.384 INFO:tasks.ceph.mgr.x:Restarting daemon 2024-03-25T18:26:21.384 DEBUG:teuthology.orchestra.run.smithi115:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term ceph-mgr -f --cluster ceph -i x 2024-03-25T18:26:21.386 INFO:tasks.ceph.mgr.x:Started 2024-03-25T18:26:21.386 INFO:tasks.ceph:Setting crush tunables to default 2024-03-25T18:26:21.386 DEBUG:teuthology.orchestra.run.smithi001:> sudo ceph --cluster ceph osd crush tunables default 2024-03-25T18:26:21.488 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-25T18:26:21.486+0000 7fb36c833640 10 monclient: get_monmap_and_config 2024-03-25T18:26:21.488 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-25T18:26:21.486+0000 7fb36c833640 10 monclient: build_initial_monmap 2024-03-25T18:26:21.488 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-25T18:26:21.486+0000 7fb36c833640 10 monclient: monmap: 2024-03-25T18:26:21.488 INFO:teuthology.orchestra.run.smithi001.stderr:epoch 0 2024-03-25T18:26:21.488 INFO:teuthology.orchestra.run.smithi001.stderr:fsid c8ea72cf-9e99-4383-9b52-6c23d365fdfc 2024-03-25T18:26:21.489 INFO:teuthology.orchestra.run.smithi001.stderr:last_changed 2024-03-25T18:26:21.488425+0000 2024-03-25T18:26:21.489 INFO:teuthology.orchestra.run.smithi001.stderr:created 2024-03-25T18:26:21.488425+0000 2024-03-25T18:26:21.489 INFO:teuthology.orchestra.run.smithi001.stderr:min_mon_release 0 (unknown) 2024-03-25T18:26:21.489 INFO:teuthology.orchestra.run.smithi001.stderr:election_strategy: 3 2024-03-25T18:26:21.489 INFO:teuthology.orchestra.run.smithi001.stderr:0: v1:172.21.15.1:6789/0 mon.noname-a 2024-03-25T18:26:21.489 INFO:teuthology.orchestra.run.smithi001.stderr:1: v1:172.21.15.115:6789/0 mon.noname-c 2024-03-25T18:26:21.489 INFO:teuthology.orchestra.run.smithi001.stderr:2: v1:172.21.15.1:6790/0 mon.noname-b 2024-03-25T18:26:21.489 INFO:teuthology.orchestra.run.smithi001.stderr: 2024-03-25T18:26:21.490 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-25T18:26:21.487+0000 7fb36c833640 1 Processor -- start 2024-03-25T18:26:21.491 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-25T18:26:21.487+0000 7fb36c833640 1 -- start start 2024-03-25T18:26:21.491 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-25T18:26:21.487+0000 7fb36c833640 10 monclient: init 2024-03-25T18:26:21.491 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-25T18:26:21.488+0000 7fb36c833640 10 monclient: _reopen_session rank -1 2024-03-25T18:26:21.491 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-25T18:26:21.488+0000 7fb36c833640 10 monclient: _add_conns ranks=[2,0,1] 2024-03-25T18:26:21.491 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-25T18:26:21.488+0000 7fb36c833640 10 monclient(hunting): picked mon.noname-b con 0x7fb3680b4f00 addr v1:172.21.15.1:6790/0 2024-03-25T18:26:21.491 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-25T18:26:21.488+0000 7fb36c833640 10 monclient(hunting): picked mon.noname-a con 0x7fb3680b4510 addr v1:172.21.15.1:6789/0 2024-03-25T18:26:21.491 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-25T18:26:21.488+0000 7fb36c833640 10 monclient(hunting): picked mon.noname-c con 0x7fb368142ad0 addr v1:172.21.15.115:6789/0 2024-03-25T18:26:21.491 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-25T18:26:21.488+0000 7fb36c833640 1 -- --> v1:172.21.15.1:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb3680a9d90 con 0x7fb3680b4510 2024-03-25T18:26:21.491 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-25T18:26:21.488+0000 7fb36c833640 1 -- --> v1:172.21.15.115:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb3680aa2b0 con 0x7fb368142ad0 2024-03-25T18:26:21.491 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-25T18:26:21.488+0000 7fb36c833640 1 -- --> v1:172.21.15.1:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb36813fc60 con 0x7fb3680b4f00 2024-03-25T18:26:21.491 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-25T18:26:21.488+0000 7fb36c833640 10 monclient(hunting): _renew_subs 2024-03-25T18:26:21.491 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-25T18:26:21.488+0000 7fb36c833640 10 monclient(hunting): authenticate will time out at 7461.958934s 2024-03-25T18:26:21.492 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-25T18:26:21.488+0000 7fb36659c640 1 -- >> v1:172.21.15.1:6790/0 conn(0x7fb3680b4f00 legacy=0x7fb3681423c0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.1:6790/0 2024-03-25T18:26:21.492 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-25T18:26:21.488+0000 7fb35ffff640 1 -- >> v1:172.21.15.1:6789/0 conn(0x7fb3680b4510 legacy=0x7fb36813f550 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.1:6789/0 2024-03-25T18:26:21.492 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-25T18:26:21.488+0000 7fb366d9d640 1 -- >> v1:172.21.15.115:6789/0 conn(0x7fb368142ad0 legacy=0x7fb36814dc80 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.115:6789/0 2024-03-25T18:26:21.690 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-25T18:26:21.689+0000 7fb35ffff640 1 -- >> v1:172.21.15.1:6789/0 conn(0x7fb3680b4510 legacy=0x7fb36813f550 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.1:6789/0 2024-03-25T18:26:21.690 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-25T18:26:21.689+0000 7fb36659c640 1 -- >> v1:172.21.15.1:6790/0 conn(0x7fb3680b4f00 legacy=0x7fb3681423c0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.1:6790/0 2024-03-25T18:26:21.691 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-25T18:26:21.689+0000 7fb366d9d640 1 -- >> v1:172.21.15.115:6789/0 conn(0x7fb368142ad0 legacy=0x7fb36814dc80 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.115:6789/0 2024-03-25T18:26:22.091 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-25T18:26:22.090+0000 7fb35ffff640 1 -- >> v1:172.21.15.1:6789/0 conn(0x7fb3680b4510 legacy=0x7fb36813f550 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.1:6789/0 2024-03-25T18:26:22.092 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-25T18:26:22.090+0000 7fb36659c640 1 -- >> v1:172.21.15.1:6790/0 conn(0x7fb3680b4f00 legacy=0x7fb3681423c0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.1:6790/0 2024-03-25T18:26:22.092 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-25T18:26:22.090+0000 7fb366d9d640 1 -- >> v1:172.21.15.115:6789/0 conn(0x7fb368142ad0 legacy=0x7fb36814dc80 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.115:6789/0 2024-03-25T18:26:22.892 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-25T18:26:22.891+0000 7fb36659c640 1 -- >> v1:172.21.15.1:6790/0 conn(0x7fb3680b4f00 legacy=0x7fb3681423c0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.1:6790/0 2024-03-25T18:26:22.892 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-25T18:26:22.891+0000 7fb35ffff640 1 -- >> v1:172.21.15.1:6789/0 conn(0x7fb3680b4510 legacy=0x7fb36813f550 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.1:6789/0 2024-03-25T18:26:22.892 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-25T18:26:22.891+0000 7fb366d9d640 1 -- >> v1:172.21.15.115:6789/0 conn(0x7fb368142ad0 legacy=0x7fb36814dc80 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.115:6789/0 2024-03-25T18:26:24.490 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-25T18:26:24.488+0000 7fb364d99640 10 monclient(hunting): tick 2024-03-25T18:26:24.490 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-25T18:26:24.488+0000 7fb364d99640 10 monclient(hunting): _check_auth_tickets 2024-03-25T18:26:24.490 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-25T18:26:24.488+0000 7fb364d99640 1 monclient(hunting): continuing hunt 2024-03-25T18:26:24.490 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-25T18:26:24.488+0000 7fb364d99640 10 monclient(hunting): _reopen_session rank -1 2024-03-25T18:26:24.491 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-25T18:26:24.488+0000 7fb364d99640 1 -- >> v1:172.21.15.1:6790/0 conn(0x7fb3680b4f00 legacy=0x7fb3681423c0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-03-25T18:26:24.491 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-25T18:26:24.488+0000 7fb364d99640 1 -- >> v1:172.21.15.115:6789/0 conn(0x7fb368142ad0 legacy=0x7fb36814dc80 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-03-25T18:26:24.491 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-25T18:26:24.488+0000 7fb364d99640 1 -- >> v1:172.21.15.1:6789/0 conn(0x7fb3680b4510 legacy=0x7fb36813f550 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-03-25T18:26:24.491 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-25T18:26:24.488+0000 7fb364d99640 10 monclient: _add_conns ranks=[1,2,0] 2024-03-25T18:26:24.491 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-25T18:26:24.488+0000 7fb364d99640 10 monclient(hunting): picked mon.noname-c con 0x7fb368142ad0 addr v1:172.21.15.115:6789/0 2024-03-25T18:26:24.491 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-25T18:26:24.488+0000 7fb364d99640 10 monclient(hunting): picked mon.noname-b con 0x7fb3680b4f00 addr v1:172.21.15.1:6790/0 2024-03-25T18:26:24.491 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-25T18:26:24.488+0000 7fb364d99640 10 monclient(hunting): picked mon.noname-a con 0x7fb3680b4510 addr v1:172.21.15.1:6789/0 2024-03-25T18:26:24.491 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-25T18:26:24.488+0000 7fb364d99640 1 -- --> v1:172.21.15.1:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb36813fdf0 con 0x7fb3680b4510 2024-03-25T18:26:24.491 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-25T18:26:24.488+0000 7fb364d99640 1 -- --> v1:172.21.15.115:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb3680a9d90 con 0x7fb368142ad0 2024-03-25T18:26:24.491 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-25T18:26:24.488+0000 7fb364d99640 1 -- --> v1:172.21.15.1:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb3680aa2b0 con 0x7fb3680b4f00 2024-03-25T18:26:24.491 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-25T18:26:24.488+0000 7fb364d99640 10 monclient(hunting): _renew_subs 2024-03-25T18:26:24.491 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-25T18:26:24.488+0000 7fb35ffff640 1 -- >> v1:172.21.15.1:6790/0 conn(0x7fb3680b4f00 legacy=0x7fb348008ea0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.1:6790/0 2024-03-25T18:26:24.491 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-25T18:26:24.488+0000 7fb366d9d640 1 -- >> v1:172.21.15.1:6789/0 conn(0x7fb3680b4510 legacy=0x7fb34800c700 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.1:6789/0 2024-03-25T18:26:24.491 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-25T18:26:24.488+0000 7fb36659c640 1 -- >> v1:172.21.15.115:6789/0 conn(0x7fb368142ad0 legacy=0x7fb3480056d0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.115:6789/0 2024-03-25T18:26:24.690 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-25T18:26:24.689+0000 7fb35ffff640 1 -- >> v1:172.21.15.1:6790/0 conn(0x7fb3680b4f00 legacy=0x7fb348008ea0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.1:6790/0 2024-03-25T18:26:24.690 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-25T18:26:24.689+0000 7fb366d9d640 1 -- >> v1:172.21.15.1:6789/0 conn(0x7fb3680b4510 legacy=0x7fb34800c700 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.1:6789/0 2024-03-25T18:26:24.691 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-25T18:26:24.689+0000 7fb36659c640 1 -- >> v1:172.21.15.115:6789/0 conn(0x7fb368142ad0 legacy=0x7fb3480056d0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.115:6789/0 2024-03-25T18:26:25.091 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-25T18:26:25.090+0000 7fb366d9d640 1 -- >> v1:172.21.15.1:6789/0 conn(0x7fb3680b4510 legacy=0x7fb34800c700 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.1:6789/0 2024-03-25T18:26:25.091 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-25T18:26:25.090+0000 7fb35ffff640 1 -- >> v1:172.21.15.1:6790/0 conn(0x7fb3680b4f00 legacy=0x7fb348008ea0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.1:6790/0 2024-03-25T18:26:25.092 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-25T18:26:25.090+0000 7fb36659c640 1 -- >> v1:172.21.15.115:6789/0 conn(0x7fb368142ad0 legacy=0x7fb3480056d0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.115:6789/0 2024-03-25T18:26:25.892 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-25T18:26:25.891+0000 7fb35ffff640 1 -- >> v1:172.21.15.1:6790/0 conn(0x7fb3680b4f00 legacy=0x7fb348008ea0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.1:6790/0 2024-03-25T18:26:25.892 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-25T18:26:25.891+0000 7fb366d9d640 1 -- >> v1:172.21.15.1:6789/0 conn(0x7fb3680b4510 legacy=0x7fb34800c700 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.1:6789/0 2024-03-25T18:26:25.893 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-25T18:26:25.891+0000 7fb36659c640 1 -- >> v1:172.21.15.115:6789/0 conn(0x7fb368142ad0 legacy=0x7fb3480056d0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.115:6789/0 2024-03-25T18:26:27.490 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-25T18:26:27.488+0000 7fb364d99640 10 monclient(hunting): tick 2024-03-25T18:26:27.490 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-25T18:26:27.488+0000 7fb364d99640 10 monclient(hunting): _check_auth_tickets 2024-03-25T18:26:27.490 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-25T18:26:27.488+0000 7fb364d99640 1 monclient(hunting): continuing hunt 2024-03-25T18:26:27.490 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-25T18:26:27.488+0000 7fb364d99640 10 monclient(hunting): _reopen_session rank -1 2024-03-25T18:26:27.491 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-25T18:26:27.488+0000 7fb364d99640 1 -- >> v1:172.21.15.1:6790/0 conn(0x7fb3680b4f00 legacy=0x7fb348008ea0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-03-25T18:26:27.491 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-25T18:26:27.488+0000 7fb364d99640 1 -- >> v1:172.21.15.115:6789/0 conn(0x7fb368142ad0 legacy=0x7fb3480056d0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-03-25T18:26:27.491 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-25T18:26:27.488+0000 7fb364d99640 1 -- >> v1:172.21.15.1:6789/0 conn(0x7fb3680b4510 legacy=0x7fb34800c700 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-03-25T18:26:27.491 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-25T18:26:27.488+0000 7fb364d99640 10 monclient: _add_conns ranks=[1,0,2] 2024-03-25T18:26:27.491 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-25T18:26:27.488+0000 7fb364d99640 10 monclient(hunting): picked mon.noname-c con 0x7fb368142ad0 addr v1:172.21.15.115:6789/0 2024-03-25T18:26:27.491 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-25T18:26:27.488+0000 7fb364d99640 10 monclient(hunting): picked mon.noname-a con 0x7fb3680b4510 addr v1:172.21.15.1:6789/0 2024-03-25T18:26:27.491 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-25T18:26:27.488+0000 7fb364d99640 10 monclient(hunting): picked mon.noname-b con 0x7fb3680b4f00 addr v1:172.21.15.1:6790/0 2024-03-25T18:26:27.491 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-25T18:26:27.488+0000 7fb364d99640 1 -- --> v1:172.21.15.1:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb36813fc60 con 0x7fb3680b4510 2024-03-25T18:26:27.491 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-25T18:26:27.488+0000 7fb364d99640 1 -- --> v1:172.21.15.115:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb36813fdf0 con 0x7fb368142ad0 2024-03-25T18:26:27.491 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-25T18:26:27.488+0000 7fb364d99640 1 -- --> v1:172.21.15.1:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb3680a9d90 con 0x7fb3680b4f00 2024-03-25T18:26:27.491 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-25T18:26:27.488+0000 7fb364d99640 10 monclient(hunting): _renew_subs 2024-03-25T18:26:27.491 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-25T18:26:27.488+0000 7fb35ffff640 1 -- >> v1:172.21.15.1:6789/0 conn(0x7fb3680b4510 legacy=0x7fb34800c680 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.1:6789/0 2024-03-25T18:26:27.491 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-25T18:26:27.488+0000 7fb366d9d640 1 -- >> v1:172.21.15.1:6790/0 conn(0x7fb3680b4f00 legacy=0x7fb348008ea0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.1:6790/0 2024-03-25T18:26:27.491 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-25T18:26:27.488+0000 7fb36659c640 1 -- >> v1:172.21.15.115:6789/0 conn(0x7fb368142ad0 legacy=0x7fb348005e00 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.115:6789/0 2024-03-25T18:26:27.690 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-25T18:26:27.689+0000 7fb35ffff640 1 -- >> v1:172.21.15.1:6789/0 conn(0x7fb3680b4510 legacy=0x7fb34800c680 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.1:6789/0 2024-03-25T18:26:27.690 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-25T18:26:27.689+0000 7fb366d9d640 1 -- >> v1:172.21.15.1:6790/0 conn(0x7fb3680b4f00 legacy=0x7fb348008ea0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.1:6790/0 2024-03-25T18:26:27.691 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-25T18:26:27.689+0000 7fb36659c640 1 -- >> v1:172.21.15.115:6789/0 conn(0x7fb368142ad0 legacy=0x7fb348005e00 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.115:6789/0 2024-03-25T18:26:28.091 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-25T18:26:28.090+0000 7fb366d9d640 1 -- >> v1:172.21.15.1:6790/0 conn(0x7fb3680b4f00 legacy=0x7fb348008ea0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.1:6790/0 2024-03-25T18:26:28.091 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-25T18:26:28.090+0000 7fb35ffff640 1 -- >> v1:172.21.15.1:6789/0 conn(0x7fb3680b4510 legacy=0x7fb34800c680 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.1:6789/0 2024-03-25T18:26:28.092 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-25T18:26:28.090+0000 7fb36659c640 1 -- >> v1:172.21.15.115:6789/0 conn(0x7fb368142ad0 legacy=0x7fb348005e00 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.115:6789/0 2024-03-25T18:26:28.344 INFO:tasks.ceph.mon.b.smithi115.stderr:==00:00:00:06.827 34836== Warning: unimplemented fcntl command: 1036 2024-03-25T18:26:28.649 INFO:tasks.ceph.mon.b.smithi115.stderr:==00:00:00:07.133 34836== Warning: unimplemented fcntl command: 1036 2024-03-25T18:26:28.818 INFO:tasks.ceph.mon.a.smithi001.stderr:==00:00:00:07.377 34876== Warning: unimplemented fcntl command: 1036 2024-03-25T18:26:28.819 INFO:tasks.ceph.mon.c.smithi001.stderr:==00:00:00:07.377 34877== Warning: unimplemented fcntl command: 1036 2024-03-25T18:26:28.892 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-25T18:26:28.891+0000 7fb366d9d640 1 -- >> v1:172.21.15.1:6790/0 conn(0x7fb3680b4f00 legacy=0x7fb348008ea0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.1:6790/0 2024-03-25T18:26:28.892 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-25T18:26:28.891+0000 7fb35ffff640 1 -- >> v1:172.21.15.1:6789/0 conn(0x7fb3680b4510 legacy=0x7fb34800c680 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.1:6789/0 2024-03-25T18:26:28.893 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-25T18:26:28.891+0000 7fb36659c640 1 -- >> v1:172.21.15.115:6789/0 conn(0x7fb368142ad0 legacy=0x7fb348005e00 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.115:6789/0 2024-03-25T18:26:29.146 INFO:tasks.ceph.mon.a.smithi001.stderr:==00:00:00:07.705 34876== Warning: unimplemented fcntl command: 1036 2024-03-25T18:26:29.147 INFO:tasks.ceph.mon.c.smithi001.stderr:==00:00:00:07.705 34877== Warning: unimplemented fcntl command: 1036 2024-03-25T18:26:30.490 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-25T18:26:30.488+0000 7fb364d99640 10 monclient(hunting): tick 2024-03-25T18:26:30.490 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-25T18:26:30.488+0000 7fb364d99640 10 monclient(hunting): _check_auth_tickets 2024-03-25T18:26:30.490 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-25T18:26:30.488+0000 7fb364d99640 1 monclient(hunting): continuing hunt 2024-03-25T18:26:30.490 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-25T18:26:30.488+0000 7fb364d99640 10 monclient(hunting): _reopen_session rank -1 2024-03-25T18:26:30.490 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-25T18:26:30.488+0000 7fb364d99640 1 -- >> v1:172.21.15.1:6790/0 conn(0x7fb3680b4f00 legacy=0x7fb348008ea0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-03-25T18:26:30.491 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-25T18:26:30.488+0000 7fb364d99640 1 -- >> v1:172.21.15.115:6789/0 conn(0x7fb368142ad0 legacy=0x7fb348005e00 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-03-25T18:26:30.491 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-25T18:26:30.488+0000 7fb364d99640 1 -- >> v1:172.21.15.1:6789/0 conn(0x7fb3680b4510 legacy=0x7fb34800c680 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-03-25T18:26:30.491 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-25T18:26:30.488+0000 7fb364d99640 10 monclient: _add_conns ranks=[0,2,1] 2024-03-25T18:26:30.491 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-25T18:26:30.488+0000 7fb364d99640 10 monclient(hunting): picked mon.noname-a con 0x7fb3680b4510 addr v1:172.21.15.1:6789/0 2024-03-25T18:26:30.491 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-25T18:26:30.488+0000 7fb364d99640 10 monclient(hunting): picked mon.noname-b con 0x7fb3680b4f00 addr v1:172.21.15.1:6790/0 2024-03-25T18:26:30.491 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-25T18:26:30.488+0000 7fb364d99640 10 monclient(hunting): picked mon.noname-c con 0x7fb368142ad0 addr v1:172.21.15.115:6789/0 2024-03-25T18:26:30.491 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-25T18:26:30.488+0000 7fb364d99640 1 -- --> v1:172.21.15.1:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb3680aa2b0 con 0x7fb3680b4510 2024-03-25T18:26:30.491 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-25T18:26:30.488+0000 7fb364d99640 1 -- --> v1:172.21.15.115:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb36813fc60 con 0x7fb368142ad0 2024-03-25T18:26:30.491 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-25T18:26:30.488+0000 7fb364d99640 1 -- --> v1:172.21.15.1:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb36813fdf0 con 0x7fb3680b4f00 2024-03-25T18:26:30.491 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-25T18:26:30.488+0000 7fb364d99640 10 monclient(hunting): _renew_subs 2024-03-25T18:26:30.546 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-25T18:26:30.545+0000 7fb366d9d640 1 --1- >> v1:172.21.15.115:6789/0 conn(0x7fb368142ad0 0x7fb3480065d0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.115:6789/0 says I am v1:172.21.15.1:45546/0 (socket says 172.21.15.1:45546) 2024-03-25T18:26:30.546 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-25T18:26:30.545+0000 7fb366d9d640 1 -- 172.21.15.1:0/2382153330 learned_addr learned my addr 172.21.15.1:0/2382153330 (peer_addr_for_me v1:172.21.15.1:0/0) 2024-03-25T18:26:30.850 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-25T18:26:30.848+0000 7fb36659c640 1 -- 172.21.15.1:0/2382153330 >> v1:172.21.15.1:6789/0 conn(0x7fb3680b4510 legacy=0x7fb3480056a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 13 2024-03-25T18:26:30.850 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-25T18:26:30.848+0000 7fb36659c640 1 -- 172.21.15.1:0/2382153330 >> v1:172.21.15.1:6789/0 conn(0x7fb3680b4510 legacy=0x7fb3480056a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-03-25T18:26:30.850 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-25T18:26:30.848+0000 7fb36659c640 1 --1- 172.21.15.1:0/2382153330 >> v1:172.21.15.1:6789/0 conn(0x7fb3680b4510 0x7fb3480056a0 :-1 s=CONNECTING_SEND_CONNECT_MSG pgs=0 cs=0 l=0).handle_connect_reply_1 read connect reply failed 2024-03-25T18:26:30.852 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-25T18:26:30.851+0000 7fb35ffff640 1 -- 172.21.15.1:0/2382153330 >> v1:172.21.15.1:6790/0 conn(0x7fb3680b4f00 legacy=0x7fb348008ea0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 14 2024-03-25T18:26:30.852 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-25T18:26:30.851+0000 7fb35ffff640 1 -- 172.21.15.1:0/2382153330 >> v1:172.21.15.1:6790/0 conn(0x7fb3680b4f00 legacy=0x7fb348008ea0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-03-25T18:26:30.852 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-25T18:26:30.851+0000 7fb35ffff640 1 --1- 172.21.15.1:0/2382153330 >> v1:172.21.15.1:6790/0 conn(0x7fb3680b4f00 0x7fb348008ea0 :-1 s=CONNECTING_SEND_CONNECT_MSG pgs=0 cs=0 l=0).handle_connect_reply_1 read connect reply failed 2024-03-25T18:26:33.491 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-25T18:26:33.489+0000 7fb364d99640 10 monclient(hunting): tick 2024-03-25T18:26:33.491 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-25T18:26:33.489+0000 7fb364d99640 10 monclient(hunting): _check_auth_tickets 2024-03-25T18:26:33.492 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-25T18:26:33.489+0000 7fb364d99640 1 monclient(hunting): continuing hunt 2024-03-25T18:26:33.492 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-25T18:26:33.489+0000 7fb364d99640 10 monclient(hunting): _reopen_session rank -1 2024-03-25T18:26:33.492 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-25T18:26:33.489+0000 7fb364d99640 1 -- 172.21.15.1:0/2382153330 >> v1:172.21.15.1:6790/0 conn(0x7fb3680b4f00 legacy=0x7fb348008ea0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:26:33.492 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-25T18:26:33.489+0000 7fb364d99640 1 -- 172.21.15.1:0/2382153330 >> v1:172.21.15.115:6789/0 conn(0x7fb368142ad0 legacy=0x7fb3480065d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:26:33.492 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-25T18:26:33.489+0000 7fb364d99640 1 -- 172.21.15.1:0/2382153330 >> v1:172.21.15.1:6789/0 conn(0x7fb3680b4510 legacy=0x7fb3480056a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:26:33.492 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-25T18:26:33.489+0000 7fb364d99640 10 monclient: _add_conns ranks=[0,1,2] 2024-03-25T18:26:33.492 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-25T18:26:33.489+0000 7fb364d99640 10 monclient(hunting): picked mon.noname-a con 0x7fb348005e40 addr v1:172.21.15.1:6789/0 2024-03-25T18:26:33.492 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-25T18:26:33.489+0000 7fb364d99640 10 monclient(hunting): picked mon.noname-c con 0x7fb348012030 addr v1:172.21.15.115:6789/0 2024-03-25T18:26:33.492 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-25T18:26:33.489+0000 7fb364d99640 10 monclient(hunting): picked mon.noname-b con 0x7fb3680b4f00 addr v1:172.21.15.1:6790/0 2024-03-25T18:26:33.492 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-25T18:26:33.489+0000 7fb364d99640 1 -- 172.21.15.1:0/2382153330 --> v1:172.21.15.1:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb3680a9d90 con 0x7fb348005e40 2024-03-25T18:26:33.492 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-25T18:26:33.489+0000 7fb364d99640 1 -- 172.21.15.1:0/2382153330 --> v1:172.21.15.115:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb3480065d0 con 0x7fb348012030 2024-03-25T18:26:33.492 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-25T18:26:33.489+0000 7fb364d99640 1 -- 172.21.15.1:0/2382153330 --> v1:172.21.15.1:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb348006760 con 0x7fb3680b4f00 2024-03-25T18:26:33.492 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-25T18:26:33.489+0000 7fb364d99640 10 monclient(hunting): _renew_subs 2024-03-25T18:26:36.491 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-25T18:26:36.490+0000 7fb364d99640 10 monclient(hunting): tick 2024-03-25T18:26:36.491 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-25T18:26:36.490+0000 7fb364d99640 10 monclient(hunting): _check_auth_tickets 2024-03-25T18:26:36.492 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-25T18:26:36.490+0000 7fb364d99640 1 monclient(hunting): continuing hunt 2024-03-25T18:26:36.492 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-25T18:26:36.490+0000 7fb364d99640 10 monclient(hunting): _reopen_session rank -1 2024-03-25T18:26:36.492 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-25T18:26:36.490+0000 7fb364d99640 1 -- 172.21.15.1:0/2382153330 >> v1:172.21.15.1:6790/0 conn(0x7fb3680b4f00 legacy=0x7fb3480056a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:26:36.492 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-25T18:26:36.490+0000 7fb364d99640 1 -- 172.21.15.1:0/2382153330 >> v1:172.21.15.115:6789/0 conn(0x7fb348012030 legacy=0x7fb348012410 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:26:36.492 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-25T18:26:36.490+0000 7fb364d99640 1 -- 172.21.15.1:0/2382153330 >> v1:172.21.15.1:6789/0 conn(0x7fb348005e40 legacy=0x7fb34800c680 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:26:36.492 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-25T18:26:36.490+0000 7fb364d99640 10 monclient: _add_conns ranks=[2,0,1] 2024-03-25T18:26:36.492 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-25T18:26:36.490+0000 7fb364d99640 10 monclient(hunting): picked mon.noname-b con 0x7fb34800ce30 addr v1:172.21.15.1:6790/0 2024-03-25T18:26:36.492 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-25T18:26:36.490+0000 7fb364d99640 10 monclient(hunting): picked mon.noname-a con 0x7fb3480056a0 addr v1:172.21.15.1:6789/0 2024-03-25T18:26:36.492 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-25T18:26:36.490+0000 7fb364d99640 10 monclient(hunting): picked mon.noname-c con 0x7fb348012030 addr v1:172.21.15.115:6789/0 2024-03-25T18:26:36.492 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-25T18:26:36.490+0000 7fb364d99640 1 -- 172.21.15.1:0/2382153330 --> v1:172.21.15.1:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb3480068f0 con 0x7fb3480056a0 2024-03-25T18:26:36.492 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-25T18:26:36.490+0000 7fb364d99640 1 -- 172.21.15.1:0/2382153330 --> v1:172.21.15.115:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb348005b80 con 0x7fb348012030 2024-03-25T18:26:36.492 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-25T18:26:36.490+0000 7fb364d99640 1 -- 172.21.15.1:0/2382153330 --> v1:172.21.15.1:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb34800d980 con 0x7fb34800ce30 2024-03-25T18:26:36.492 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-25T18:26:36.490+0000 7fb364d99640 10 monclient(hunting): _renew_subs 2024-03-25T18:26:39.491 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-25T18:26:39.489+0000 7fb364d99640 10 monclient(hunting): tick 2024-03-25T18:26:39.491 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-25T18:26:39.489+0000 7fb364d99640 10 monclient(hunting): _check_auth_tickets 2024-03-25T18:26:39.491 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-25T18:26:39.489+0000 7fb364d99640 1 monclient(hunting): continuing hunt 2024-03-25T18:26:39.492 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-25T18:26:39.489+0000 7fb364d99640 10 monclient(hunting): _reopen_session rank -1 2024-03-25T18:26:39.492 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-25T18:26:39.489+0000 7fb364d99640 1 -- 172.21.15.1:0/2382153330 >> v1:172.21.15.1:6790/0 conn(0x7fb34800ce30 legacy=0x7fb348008ea0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:26:39.492 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-25T18:26:39.489+0000 7fb364d99640 1 -- 172.21.15.1:0/2382153330 >> v1:172.21.15.115:6789/0 conn(0x7fb348012030 legacy=0x7fb348012410 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:26:39.492 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-25T18:26:39.489+0000 7fb364d99640 1 -- 172.21.15.1:0/2382153330 >> v1:172.21.15.1:6789/0 conn(0x7fb3480056a0 legacy=0x7fb34800d270 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:26:39.492 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-25T18:26:39.489+0000 7fb364d99640 10 monclient: _add_conns ranks=[0,1,2] 2024-03-25T18:26:39.492 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-25T18:26:39.489+0000 7fb364d99640 10 monclient(hunting): picked mon.noname-a con 0x7fb348011020 addr v1:172.21.15.1:6789/0 2024-03-25T18:26:39.492 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-25T18:26:39.489+0000 7fb364d99640 10 monclient(hunting): picked mon.noname-c con 0x7fb348012030 addr v1:172.21.15.115:6789/0 2024-03-25T18:26:39.492 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-25T18:26:39.489+0000 7fb364d99640 10 monclient(hunting): picked mon.noname-b con 0x7fb34800ce30 addr v1:172.21.15.1:6790/0 2024-03-25T18:26:39.492 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-25T18:26:39.489+0000 7fb364d99640 1 -- 172.21.15.1:0/2382153330 --> v1:172.21.15.1:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb34800dbb0 con 0x7fb348011020 2024-03-25T18:26:39.492 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-25T18:26:39.489+0000 7fb364d99640 1 -- 172.21.15.1:0/2382153330 --> v1:172.21.15.115:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb3480134f0 con 0x7fb348012030 2024-03-25T18:26:39.492 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-25T18:26:39.489+0000 7fb364d99640 1 -- 172.21.15.1:0/2382153330 --> v1:172.21.15.1:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb348013720 con 0x7fb34800ce30 2024-03-25T18:26:39.492 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-25T18:26:39.489+0000 7fb364d99640 10 monclient(hunting): _renew_subs 2024-03-25T18:26:39.649 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-25T18:26:39.648+0000 7fb365d9b640 1 -- 172.21.15.1:0/2382153330 <== mon.0 v1:172.21.15.1:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3862959839 0 0) 0x7fb34800dbb0 con 0x7fb348011020 2024-03-25T18:26:39.650 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-25T18:26:39.648+0000 7fb365d9b640 10 monclient(hunting): _negotiate 2024-03-25T18:26:39.650 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-25T18:26:39.648+0000 7fb365d9b640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:26:39.650 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-25T18:26:39.648+0000 7fb365d9b640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:26:39.650 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-25T18:26:39.648+0000 7fb365d9b640 10 monclient(hunting): my global_id is 4101 2024-03-25T18:26:39.650 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-25T18:26:39.648+0000 7fb365d9b640 1 -- 172.21.15.1:0/2382153330 --> v1:172.21.15.1:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fb354003520 con 0x7fb348011020 2024-03-25T18:26:39.654 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-25T18:26:39.652+0000 7fb365d9b640 1 -- 172.21.15.1:0/2382153330 <== mon.1 v1:172.21.15.115:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3159085135 0 0) 0x7fb3480134f0 con 0x7fb348012030 2024-03-25T18:26:39.654 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-25T18:26:39.652+0000 7fb365d9b640 10 monclient(hunting): _negotiate 2024-03-25T18:26:39.654 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-25T18:26:39.652+0000 7fb365d9b640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:26:39.654 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-25T18:26:39.652+0000 7fb365d9b640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:26:39.654 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-25T18:26:39.652+0000 7fb365d9b640 10 monclient(hunting): my global_id is 4102 2024-03-25T18:26:39.654 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-25T18:26:39.652+0000 7fb365d9b640 1 -- 172.21.15.1:0/2382153330 --> v1:172.21.15.115:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fb34800dbb0 con 0x7fb348012030 2024-03-25T18:26:39.669 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-25T18:26:39.668+0000 7fb365d9b640 1 -- 172.21.15.1:0/2382153330 <== mon.2 v1:172.21.15.1:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 315144308 0 0) 0x7fb348013720 con 0x7fb34800ce30 2024-03-25T18:26:39.670 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-25T18:26:39.668+0000 7fb365d9b640 10 monclient(hunting): _negotiate 2024-03-25T18:26:39.670 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-25T18:26:39.668+0000 7fb365d9b640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:26:39.670 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-25T18:26:39.668+0000 7fb365d9b640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:26:39.670 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-25T18:26:39.668+0000 7fb365d9b640 10 monclient(hunting): my global_id is 4106 2024-03-25T18:26:39.670 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-25T18:26:39.668+0000 7fb365d9b640 1 -- 172.21.15.1:0/2382153330 --> v1:172.21.15.1:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fb3480134f0 con 0x7fb34800ce30 2024-03-25T18:26:39.707 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-25T18:26:39.705+0000 7fb365d9b640 1 -- 172.21.15.1:0/2382153330 <== mon.0 v1:172.21.15.1:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3229898923 0 0) 0x7fb354003520 con 0x7fb348011020 2024-03-25T18:26:39.707 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-25T18:26:39.705+0000 7fb365d9b640 1 -- 172.21.15.1:0/2382153330 >> v1:172.21.15.1:6790/0 conn(0x7fb34800ce30 legacy=0x7fb348008ea0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:26:39.707 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-25T18:26:39.705+0000 7fb365d9b640 1 -- 172.21.15.1:0/2382153330 >> v1:172.21.15.115:6789/0 conn(0x7fb348012030 legacy=0x7fb34800d270 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:26:39.707 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-25T18:26:39.705+0000 7fb365d9b640 10 monclient: _finish_hunting 0 2024-03-25T18:26:39.708 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-25T18:26:39.705+0000 7fb365d9b640 1 monclient: found mon.noname-a 2024-03-25T18:26:39.708 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-25T18:26:39.705+0000 7fb365d9b640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:26:39.708 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-25T18:26:39.705+0000 7fb365d9b640 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.1:6789/0 2024-03-25T18:26:39.708 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-25T18:26:39.705+0000 7fb365d9b640 1 -- 172.21.15.1:0/2382153330 --> v1:172.21.15.1:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb348013950 con 0x7fb348011020 2024-03-25T18:26:39.708 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-25T18:26:39.705+0000 7fb365d9b640 10 monclient: _finish_auth 0 2024-03-25T18:26:39.708 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-25T18:26:39.705+0000 7fb365d9b640 10 monclient: _check_auth_tickets 2024-03-25T18:26:39.708 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-25T18:26:39.705+0000 7fb365d9b640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:26:39.708 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-25T18:26:39.705+0000 7fb365d9b640 1 -- 172.21.15.1:0/2382153330 <== mon.0 v1:172.21.15.1:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1586965025 0 0) 0x7fb358003ea0 con 0x7fb348011020 2024-03-25T18:26:39.708 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-25T18:26:39.705+0000 7fb365d9b640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:26:39.708 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-25T18:26:39.705+0000 7fb365d9b640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-03-25T18:26:39.708 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-25T18:26:39.705+0000 7fb365d9b640 10 monclient: dump: 2024-03-25T18:26:39.708 INFO:teuthology.orchestra.run.smithi001.stderr:epoch 1 2024-03-25T18:26:39.708 INFO:teuthology.orchestra.run.smithi001.stderr:fsid c8ea72cf-9e99-4383-9b52-6c23d365fdfc 2024-03-25T18:26:39.708 INFO:teuthology.orchestra.run.smithi001.stderr:last_changed 2024-03-25T18:25:50.510519+0000 2024-03-25T18:26:39.709 INFO:teuthology.orchestra.run.smithi001.stderr:created 2024-03-25T18:25:50.510519+0000 2024-03-25T18:26:39.709 INFO:teuthology.orchestra.run.smithi001.stderr:min_mon_release 19 (squid) 2024-03-25T18:26:39.709 INFO:teuthology.orchestra.run.smithi001.stderr:election_strategy: 3 2024-03-25T18:26:39.709 INFO:teuthology.orchestra.run.smithi001.stderr:0: v1:172.21.15.1:6789/0 mon.a 2024-03-25T18:26:39.709 INFO:teuthology.orchestra.run.smithi001.stderr:1: v1:172.21.15.115:6789/0 mon.b 2024-03-25T18:26:39.709 INFO:teuthology.orchestra.run.smithi001.stderr:2: v1:172.21.15.1:6790/0 mon.c 2024-03-25T18:26:39.709 INFO:teuthology.orchestra.run.smithi001.stderr: 2024-03-25T18:26:39.709 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-25T18:26:39.705+0000 7fb36c833640 5 monclient: authenticate success, global_id 4101 2024-03-25T18:26:39.709 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-25T18:26:39.705+0000 7fb36c833640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T18:26:39.785 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-25T18:26:39.782+0000 7fb365d9b640 1 -- 172.21.15.1:0/2382153330 <== mon.0 v1:172.21.15.1:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fb358004000 con 0x7fb348011020 2024-03-25T18:26:39.785 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-25T18:26:39.782+0000 7fb365d9b640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:26:39.785 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-25T18:26:39.782+0000 7fb365d9b640 1 -- 172.21.15.1:0/2382153330 <== mon.0 v1:172.21.15.1:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1586965025 0 0) 0x7fb358004450 con 0x7fb348011020 2024-03-25T18:26:39.786 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-25T18:26:39.782+0000 7fb365d9b640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:26:39.786 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-25T18:26:39.782+0000 7fb365d9b640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-25T18:26:39.786 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-25T18:26:39.782+0000 7fb365d9b640 10 monclient: dump: 2024-03-25T18:26:39.786 INFO:teuthology.orchestra.run.smithi001.stderr:epoch 1 2024-03-25T18:26:39.786 INFO:teuthology.orchestra.run.smithi001.stderr:fsid c8ea72cf-9e99-4383-9b52-6c23d365fdfc 2024-03-25T18:26:39.786 INFO:teuthology.orchestra.run.smithi001.stderr:last_changed 2024-03-25T18:25:50.510519+0000 2024-03-25T18:26:39.786 INFO:teuthology.orchestra.run.smithi001.stderr:created 2024-03-25T18:25:50.510519+0000 2024-03-25T18:26:39.787 INFO:teuthology.orchestra.run.smithi001.stderr:min_mon_release 19 (squid) 2024-03-25T18:26:39.787 INFO:teuthology.orchestra.run.smithi001.stderr:election_strategy: 3 2024-03-25T18:26:39.787 INFO:teuthology.orchestra.run.smithi001.stderr:0: v1:172.21.15.1:6789/0 mon.a 2024-03-25T18:26:39.787 INFO:teuthology.orchestra.run.smithi001.stderr:1: v1:172.21.15.115:6789/0 mon.b 2024-03-25T18:26:39.787 INFO:teuthology.orchestra.run.smithi001.stderr:2: v1:172.21.15.1:6790/0 mon.c 2024-03-25T18:26:39.787 INFO:teuthology.orchestra.run.smithi001.stderr: 2024-03-25T18:26:39.787 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-25T18:26:39.782+0000 7fb36c833640 10 monclient: get_monmap_and_config success 2024-03-25T18:26:39.787 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-25T18:26:39.782+0000 7fb36c833640 4 set_mon_vals no callback set 2024-03-25T18:26:39.787 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-25T18:26:39.783+0000 7fb36c833640 10 monclient: shutdown 2024-03-25T18:26:39.788 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-25T18:26:39.783+0000 7fb36c833640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:26:39.788 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-25T18:26:39.783+0000 7fb36c833640 1 -- 172.21.15.1:0/2382153330 >> v1:172.21.15.1:6789/0 conn(0x7fb348011020 legacy=0x7fb34800c680 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:26:39.788 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-25T18:26:39.783+0000 7fb36c833640 1 -- 172.21.15.1:0/2382153330 shutdown_connections 2024-03-25T18:26:39.788 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-25T18:26:39.783+0000 7fb36c833640 1 -- 172.21.15.1:0/2382153330 >> 172.21.15.1:0/2382153330 conn(0x7fb368066b00 msgr2=0x7fb368066f10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:26:39.788 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-25T18:26:39.783+0000 7fb36c833640 1 -- 172.21.15.1:0/2382153330 shutdown_connections 2024-03-25T18:26:39.788 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-25T18:26:39.783+0000 7fb36c833640 1 -- 172.21.15.1:0/2382153330 wait complete. 2024-03-25T18:26:39.791 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-25T18:26:39.784+0000 7fb36c833640 10 monclient: build_initial_monmap 2024-03-25T18:26:39.791 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-25T18:26:39.784+0000 7fb36c833640 10 monclient: monmap: 2024-03-25T18:26:39.791 INFO:teuthology.orchestra.run.smithi001.stderr:epoch 0 2024-03-25T18:26:39.791 INFO:teuthology.orchestra.run.smithi001.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T18:26:39.791 INFO:teuthology.orchestra.run.smithi001.stderr:last_changed 0.000000 2024-03-25T18:26:39.791 INFO:teuthology.orchestra.run.smithi001.stderr:created 0.000000 2024-03-25T18:26:39.791 INFO:teuthology.orchestra.run.smithi001.stderr:min_mon_release 0 (unknown) 2024-03-25T18:26:39.791 INFO:teuthology.orchestra.run.smithi001.stderr:election_strategy: 1 2024-03-25T18:26:39.791 INFO:teuthology.orchestra.run.smithi001.stderr:0: v1:172.21.15.1:6789/0 mon.noname-a 2024-03-25T18:26:39.791 INFO:teuthology.orchestra.run.smithi001.stderr:1: v1:172.21.15.115:6789/0 mon.noname-b 2024-03-25T18:26:39.792 INFO:teuthology.orchestra.run.smithi001.stderr:2: v1:172.21.15.1:6790/0 mon.noname-c 2024-03-25T18:26:39.792 INFO:teuthology.orchestra.run.smithi001.stderr: 2024-03-25T18:26:39.792 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-25T18:26:39.784+0000 7fb36c833640 1 librados: starting msgr at 2024-03-25T18:26:39.792 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-25T18:26:39.784+0000 7fb36c833640 1 librados: starting objecter 2024-03-25T18:26:39.792 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-25T18:26:39.784+0000 7fb36c833640 1 Processor -- start 2024-03-25T18:26:39.792 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-25T18:26:39.784+0000 7fb36c833640 1 -- start start 2024-03-25T18:26:39.792 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-25T18:26:39.784+0000 7fb36c833640 1 librados: setting wanted keys 2024-03-25T18:26:39.792 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-25T18:26:39.784+0000 7fb36c833640 1 librados: calling monclient init 2024-03-25T18:26:39.792 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-25T18:26:39.784+0000 7fb36c833640 10 monclient: init 2024-03-25T18:26:39.792 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-25T18:26:39.785+0000 7fb36c833640 10 monclient: _reopen_session rank -1 2024-03-25T18:26:39.792 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-25T18:26:39.785+0000 7fb36c833640 10 monclient: _add_conns ranks=[1,0,2] 2024-03-25T18:26:39.793 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-25T18:26:39.785+0000 7fb36c833640 10 monclient(hunting): picked mon.noname-b con 0x7fb34800ce30 addr v1:172.21.15.115:6789/0 2024-03-25T18:26:39.793 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-25T18:26:39.785+0000 7fb36c833640 10 monclient(hunting): picked mon.noname-a con 0x7fb348011020 addr v1:172.21.15.1:6789/0 2024-03-25T18:26:39.793 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-25T18:26:39.785+0000 7fb36c833640 10 monclient(hunting): picked mon.noname-c con 0x7fb348012030 addr v1:172.21.15.1:6790/0 2024-03-25T18:26:39.793 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-25T18:26:39.785+0000 7fb36c833640 1 -- --> v1:172.21.15.1:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb3680aec00 con 0x7fb348011020 2024-03-25T18:26:39.793 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-25T18:26:39.785+0000 7fb36c833640 1 -- --> v1:172.21.15.115:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb3681d7910 con 0x7fb34800ce30 2024-03-25T18:26:39.793 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-25T18:26:39.785+0000 7fb36c833640 1 -- --> v1:172.21.15.1:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb3680adb60 con 0x7fb348012030 2024-03-25T18:26:39.793 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-25T18:26:39.785+0000 7fb36c833640 10 monclient(hunting): _renew_subs 2024-03-25T18:26:39.793 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-25T18:26:39.785+0000 7fb36c833640 10 monclient(hunting): authenticate will time out at 7480.256176s 2024-03-25T18:26:39.793 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-25T18:26:39.786+0000 7fb35ffff640 1 --1- >> v1:172.21.15.1:6789/0 conn(0x7fb348011020 0x7fb3681d7e30 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.1:6789/0 says I am v1:172.21.15.1:39158/0 (socket says 172.21.15.1:39158) 2024-03-25T18:26:39.793 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-25T18:26:39.786+0000 7fb35ffff640 1 -- 172.21.15.1:0/307914922 learned_addr learned my addr 172.21.15.1:0/307914922 (peer_addr_for_me v1:172.21.15.1:0/0) 2024-03-25T18:26:39.794 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-25T18:26:39.792+0000 7fb35effd640 10 client.?.objecter ms_handle_connect 0x7fb348011020 2024-03-25T18:26:39.794 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-25T18:26:39.792+0000 7fb35effd640 10 client.?.objecter resend_mon_ops 2024-03-25T18:26:39.805 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-25T18:26:39.804+0000 7fb35effd640 10 client.?.objecter ms_handle_connect 0x7fb34800ce30 2024-03-25T18:26:39.805 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-25T18:26:39.804+0000 7fb35effd640 10 client.?.objecter resend_mon_ops 2024-03-25T18:26:39.806 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-25T18:26:39.804+0000 7fb36659c640 1 -- 172.21.15.1:0/307914922 >> v1:172.21.15.115:6789/0 conn(0x7fb34800ce30 legacy=0x7fb3680ad450 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 16 2024-03-25T18:26:39.807 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-25T18:26:39.804+0000 7fb36659c640 1 -- 172.21.15.1:0/307914922 >> v1:172.21.15.115:6789/0 conn(0x7fb34800ce30 legacy=0x7fb3680ad450 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-03-25T18:26:39.807 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-25T18:26:39.804+0000 7fb36659c640 1 --1- 172.21.15.1:0/307914922 >> v1:172.21.15.115:6789/0 conn(0x7fb34800ce30 0x7fb3680ad450 :-1 s=OPENED pgs=25 cs=1 l=1).handle_message read tag failed 2024-03-25T18:26:39.807 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-25T18:26:39.804+0000 7fb36659c640 1 --1- 172.21.15.1:0/307914922 >> v1:172.21.15.115:6789/0 conn(0x7fb34800ce30 0x7fb3680ad450 :-1 s=OPENED pgs=25 cs=1 l=1).fault on lossy channel, failing 2024-03-25T18:26:39.807 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-25T18:26:39.804+0000 7fb35effd640 10 monclient(hunting): ms_handle_reset hunted mon v1:172.21.15.115:6789/0 2024-03-25T18:26:39.819 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-25T18:26:39.817+0000 7fb35effd640 1 -- 172.21.15.1:0/307914922 <== mon.0 v1:172.21.15.1:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1899360139 0 0) 0x7fb3680aec00 con 0x7fb348011020 2024-03-25T18:26:39.819 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-25T18:26:39.817+0000 7fb35effd640 10 monclient(hunting): _negotiate 2024-03-25T18:26:39.819 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-25T18:26:39.817+0000 7fb35effd640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:26:39.820 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-25T18:26:39.817+0000 7fb35effd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:26:39.820 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-25T18:26:39.817+0000 7fb35effd640 10 monclient(hunting): my global_id is 4107 2024-03-25T18:26:39.820 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-25T18:26:39.817+0000 7fb35effd640 1 -- 172.21.15.1:0/307914922 --> v1:172.21.15.1:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fb348013720 con 0x7fb348011020 2024-03-25T18:26:39.846 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-25T18:26:39.844+0000 7fb35effd640 1 -- 172.21.15.1:0/307914922 <== mon.0 v1:172.21.15.1:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2804178254 0 0) 0x7fb348013720 con 0x7fb348011020 2024-03-25T18:26:39.846 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-25T18:26:39.844+0000 7fb35effd640 1 -- 172.21.15.1:0/307914922 --> v1:172.21.15.1:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fb3680aec00 con 0x7fb348011020 2024-03-25T18:26:39.846 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-25T18:26:39.844+0000 7fb35effd640 1 -- 172.21.15.1:0/307914922 <== mon.0 v1:172.21.15.1:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1586965025 0 0) 0x7fb3500042d0 con 0x7fb348011020 2024-03-25T18:26:39.846 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-25T18:26:39.844+0000 7fb35effd640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:26:39.846 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-25T18:26:39.844+0000 7fb35effd640 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2024-03-25T18:26:39.846 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-25T18:26:39.844+0000 7fb35effd640 10 monclient(hunting): dump: 2024-03-25T18:26:39.846 INFO:teuthology.orchestra.run.smithi001.stderr:epoch 1 2024-03-25T18:26:39.846 INFO:teuthology.orchestra.run.smithi001.stderr:fsid c8ea72cf-9e99-4383-9b52-6c23d365fdfc 2024-03-25T18:26:39.847 INFO:teuthology.orchestra.run.smithi001.stderr:last_changed 2024-03-25T18:25:50.510519+0000 2024-03-25T18:26:39.847 INFO:teuthology.orchestra.run.smithi001.stderr:created 2024-03-25T18:25:50.510519+0000 2024-03-25T18:26:39.847 INFO:teuthology.orchestra.run.smithi001.stderr:min_mon_release 19 (squid) 2024-03-25T18:26:39.847 INFO:teuthology.orchestra.run.smithi001.stderr:election_strategy: 3 2024-03-25T18:26:39.847 INFO:teuthology.orchestra.run.smithi001.stderr:0: v1:172.21.15.1:6789/0 mon.a 2024-03-25T18:26:39.847 INFO:teuthology.orchestra.run.smithi001.stderr:1: v1:172.21.15.115:6789/0 mon.b 2024-03-25T18:26:39.847 INFO:teuthology.orchestra.run.smithi001.stderr:2: v1:172.21.15.1:6790/0 mon.c 2024-03-25T18:26:39.847 INFO:teuthology.orchestra.run.smithi001.stderr: 2024-03-25T18:26:39.847 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-25T18:26:39.844+0000 7fb35effd640 10 monclient(hunting): _finish_auth 0 2024-03-25T18:26:39.875 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-25T18:26:39.873+0000 7fb35effd640 10 client.?.objecter ms_handle_connect 0x7fb348012030 2024-03-25T18:26:39.875 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-25T18:26:39.873+0000 7fb35effd640 10 client.?.objecter resend_mon_ops 2024-03-25T18:26:39.876 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-25T18:26:39.874+0000 7fb366d9d640 1 -- 172.21.15.1:0/307914922 >> v1:172.21.15.1:6790/0 conn(0x7fb348012030 legacy=0x7fb368139340 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 18 2024-03-25T18:26:39.876 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-25T18:26:39.874+0000 7fb366d9d640 1 -- 172.21.15.1:0/307914922 >> v1:172.21.15.1:6790/0 conn(0x7fb348012030 legacy=0x7fb368139340 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-03-25T18:26:39.876 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-25T18:26:39.874+0000 7fb366d9d640 1 --1- 172.21.15.1:0/307914922 >> v1:172.21.15.1:6790/0 conn(0x7fb348012030 0x7fb368139340 :-1 s=OPENED pgs=23 cs=1 l=1).handle_message read tag failed 2024-03-25T18:26:39.876 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-25T18:26:39.874+0000 7fb366d9d640 1 --1- 172.21.15.1:0/307914922 >> v1:172.21.15.1:6790/0 conn(0x7fb348012030 0x7fb368139340 :-1 s=OPENED pgs=23 cs=1 l=1).fault on lossy channel, failing 2024-03-25T18:26:39.876 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-25T18:26:39.874+0000 7fb35effd640 10 monclient(hunting): ms_handle_reset hunted mon v1:172.21.15.1:6790/0 2024-03-25T18:26:39.899 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-25T18:26:39.895+0000 7fb35effd640 1 -- 172.21.15.1:0/307914922 <== mon.0 v1:172.21.15.1:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 239253332 0 0) 0x7fb3680aec00 con 0x7fb348011020 2024-03-25T18:26:39.899 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-25T18:26:39.895+0000 7fb35effd640 1 -- 172.21.15.1:0/307914922 >> v1:172.21.15.1:6790/0 conn(0x7fb348012030 legacy=0x7fb368139340 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-03-25T18:26:39.899 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-25T18:26:39.895+0000 7fb35effd640 1 -- 172.21.15.1:0/307914922 >> v1:172.21.15.115:6789/0 conn(0x7fb34800ce30 legacy=0x7fb3680ad450 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-03-25T18:26:39.900 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-25T18:26:39.895+0000 7fb35effd640 10 monclient: _finish_hunting 0 2024-03-25T18:26:39.900 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-25T18:26:39.895+0000 7fb35effd640 1 monclient: found mon.a 2024-03-25T18:26:39.900 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-25T18:26:39.895+0000 7fb35effd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:26:39.900 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-25T18:26:39.895+0000 7fb35effd640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.1:6789/0 2024-03-25T18:26:39.900 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-25T18:26:39.895+0000 7fb35effd640 1 -- 172.21.15.1:0/307914922 --> v1:172.21.15.1:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb3680aced0 con 0x7fb348011020 2024-03-25T18:26:39.900 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-25T18:26:39.895+0000 7fb35effd640 10 monclient: _finish_auth 0 2024-03-25T18:26:39.900 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-25T18:26:39.895+0000 7fb35effd640 10 monclient: _check_auth_tickets 2024-03-25T18:26:39.900 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-25T18:26:39.895+0000 7fb35effd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:26:39.901 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-25T18:26:39.895+0000 7fb36c833640 5 monclient: authenticate success, global_id 4107 2024-03-25T18:26:39.901 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-25T18:26:39.895+0000 7fb36c833640 10 monclient: _renew_subs 2024-03-25T18:26:39.901 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-25T18:26:39.895+0000 7fb36c833640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.1:6789/0 2024-03-25T18:26:39.901 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-25T18:26:39.895+0000 7fb36c833640 1 -- 172.21.15.1:0/307914922 --> v1:172.21.15.1:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb368144c40 con 0x7fb348011020 2024-03-25T18:26:39.901 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-25T18:26:39.895+0000 7fb36c833640 10 client.4107.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T18:26:39.901 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-25T18:26:39.895+0000 7fb36c833640 10 monclient: _renew_subs 2024-03-25T18:26:39.902 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-25T18:26:39.895+0000 7fb36c833640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.1:6789/0 2024-03-25T18:26:39.902 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-25T18:26:39.895+0000 7fb36c833640 1 -- 172.21.15.1:0/307914922 --> v1:172.21.15.1:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7fb3680ac710 con 0x7fb348011020 2024-03-25T18:26:39.902 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-25T18:26:39.895+0000 7fb36c833640 1 librados: init done 2024-03-25T18:26:39.902 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-25T18:26:39.896+0000 7fb36c833640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T18:26:39.902 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-25T18:26:39.896+0000 7fb36c833640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:26:39.903 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-25T18:26:39.896+0000 7fb36c833640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.1:6789/0 2024-03-25T18:26:39.903 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-25T18:26:39.896+0000 7fb36c833640 1 -- 172.21.15.1:0/307914922 --> v1:172.21.15.1:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb36804f470 con 0x7fb348011020 2024-03-25T18:26:40.107 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-25T18:26:40.105+0000 7fb35effd640 1 -- 172.21.15.1:0/307914922 <== mon.0 v1:172.21.15.1:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fb3500046e0 con 0x7fb348011020 2024-03-25T18:26:40.107 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-25T18:26:40.105+0000 7fb35effd640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:26:40.107 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-25T18:26:40.105+0000 7fb35effd640 1 -- 172.21.15.1:0/307914922 <== mon.0 v1:172.21.15.1:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1586965025 0 0) 0x7fb350004950 con 0x7fb348011020 2024-03-25T18:26:40.107 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-25T18:26:40.105+0000 7fb35effd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:26:40.107 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-25T18:26:40.105+0000 7fb35effd640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-25T18:26:40.107 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-25T18:26:40.105+0000 7fb35effd640 10 monclient: dump: 2024-03-25T18:26:40.108 INFO:teuthology.orchestra.run.smithi001.stderr:epoch 1 2024-03-25T18:26:40.108 INFO:teuthology.orchestra.run.smithi001.stderr:fsid c8ea72cf-9e99-4383-9b52-6c23d365fdfc 2024-03-25T18:26:40.108 INFO:teuthology.orchestra.run.smithi001.stderr:last_changed 2024-03-25T18:25:50.510519+0000 2024-03-25T18:26:40.108 INFO:teuthology.orchestra.run.smithi001.stderr:created 2024-03-25T18:25:50.510519+0000 2024-03-25T18:26:40.108 INFO:teuthology.orchestra.run.smithi001.stderr:min_mon_release 19 (squid) 2024-03-25T18:26:40.108 INFO:teuthology.orchestra.run.smithi001.stderr:election_strategy: 3 2024-03-25T18:26:40.108 INFO:teuthology.orchestra.run.smithi001.stderr:0: v1:172.21.15.1:6789/0 mon.a 2024-03-25T18:26:40.108 INFO:teuthology.orchestra.run.smithi001.stderr:1: v1:172.21.15.115:6789/0 mon.b 2024-03-25T18:26:40.108 INFO:teuthology.orchestra.run.smithi001.stderr:2: v1:172.21.15.1:6790/0 mon.c 2024-03-25T18:26:40.108 INFO:teuthology.orchestra.run.smithi001.stderr: 2024-03-25T18:26:40.108 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-25T18:26:40.105+0000 7fb35effd640 1 -- 172.21.15.1:0/307914922 <== mon.0 v1:172.21.15.1:6789/0 7 ==== mgrmap(e 1) v1 ==== 807+0+0 (unknown 3189685207 0 0) 0x7fb350004f00 con 0x7fb348011020 2024-03-25T18:26:40.108 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-25T18:26:40.105+0000 7fb36559a640 4 set_mon_vals no callback set 2024-03-25T18:26:40.108 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-25T18:26:40.105+0000 7fb35effd640 1 -- 172.21.15.1:0/307914922 <== mon.0 v1:172.21.15.1:6789/0 8 ==== osd_map(1..1 src has 1..1) v4 ==== 725+0+0 (unknown 4108718690 0 0) 0x7fb3500063d0 con 0x7fb348011020 2024-03-25T18:26:40.108 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-25T18:26:40.105+0000 7fb35effd640 10 client.4107.objecter ms_dispatch 0x7fb368000b90 osd_map(1..1 src has 1..1) v4 2024-03-25T18:26:40.109 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-25T18:26:40.105+0000 7fb35effd640 3 client.4107.objecter handle_osd_map got epochs [1,1] > 0 2024-03-25T18:26:40.109 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-25T18:26:40.105+0000 7fb35effd640 3 client.4107.objecter handle_osd_map decoding full epoch 1 2024-03-25T18:26:40.109 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-25T18:26:40.105+0000 7fb35effd640 20 client.4107.objecter dump_active .. 0 homeless 2024-03-25T18:26:40.109 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-25T18:26:40.105+0000 7fb35effd640 1 -- 172.21.15.1:0/307914922 <== mon.0 v1:172.21.15.1:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+73141 (unknown 1092875540 0 2031569661) 0x7fb350006840 con 0x7fb348011020 2024-03-25T18:26:40.109 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-25T18:26:40.105+0000 7fb35effd640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:26:40.109 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-25T18:26:40.105+0000 7fb35effd640 10 monclient: _finish_command 1 = system:0 2024-03-25T18:26:40.178 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-25T18:26:40.176+0000 7fb36c833640 10 start_mon_command cmd=[{"prefix": "osd crush tunables", "profile": "default"}] 2024-03-25T18:26:40.178 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-25T18:26:40.176+0000 7fb36c833640 10 monclient: _send_command 2 [{"prefix": "osd crush tunables", "profile": "default"}] 2024-03-25T18:26:40.179 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-25T18:26:40.176+0000 7fb36c833640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.1:6789/0 2024-03-25T18:26:40.179 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-25T18:26:40.176+0000 7fb36c833640 1 -- 172.21.15.1:0/307914922 --> v1:172.21.15.1:6789/0 -- mon_command({"prefix": "osd crush tunables", "profile": "default"} v 0) v1 -- 0x7fb3680ab140 con 0x7fb348011020 2024-03-25T18:26:40.313 INFO:tasks.ceph.mgr.x.smithi115.stderr:2024-03-25T18:26:40.312+0000 7f99dd247040 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2024-03-25T18:26:40.349 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-25T18:26:40.347+0000 7fb35effd640 1 -- 172.21.15.1:0/307914922 <== mon.0 v1:172.21.15.1:6789/0 10 ==== mon_command_ack([{"prefix": "osd crush tunables", "profile": "default"}]=0 adjusted tunables profile to default v2) v1 ==== 124+0+0 (unknown 2277712955 0 0) 0x7fb3500054c0 con 0x7fb348011020 2024-03-25T18:26:40.349 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-25T18:26:40.347+0000 7fb35effd640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd crush tunables", "profile": "default"}] 2024-03-25T18:26:40.349 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-25T18:26:40.347+0000 7fb35effd640 10 monclient: _finish_command 2 = system:0 adjusted tunables profile to default 2024-03-25T18:26:40.349 INFO:teuthology.orchestra.run.smithi001.stderr:adjusted tunables profile to default 2024-03-25T18:26:40.350 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-25T18:26:40.348+0000 7fb36c833640 10 librados: watch_flush enter 2024-03-25T18:26:40.350 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-25T18:26:40.348+0000 7fb36c833640 10 librados: watch_flush exit 2024-03-25T18:26:40.351 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-25T18:26:40.348+0000 7fb36c833640 20 client.4107.objecter shutdown clearing up homeless session... 2024-03-25T18:26:40.351 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-25T18:26:40.348+0000 7fb36c833640 10 client.4107.objecter successfully canceled tick 2024-03-25T18:26:40.351 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-25T18:26:40.348+0000 7fb36c833640 10 monclient: shutdown 2024-03-25T18:26:40.351 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-25T18:26:40.348+0000 7fb36c833640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:26:40.351 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-25T18:26:40.348+0000 7fb36c833640 1 -- 172.21.15.1:0/307914922 >> v1:172.21.15.1:6789/0 conn(0x7fb348011020 legacy=0x7fb3681d7e30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:26:40.351 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-25T18:26:40.348+0000 7fb36c833640 1 -- 172.21.15.1:0/307914922 shutdown_connections 2024-03-25T18:26:40.351 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-25T18:26:40.348+0000 7fb36c833640 1 -- 172.21.15.1:0/307914922 >> 172.21.15.1:0/307914922 conn(0x7fb368066b00 msgr2=0x7fb36813f0a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:26:40.351 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-25T18:26:40.348+0000 7fb36c833640 1 -- 172.21.15.1:0/307914922 shutdown_connections 2024-03-25T18:26:40.351 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-25T18:26:40.348+0000 7fb36c833640 1 -- 172.21.15.1:0/307914922 wait complete. 2024-03-25T18:26:40.351 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-25T18:26:40.348+0000 7fb36c833640 1 librados: shutdown 2024-03-25T18:26:40.361 INFO:tasks.ceph:check_enable_crimson: False 2024-03-25T18:26:40.361 INFO:tasks.ceph:Starting osd daemons in cluster ceph... 2024-03-25T18:26:40.362 DEBUG:teuthology.orchestra.run.smithi001:> set -ex 2024-03-25T18:26:40.362 DEBUG:teuthology.orchestra.run.smithi001:> sudo dd if=/var/lib/ceph/osd/ceph-0/fsid of=/dev/stdout 2024-03-25T18:26:40.368 INFO:tasks.ceph.mgr.x.smithi115.stderr:2024-03-25T18:26:40.367+0000 7f99dd247040 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2024-03-25T18:26:40.390 DEBUG:teuthology.orchestra.run.smithi001:> set -ex 2024-03-25T18:26:40.390 DEBUG:teuthology.orchestra.run.smithi001:> sudo dd if=/var/lib/ceph/osd/ceph-1/fsid of=/dev/stdout 2024-03-25T18:26:40.458 DEBUG:teuthology.orchestra.run.smithi001:> set -ex 2024-03-25T18:26:40.458 DEBUG:teuthology.orchestra.run.smithi001:> sudo dd if=/var/lib/ceph/osd/ceph-2/fsid of=/dev/stdout 2024-03-25T18:26:40.526 DEBUG:teuthology.orchestra.run.smithi001:> set -ex 2024-03-25T18:26:40.526 DEBUG:teuthology.orchestra.run.smithi001:> sudo dd if=/var/lib/ceph/osd/ceph-3/fsid of=/dev/stdout 2024-03-25T18:26:40.592 DEBUG:teuthology.orchestra.run.smithi115:> set -ex 2024-03-25T18:26:40.593 DEBUG:teuthology.orchestra.run.smithi115:> sudo dd if=/var/lib/ceph/osd/ceph-4/fsid of=/dev/stdout 2024-03-25T18:26:40.621 DEBUG:teuthology.orchestra.run.smithi115:> set -ex 2024-03-25T18:26:40.621 DEBUG:teuthology.orchestra.run.smithi115:> sudo dd if=/var/lib/ceph/osd/ceph-5/fsid of=/dev/stdout 2024-03-25T18:26:40.661 INFO:tasks.ceph.mgr.y.smithi001.stderr:2024-03-25T18:26:40.660+0000 7faca7441040 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2024-03-25T18:26:40.688 DEBUG:teuthology.orchestra.run.smithi115:> set -ex 2024-03-25T18:26:40.688 DEBUG:teuthology.orchestra.run.smithi115:> sudo dd if=/var/lib/ceph/osd/ceph-6/fsid of=/dev/stdout 2024-03-25T18:26:40.729 INFO:tasks.ceph.mgr.y.smithi001.stderr:2024-03-25T18:26:40.727+0000 7faca7441040 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2024-03-25T18:26:40.754 DEBUG:teuthology.orchestra.run.smithi115:> set -ex 2024-03-25T18:26:40.754 DEBUG:teuthology.orchestra.run.smithi115:> sudo dd if=/var/lib/ceph/osd/ceph-7/fsid of=/dev/stdout 2024-03-25T18:26:40.822 DEBUG:teuthology.orchestra.run.smithi115:> sudo ceph --cluster ceph osd new b9f34922-3a84-49be-8426-807c1eceaefc 0 2024-03-25T18:26:40.907 INFO:tasks.ceph.mgr.x.smithi115.stderr:2024-03-25T18:26:40.906+0000 7f99dd247040 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2024-03-25T18:26:40.962 INFO:teuthology.orchestra.run.smithi115.stderr:2024-03-25T18:26:40.961+0000 7fc56b3af640 10 monclient: get_monmap_and_config 2024-03-25T18:26:40.962 INFO:teuthology.orchestra.run.smithi115.stderr:2024-03-25T18:26:40.961+0000 7fc56b3af640 10 monclient: build_initial_monmap 2024-03-25T18:26:40.962 INFO:teuthology.orchestra.run.smithi115.stderr:2024-03-25T18:26:40.961+0000 7fc56b3af640 10 monclient: monmap: 2024-03-25T18:26:40.962 INFO:teuthology.orchestra.run.smithi115.stderr:epoch 0 2024-03-25T18:26:40.962 INFO:teuthology.orchestra.run.smithi115.stderr:fsid c8ea72cf-9e99-4383-9b52-6c23d365fdfc 2024-03-25T18:26:40.962 INFO:teuthology.orchestra.run.smithi115.stderr:last_changed 2024-03-25T18:26:40.962201+0000 2024-03-25T18:26:40.962 INFO:teuthology.orchestra.run.smithi115.stderr:created 2024-03-25T18:26:40.962201+0000 2024-03-25T18:26:40.962 INFO:teuthology.orchestra.run.smithi115.stderr:min_mon_release 0 (unknown) 2024-03-25T18:26:40.962 INFO:teuthology.orchestra.run.smithi115.stderr:election_strategy: 3 2024-03-25T18:26:40.962 INFO:teuthology.orchestra.run.smithi115.stderr:0: v1:172.21.15.1:6789/0 mon.noname-a 2024-03-25T18:26:40.963 INFO:teuthology.orchestra.run.smithi115.stderr:1: v1:172.21.15.115:6789/0 mon.noname-c 2024-03-25T18:26:40.963 INFO:teuthology.orchestra.run.smithi115.stderr:2: v1:172.21.15.1:6790/0 mon.noname-b 2024-03-25T18:26:40.963 INFO:teuthology.orchestra.run.smithi115.stderr: 2024-03-25T18:26:40.964 INFO:teuthology.orchestra.run.smithi115.stderr:2024-03-25T18:26:40.961+0000 7fc56b3af640 1 Processor -- start 2024-03-25T18:26:40.964 INFO:teuthology.orchestra.run.smithi115.stderr:2024-03-25T18:26:40.962+0000 7fc56b3af640 1 -- start start 2024-03-25T18:26:40.964 INFO:teuthology.orchestra.run.smithi115.stderr:2024-03-25T18:26:40.962+0000 7fc56b3af640 10 monclient: init 2024-03-25T18:26:40.964 INFO:teuthology.orchestra.run.smithi115.stderr:2024-03-25T18:26:40.962+0000 7fc56b3af640 10 monclient: _reopen_session rank -1 2024-03-25T18:26:40.964 INFO:teuthology.orchestra.run.smithi115.stderr:2024-03-25T18:26:40.962+0000 7fc56b3af640 10 monclient: _add_conns ranks=[0,1,2] 2024-03-25T18:26:40.964 INFO:teuthology.orchestra.run.smithi115.stderr:2024-03-25T18:26:40.962+0000 7fc56b3af640 10 monclient(hunting): picked mon.noname-a con 0x7fc564070800 addr v1:172.21.15.1:6789/0 2024-03-25T18:26:40.964 INFO:teuthology.orchestra.run.smithi115.stderr:2024-03-25T18:26:40.962+0000 7fc56b3af640 10 monclient(hunting): picked mon.noname-c con 0x7fc5640719f0 addr v1:172.21.15.115:6789/0 2024-03-25T18:26:40.964 INFO:teuthology.orchestra.run.smithi115.stderr:2024-03-25T18:26:40.962+0000 7fc56b3af640 10 monclient(hunting): picked mon.noname-b con 0x7fc564070be0 addr v1:172.21.15.1:6790/0 2024-03-25T18:26:40.965 INFO:teuthology.orchestra.run.smithi115.stderr:2024-03-25T18:26:40.962+0000 7fc56b3af640 1 -- --> v1:172.21.15.1:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc5640715a0 con 0x7fc564070800 2024-03-25T18:26:40.965 INFO:teuthology.orchestra.run.smithi115.stderr:2024-03-25T18:26:40.962+0000 7fc56b3af640 1 -- --> v1:172.21.15.115:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc564143aa0 con 0x7fc5640719f0 2024-03-25T18:26:40.965 INFO:teuthology.orchestra.run.smithi115.stderr:2024-03-25T18:26:40.962+0000 7fc56b3af640 1 -- --> v1:172.21.15.1:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc5640b07e0 con 0x7fc564070be0 2024-03-25T18:26:40.965 INFO:teuthology.orchestra.run.smithi115.stderr:2024-03-25T18:26:40.962+0000 7fc56b3af640 10 monclient(hunting): _renew_subs 2024-03-25T18:26:40.965 INFO:teuthology.orchestra.run.smithi115.stderr:2024-03-25T18:26:40.962+0000 7fc56b3af640 10 monclient(hunting): authenticate will time out at 7492.118480s 2024-03-25T18:26:40.965 INFO:tasks.ceph.mgr.x.smithi115.stderr:2024-03-25T18:26:40.963+0000 7f99dd247040 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2024-03-25T18:26:40.966 INFO:teuthology.orchestra.run.smithi115.stderr:2024-03-25T18:26:40.963+0000 7fc56894a640 1 --1- >> v1:172.21.15.115:6789/0 conn(0x7fc5640719f0 0x7fc564143fc0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.115:6789/0 says I am v1:172.21.15.115:34318/0 (socket says 172.21.15.115:34318) 2024-03-25T18:26:40.966 INFO:teuthology.orchestra.run.smithi115.stderr:2024-03-25T18:26:40.963+0000 7fc56894a640 1 -- 172.21.15.115:0/2358449496 learned_addr learned my addr 172.21.15.115:0/2358449496 (peer_addr_for_me v1:172.21.15.115:0/0) 2024-03-25T18:26:40.967 INFO:teuthology.orchestra.run.smithi115.stderr:2024-03-25T18:26:40.965+0000 7fc55bfff640 1 -- 172.21.15.115:0/2358449496 <== mon.2 v1:172.21.15.1:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 252511293 0 0) 0x7fc5640b07e0 con 0x7fc564070be0 2024-03-25T18:26:40.967 INFO:teuthology.orchestra.run.smithi115.stderr:2024-03-25T18:26:40.965+0000 7fc55bfff640 10 monclient(hunting): _negotiate 2024-03-25T18:26:40.967 INFO:teuthology.orchestra.run.smithi115.stderr:2024-03-25T18:26:40.965+0000 7fc55bfff640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:26:40.967 INFO:teuthology.orchestra.run.smithi115.stderr:2024-03-25T18:26:40.965+0000 7fc55bfff640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:26:40.967 INFO:teuthology.orchestra.run.smithi115.stderr:2024-03-25T18:26:40.965+0000 7fc55bfff640 10 monclient(hunting): my global_id is 4112 2024-03-25T18:26:40.967 INFO:teuthology.orchestra.run.smithi115.stderr:2024-03-25T18:26:40.965+0000 7fc55bfff640 1 -- 172.21.15.115:0/2358449496 --> v1:172.21.15.1:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fc550003520 con 0x7fc564070be0 2024-03-25T18:26:40.967 INFO:teuthology.orchestra.run.smithi115.stderr:2024-03-25T18:26:40.965+0000 7fc55bfff640 1 -- 172.21.15.115:0/2358449496 <== mon.1 v1:172.21.15.115:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1383000262 0 0) 0x7fc564143aa0 con 0x7fc5640719f0 2024-03-25T18:26:40.967 INFO:teuthology.orchestra.run.smithi115.stderr:2024-03-25T18:26:40.965+0000 7fc55bfff640 10 monclient(hunting): _negotiate 2024-03-25T18:26:40.967 INFO:teuthology.orchestra.run.smithi115.stderr:2024-03-25T18:26:40.965+0000 7fc55bfff640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:26:40.967 INFO:teuthology.orchestra.run.smithi115.stderr:2024-03-25T18:26:40.965+0000 7fc55bfff640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:26:40.967 INFO:teuthology.orchestra.run.smithi115.stderr:2024-03-25T18:26:40.965+0000 7fc55bfff640 10 monclient(hunting): my global_id is 4111 2024-03-25T18:26:40.967 INFO:teuthology.orchestra.run.smithi115.stderr:2024-03-25T18:26:40.965+0000 7fc55bfff640 1 -- 172.21.15.115:0/2358449496 --> v1:172.21.15.115:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fc5640b07e0 con 0x7fc5640719f0 2024-03-25T18:26:40.967 INFO:teuthology.orchestra.run.smithi115.stderr:2024-03-25T18:26:40.966+0000 7fc55bfff640 1 -- 172.21.15.115:0/2358449496 <== mon.0 v1:172.21.15.1:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2533917221 0 0) 0x7fc5640715a0 con 0x7fc564070800 2024-03-25T18:26:40.968 INFO:teuthology.orchestra.run.smithi115.stderr:2024-03-25T18:26:40.966+0000 7fc55bfff640 10 monclient(hunting): _negotiate 2024-03-25T18:26:40.968 INFO:teuthology.orchestra.run.smithi115.stderr:2024-03-25T18:26:40.966+0000 7fc55bfff640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:26:40.968 INFO:teuthology.orchestra.run.smithi115.stderr:2024-03-25T18:26:40.966+0000 7fc55bfff640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:26:40.968 INFO:teuthology.orchestra.run.smithi115.stderr:2024-03-25T18:26:40.966+0000 7fc55bfff640 10 monclient(hunting): my global_id is 4113 2024-03-25T18:26:40.968 INFO:teuthology.orchestra.run.smithi115.stderr:2024-03-25T18:26:40.966+0000 7fc55bfff640 1 -- 172.21.15.115:0/2358449496 --> v1:172.21.15.1:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fc564143aa0 con 0x7fc564070800 2024-03-25T18:26:40.970 INFO:teuthology.orchestra.run.smithi115.stderr:2024-03-25T18:26:40.969+0000 7fc55bfff640 1 -- 172.21.15.115:0/2358449496 <== mon.2 v1:172.21.15.1:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1787394823 0 0) 0x7fc550003520 con 0x7fc564070be0 2024-03-25T18:26:40.970 INFO:teuthology.orchestra.run.smithi115.stderr:2024-03-25T18:26:40.969+0000 7fc55bfff640 1 -- 172.21.15.115:0/2358449496 >> v1:172.21.15.115:6789/0 conn(0x7fc5640719f0 legacy=0x7fc564143fc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:26:40.970 INFO:teuthology.orchestra.run.smithi115.stderr:2024-03-25T18:26:40.969+0000 7fc55bfff640 1 -- 172.21.15.115:0/2358449496 >> v1:172.21.15.1:6789/0 conn(0x7fc564070800 legacy=0x7fc56413d780 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:26:40.970 INFO:teuthology.orchestra.run.smithi115.stderr:2024-03-25T18:26:40.969+0000 7fc55bfff640 10 monclient: _finish_hunting 0 2024-03-25T18:26:40.970 INFO:teuthology.orchestra.run.smithi115.stderr:2024-03-25T18:26:40.969+0000 7fc55bfff640 1 monclient: found mon.noname-b 2024-03-25T18:26:40.971 INFO:teuthology.orchestra.run.smithi115.stderr:2024-03-25T18:26:40.969+0000 7fc55bfff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:26:40.971 INFO:teuthology.orchestra.run.smithi115.stderr:2024-03-25T18:26:40.969+0000 7fc55bfff640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.1:6790/0 2024-03-25T18:26:40.971 INFO:teuthology.orchestra.run.smithi115.stderr:2024-03-25T18:26:40.969+0000 7fc55bfff640 1 -- 172.21.15.115:0/2358449496 --> v1:172.21.15.1:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc5640b0970 con 0x7fc564070be0 2024-03-25T18:26:40.971 INFO:teuthology.orchestra.run.smithi115.stderr:2024-03-25T18:26:40.969+0000 7fc55bfff640 10 monclient: _finish_auth 0 2024-03-25T18:26:40.971 INFO:teuthology.orchestra.run.smithi115.stderr:2024-03-25T18:26:40.969+0000 7fc55bfff640 10 monclient: _check_auth_tickets 2024-03-25T18:26:40.971 INFO:teuthology.orchestra.run.smithi115.stderr:2024-03-25T18:26:40.969+0000 7fc55bfff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:26:40.971 INFO:teuthology.orchestra.run.smithi115.stderr:2024-03-25T18:26:40.969+0000 7fc55bfff640 1 -- 172.21.15.115:0/2358449496 <== mon.2 v1:172.21.15.1:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1586965025 0 0) 0x7fc55c001a10 con 0x7fc564070be0 2024-03-25T18:26:40.971 INFO:teuthology.orchestra.run.smithi115.stderr:2024-03-25T18:26:40.969+0000 7fc56b3af640 5 monclient: authenticate success, global_id 4112 2024-03-25T18:26:40.971 INFO:teuthology.orchestra.run.smithi115.stderr:2024-03-25T18:26:40.969+0000 7fc56b3af640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T18:26:40.971 INFO:teuthology.orchestra.run.smithi115.stderr:2024-03-25T18:26:40.969+0000 7fc55bfff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:26:40.971 INFO:teuthology.orchestra.run.smithi115.stderr:2024-03-25T18:26:40.969+0000 7fc55bfff640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-03-25T18:26:40.971 INFO:teuthology.orchestra.run.smithi115.stderr:2024-03-25T18:26:40.969+0000 7fc55bfff640 10 monclient: dump: 2024-03-25T18:26:40.971 INFO:teuthology.orchestra.run.smithi115.stderr:epoch 1 2024-03-25T18:26:40.972 INFO:teuthology.orchestra.run.smithi115.stderr:fsid c8ea72cf-9e99-4383-9b52-6c23d365fdfc 2024-03-25T18:26:40.972 INFO:teuthology.orchestra.run.smithi115.stderr:last_changed 2024-03-25T18:25:50.510519+0000 2024-03-25T18:26:40.972 INFO:teuthology.orchestra.run.smithi115.stderr:created 2024-03-25T18:25:50.510519+0000 2024-03-25T18:26:40.972 INFO:teuthology.orchestra.run.smithi115.stderr:min_mon_release 19 (squid) 2024-03-25T18:26:40.972 INFO:teuthology.orchestra.run.smithi115.stderr:election_strategy: 3 2024-03-25T18:26:40.972 INFO:teuthology.orchestra.run.smithi115.stderr:0: v1:172.21.15.1:6789/0 mon.a 2024-03-25T18:26:40.972 INFO:teuthology.orchestra.run.smithi115.stderr:1: v1:172.21.15.115:6789/0 mon.b 2024-03-25T18:26:40.972 INFO:teuthology.orchestra.run.smithi115.stderr:2: v1:172.21.15.1:6790/0 mon.c 2024-03-25T18:26:40.973 INFO:teuthology.orchestra.run.smithi115.stderr: 2024-03-25T18:26:40.973 INFO:teuthology.orchestra.run.smithi115.stderr:2024-03-25T18:26:40.969+0000 7fc56b3af640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T18:26:40.974 INFO:teuthology.orchestra.run.smithi115.stderr:2024-03-25T18:26:40.971+0000 7fc55bfff640 1 -- 172.21.15.115:0/2358449496 <== mon.2 v1:172.21.15.1:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fc55c001b70 con 0x7fc564070be0 2024-03-25T18:26:40.974 INFO:teuthology.orchestra.run.smithi115.stderr:2024-03-25T18:26:40.971+0000 7fc55bfff640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:26:40.974 INFO:teuthology.orchestra.run.smithi115.stderr:2024-03-25T18:26:40.971+0000 7fc55bfff640 1 -- 172.21.15.115:0/2358449496 <== mon.2 v1:172.21.15.1:6790/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1586965025 0 0) 0x7fc55c001f00 con 0x7fc564070be0 2024-03-25T18:26:40.974 INFO:teuthology.orchestra.run.smithi115.stderr:2024-03-25T18:26:40.971+0000 7fc55bfff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:26:40.974 INFO:teuthology.orchestra.run.smithi115.stderr:2024-03-25T18:26:40.971+0000 7fc55bfff640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-25T18:26:40.974 INFO:teuthology.orchestra.run.smithi115.stderr:2024-03-25T18:26:40.971+0000 7fc55bfff640 10 monclient: dump: 2024-03-25T18:26:40.974 INFO:teuthology.orchestra.run.smithi115.stderr:epoch 1 2024-03-25T18:26:40.974 INFO:teuthology.orchestra.run.smithi115.stderr:fsid c8ea72cf-9e99-4383-9b52-6c23d365fdfc 2024-03-25T18:26:40.974 INFO:teuthology.orchestra.run.smithi115.stderr:last_changed 2024-03-25T18:25:50.510519+0000 2024-03-25T18:26:40.974 INFO:teuthology.orchestra.run.smithi115.stderr:created 2024-03-25T18:25:50.510519+0000 2024-03-25T18:26:40.974 INFO:teuthology.orchestra.run.smithi115.stderr:min_mon_release 19 (squid) 2024-03-25T18:26:40.974 INFO:teuthology.orchestra.run.smithi115.stderr:election_strategy: 3 2024-03-25T18:26:40.975 INFO:teuthology.orchestra.run.smithi115.stderr:0: v1:172.21.15.1:6789/0 mon.a 2024-03-25T18:26:40.975 INFO:teuthology.orchestra.run.smithi115.stderr:1: v1:172.21.15.115:6789/0 mon.b 2024-03-25T18:26:40.975 INFO:teuthology.orchestra.run.smithi115.stderr:2: v1:172.21.15.1:6790/0 mon.c 2024-03-25T18:26:40.975 INFO:teuthology.orchestra.run.smithi115.stderr: 2024-03-25T18:26:40.975 INFO:teuthology.orchestra.run.smithi115.stderr:2024-03-25T18:26:40.971+0000 7fc56b3af640 10 monclient: get_monmap_and_config success 2024-03-25T18:26:40.975 INFO:teuthology.orchestra.run.smithi115.stderr:2024-03-25T18:26:40.971+0000 7fc56b3af640 4 set_mon_vals no callback set 2024-03-25T18:26:40.975 INFO:teuthology.orchestra.run.smithi115.stderr:2024-03-25T18:26:40.972+0000 7fc56b3af640 10 monclient: shutdown 2024-03-25T18:26:40.975 INFO:teuthology.orchestra.run.smithi115.stderr:2024-03-25T18:26:40.972+0000 7fc56b3af640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:26:40.975 INFO:teuthology.orchestra.run.smithi115.stderr:2024-03-25T18:26:40.972+0000 7fc56b3af640 1 -- 172.21.15.115:0/2358449496 >> v1:172.21.15.1:6790/0 conn(0x7fc564070be0 legacy=0x7fc56414bd10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:26:40.975 INFO:teuthology.orchestra.run.smithi115.stderr:2024-03-25T18:26:40.972+0000 7fc56b3af640 1 -- 172.21.15.115:0/2358449496 shutdown_connections 2024-03-25T18:26:40.975 INFO:teuthology.orchestra.run.smithi115.stderr:2024-03-25T18:26:40.972+0000 7fc56b3af640 1 -- 172.21.15.115:0/2358449496 >> 172.21.15.115:0/2358449496 conn(0x7fc5640b5de0 msgr2=0x7fc5640b61f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:26:40.975 INFO:teuthology.orchestra.run.smithi115.stderr:2024-03-25T18:26:40.973+0000 7fc56b3af640 1 -- 172.21.15.115:0/2358449496 shutdown_connections 2024-03-25T18:26:40.975 INFO:teuthology.orchestra.run.smithi115.stderr:2024-03-25T18:26:40.973+0000 7fc56b3af640 1 -- 172.21.15.115:0/2358449496 wait complete. 2024-03-25T18:26:40.977 INFO:teuthology.orchestra.run.smithi115.stderr:2024-03-25T18:26:40.974+0000 7fc56b3af640 10 monclient: build_initial_monmap 2024-03-25T18:26:40.977 INFO:teuthology.orchestra.run.smithi115.stderr:2024-03-25T18:26:40.974+0000 7fc56b3af640 10 monclient: monmap: 2024-03-25T18:26:40.977 INFO:teuthology.orchestra.run.smithi115.stderr:epoch 0 2024-03-25T18:26:40.978 INFO:teuthology.orchestra.run.smithi115.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T18:26:40.978 INFO:teuthology.orchestra.run.smithi115.stderr:last_changed 0.000000 2024-03-25T18:26:40.978 INFO:teuthology.orchestra.run.smithi115.stderr:created 0.000000 2024-03-25T18:26:40.978 INFO:teuthology.orchestra.run.smithi115.stderr:min_mon_release 0 (unknown) 2024-03-25T18:26:40.978 INFO:teuthology.orchestra.run.smithi115.stderr:election_strategy: 1 2024-03-25T18:26:40.978 INFO:teuthology.orchestra.run.smithi115.stderr:0: v1:172.21.15.1:6789/0 mon.noname-a 2024-03-25T18:26:40.978 INFO:teuthology.orchestra.run.smithi115.stderr:1: v1:172.21.15.115:6789/0 mon.noname-b 2024-03-25T18:26:40.978 INFO:teuthology.orchestra.run.smithi115.stderr:2: v1:172.21.15.1:6790/0 mon.noname-c 2024-03-25T18:26:40.978 INFO:teuthology.orchestra.run.smithi115.stderr: 2024-03-25T18:26:40.978 INFO:teuthology.orchestra.run.smithi115.stderr:2024-03-25T18:26:40.974+0000 7fc56b3af640 1 librados: starting msgr at 2024-03-25T18:26:40.978 INFO:teuthology.orchestra.run.smithi115.stderr:2024-03-25T18:26:40.974+0000 7fc56b3af640 1 librados: starting objecter 2024-03-25T18:26:40.978 INFO:teuthology.orchestra.run.smithi115.stderr:2024-03-25T18:26:40.974+0000 7fc56b3af640 1 Processor -- start 2024-03-25T18:26:40.978 INFO:teuthology.orchestra.run.smithi115.stderr:2024-03-25T18:26:40.974+0000 7fc56b3af640 1 -- start start 2024-03-25T18:26:40.979 INFO:teuthology.orchestra.run.smithi115.stderr:2024-03-25T18:26:40.974+0000 7fc56b3af640 1 librados: setting wanted keys 2024-03-25T18:26:40.979 INFO:teuthology.orchestra.run.smithi115.stderr:2024-03-25T18:26:40.974+0000 7fc56b3af640 1 librados: calling monclient init 2024-03-25T18:26:40.979 INFO:teuthology.orchestra.run.smithi115.stderr:2024-03-25T18:26:40.974+0000 7fc56b3af640 10 monclient: init 2024-03-25T18:26:40.979 INFO:teuthology.orchestra.run.smithi115.stderr:2024-03-25T18:26:40.975+0000 7fc56b3af640 10 monclient: _reopen_session rank -1 2024-03-25T18:26:40.979 INFO:teuthology.orchestra.run.smithi115.stderr:2024-03-25T18:26:40.975+0000 7fc56b3af640 10 monclient: _add_conns ranks=[0,1,2] 2024-03-25T18:26:40.979 INFO:teuthology.orchestra.run.smithi115.stderr:2024-03-25T18:26:40.975+0000 7fc56b3af640 10 monclient(hunting): picked mon.noname-a con 0x7fc564070800 addr v1:172.21.15.1:6789/0 2024-03-25T18:26:40.979 INFO:teuthology.orchestra.run.smithi115.stderr:2024-03-25T18:26:40.975+0000 7fc56b3af640 10 monclient(hunting): picked mon.noname-b con 0x7fc564070be0 addr v1:172.21.15.115:6789/0 2024-03-25T18:26:40.979 INFO:teuthology.orchestra.run.smithi115.stderr:2024-03-25T18:26:40.975+0000 7fc56b3af640 10 monclient(hunting): picked mon.noname-c con 0x7fc5640719f0 addr v1:172.21.15.1:6790/0 2024-03-25T18:26:40.979 INFO:teuthology.orchestra.run.smithi115.stderr:2024-03-25T18:26:40.975+0000 7fc56b3af640 1 -- --> v1:172.21.15.1:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc564132cf0 con 0x7fc564070800 2024-03-25T18:26:40.979 INFO:teuthology.orchestra.run.smithi115.stderr:2024-03-25T18:26:40.975+0000 7fc56b3af640 1 -- --> v1:172.21.15.115:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc56412e590 con 0x7fc564070be0 2024-03-25T18:26:40.979 INFO:teuthology.orchestra.run.smithi115.stderr:2024-03-25T18:26:40.975+0000 7fc56b3af640 1 -- --> v1:172.21.15.1:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc56412e720 con 0x7fc5640719f0 2024-03-25T18:26:40.980 INFO:teuthology.orchestra.run.smithi115.stderr:2024-03-25T18:26:40.975+0000 7fc56b3af640 10 monclient(hunting): _renew_subs 2024-03-25T18:26:40.980 INFO:teuthology.orchestra.run.smithi115.stderr:2024-03-25T18:26:40.975+0000 7fc56b3af640 10 monclient(hunting): authenticate will time out at 7492.131455s 2024-03-25T18:26:40.980 INFO:teuthology.orchestra.run.smithi115.stderr:2024-03-25T18:26:40.976+0000 7fc56914b640 1 --1- >> v1:172.21.15.1:6789/0 conn(0x7fc564070800 0x7fc564132490 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.1:6789/0 says I am v1:172.21.15.115:33076/0 (socket says 172.21.15.115:33076) 2024-03-25T18:26:40.980 INFO:teuthology.orchestra.run.smithi115.stderr:2024-03-25T18:26:40.976+0000 7fc56914b640 1 -- 172.21.15.115:0/2365027004 learned_addr learned my addr 172.21.15.115:0/2365027004 (peer_addr_for_me v1:172.21.15.115:0/0) 2024-03-25T18:26:40.980 INFO:teuthology.orchestra.run.smithi115.stderr:2024-03-25T18:26:40.976+0000 7fc559ffb640 10 client.?.objecter ms_handle_connect 0x7fc564070800 2024-03-25T18:26:40.980 INFO:teuthology.orchestra.run.smithi115.stderr:2024-03-25T18:26:40.976+0000 7fc559ffb640 10 client.?.objecter resend_mon_ops 2024-03-25T18:26:40.980 INFO:teuthology.orchestra.run.smithi115.stderr:2024-03-25T18:26:40.976+0000 7fc559ffb640 10 client.?.objecter ms_handle_connect 0x7fc5640719f0 2024-03-25T18:26:40.980 INFO:teuthology.orchestra.run.smithi115.stderr:2024-03-25T18:26:40.976+0000 7fc559ffb640 10 client.?.objecter resend_mon_ops 2024-03-25T18:26:40.981 INFO:teuthology.orchestra.run.smithi115.stderr:2024-03-25T18:26:40.978+0000 7fc559ffb640 1 -- 172.21.15.115:0/2365027004 <== mon.2 v1:172.21.15.1:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3953458507 0 0) 0x7fc56412e720 con 0x7fc5640719f0 2024-03-25T18:26:40.981 INFO:teuthology.orchestra.run.smithi115.stderr:2024-03-25T18:26:40.978+0000 7fc559ffb640 10 monclient(hunting): _negotiate 2024-03-25T18:26:40.981 INFO:teuthology.orchestra.run.smithi115.stderr:2024-03-25T18:26:40.978+0000 7fc559ffb640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:26:40.981 INFO:teuthology.orchestra.run.smithi115.stderr:2024-03-25T18:26:40.978+0000 7fc559ffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:26:40.981 INFO:teuthology.orchestra.run.smithi115.stderr:2024-03-25T18:26:40.978+0000 7fc559ffb640 10 monclient(hunting): my global_id is 4115 2024-03-25T18:26:40.981 INFO:teuthology.orchestra.run.smithi115.stderr:2024-03-25T18:26:40.978+0000 7fc559ffb640 1 -- 172.21.15.115:0/2365027004 --> v1:172.21.15.1:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fc548004eb0 con 0x7fc5640719f0 2024-03-25T18:26:40.981 INFO:teuthology.orchestra.run.smithi115.stderr:2024-03-25T18:26:40.978+0000 7fc559ffb640 1 -- 172.21.15.115:0/2365027004 <== mon.0 v1:172.21.15.1:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 705482121 0 0) 0x7fc564132cf0 con 0x7fc564070800 2024-03-25T18:26:40.982 INFO:teuthology.orchestra.run.smithi115.stderr:2024-03-25T18:26:40.978+0000 7fc559ffb640 10 monclient(hunting): _negotiate 2024-03-25T18:26:40.982 INFO:teuthology.orchestra.run.smithi115.stderr:2024-03-25T18:26:40.978+0000 7fc559ffb640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:26:40.982 INFO:teuthology.orchestra.run.smithi115.stderr:2024-03-25T18:26:40.978+0000 7fc559ffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:26:40.982 INFO:teuthology.orchestra.run.smithi115.stderr:2024-03-25T18:26:40.978+0000 7fc559ffb640 10 monclient(hunting): my global_id is 4116 2024-03-25T18:26:40.982 INFO:teuthology.orchestra.run.smithi115.stderr:2024-03-25T18:26:40.978+0000 7fc559ffb640 1 -- 172.21.15.115:0/2365027004 --> v1:172.21.15.1:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fc56412e720 con 0x7fc564070800 2024-03-25T18:26:40.984 INFO:teuthology.orchestra.run.smithi115.stderr:2024-03-25T18:26:40.984+0000 7fc559ffb640 1 -- 172.21.15.115:0/2365027004 <== mon.0 v1:172.21.15.1:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3370364417 0 0) 0x7fc56412e720 con 0x7fc564070800 2024-03-25T18:26:40.985 INFO:teuthology.orchestra.run.smithi115.stderr:2024-03-25T18:26:40.984+0000 7fc559ffb640 1 -- 172.21.15.115:0/2365027004 --> v1:172.21.15.1:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fc564132cf0 con 0x7fc564070800 2024-03-25T18:26:40.985 INFO:teuthology.orchestra.run.smithi115.stderr:2024-03-25T18:26:40.984+0000 7fc559ffb640 1 -- 172.21.15.115:0/2365027004 <== mon.0 v1:172.21.15.1:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1586965025 0 0) 0x7fc5600019f0 con 0x7fc564070800 2024-03-25T18:26:40.985 INFO:teuthology.orchestra.run.smithi115.stderr:2024-03-25T18:26:40.984+0000 7fc559ffb640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:26:40.985 INFO:teuthology.orchestra.run.smithi115.stderr:2024-03-25T18:26:40.984+0000 7fc559ffb640 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2024-03-25T18:26:40.985 INFO:teuthology.orchestra.run.smithi115.stderr:2024-03-25T18:26:40.984+0000 7fc559ffb640 10 monclient(hunting): dump: 2024-03-25T18:26:40.985 INFO:teuthology.orchestra.run.smithi115.stderr:epoch 1 2024-03-25T18:26:40.985 INFO:teuthology.orchestra.run.smithi115.stderr:fsid c8ea72cf-9e99-4383-9b52-6c23d365fdfc 2024-03-25T18:26:40.985 INFO:teuthology.orchestra.run.smithi115.stderr:last_changed 2024-03-25T18:25:50.510519+0000 2024-03-25T18:26:40.985 INFO:teuthology.orchestra.run.smithi115.stderr:created 2024-03-25T18:25:50.510519+0000 2024-03-25T18:26:40.985 INFO:teuthology.orchestra.run.smithi115.stderr:min_mon_release 19 (squid) 2024-03-25T18:26:40.985 INFO:teuthology.orchestra.run.smithi115.stderr:election_strategy: 3 2024-03-25T18:26:40.985 INFO:teuthology.orchestra.run.smithi115.stderr:0: v1:172.21.15.1:6789/0 mon.a 2024-03-25T18:26:40.985 INFO:teuthology.orchestra.run.smithi115.stderr:1: v1:172.21.15.115:6789/0 mon.b 2024-03-25T18:26:40.986 INFO:teuthology.orchestra.run.smithi115.stderr:2: v1:172.21.15.1:6790/0 mon.c 2024-03-25T18:26:40.986 INFO:teuthology.orchestra.run.smithi115.stderr: 2024-03-25T18:26:40.986 INFO:teuthology.orchestra.run.smithi115.stderr:2024-03-25T18:26:40.984+0000 7fc559ffb640 10 monclient(hunting): _finish_auth 0 2024-03-25T18:26:40.990 INFO:teuthology.orchestra.run.smithi115.stderr:2024-03-25T18:26:40.988+0000 7fc559ffb640 1 -- 172.21.15.115:0/2365027004 <== mon.0 v1:172.21.15.1:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 1212105730 0 0) 0x7fc564132cf0 con 0x7fc564070800 2024-03-25T18:26:40.990 INFO:teuthology.orchestra.run.smithi115.stderr:2024-03-25T18:26:40.989+0000 7fc559ffb640 1 -- 172.21.15.115:0/2365027004 >> v1:172.21.15.1:6790/0 conn(0x7fc5640719f0 legacy=0x7fc56412de80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:26:40.990 INFO:teuthology.orchestra.run.smithi115.stderr:2024-03-25T18:26:40.989+0000 7fc559ffb640 1 -- 172.21.15.115:0/2365027004 >> v1:172.21.15.115:6789/0 conn(0x7fc564070be0 legacy=0x7fc564131c30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:26:40.990 INFO:teuthology.orchestra.run.smithi115.stderr:2024-03-25T18:26:40.989+0000 7fc559ffb640 10 monclient: _finish_hunting 0 2024-03-25T18:26:40.990 INFO:teuthology.orchestra.run.smithi115.stderr:2024-03-25T18:26:40.989+0000 7fc559ffb640 1 monclient: found mon.a 2024-03-25T18:26:40.990 INFO:teuthology.orchestra.run.smithi115.stderr:2024-03-25T18:26:40.989+0000 7fc559ffb640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:26:40.990 INFO:teuthology.orchestra.run.smithi115.stderr:2024-03-25T18:26:40.989+0000 7fc559ffb640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.1:6789/0 2024-03-25T18:26:40.991 INFO:teuthology.orchestra.run.smithi115.stderr:2024-03-25T18:26:40.989+0000 7fc559ffb640 1 -- 172.21.15.115:0/2365027004 --> v1:172.21.15.1:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc56412e8b0 con 0x7fc564070800 2024-03-25T18:26:40.991 INFO:teuthology.orchestra.run.smithi115.stderr:2024-03-25T18:26:40.989+0000 7fc559ffb640 10 monclient: _finish_auth 0 2024-03-25T18:26:40.991 INFO:teuthology.orchestra.run.smithi115.stderr:2024-03-25T18:26:40.989+0000 7fc559ffb640 10 monclient: _check_auth_tickets 2024-03-25T18:26:40.991 INFO:teuthology.orchestra.run.smithi115.stderr:2024-03-25T18:26:40.989+0000 7fc559ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:26:40.991 INFO:teuthology.orchestra.run.smithi115.stderr:2024-03-25T18:26:40.989+0000 7fc56b3af640 5 monclient: authenticate success, global_id 4116 2024-03-25T18:26:40.991 INFO:teuthology.orchestra.run.smithi115.stderr:2024-03-25T18:26:40.989+0000 7fc56b3af640 10 monclient: _renew_subs 2024-03-25T18:26:40.991 INFO:teuthology.orchestra.run.smithi115.stderr:2024-03-25T18:26:40.989+0000 7fc56b3af640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.1:6789/0 2024-03-25T18:26:40.991 INFO:teuthology.orchestra.run.smithi115.stderr:2024-03-25T18:26:40.989+0000 7fc56b3af640 1 -- 172.21.15.115:0/2365027004 --> v1:172.21.15.1:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc5641d55d0 con 0x7fc564070800 2024-03-25T18:26:40.991 INFO:teuthology.orchestra.run.smithi115.stderr:2024-03-25T18:26:40.989+0000 7fc56b3af640 10 client.4116.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T18:26:40.991 INFO:teuthology.orchestra.run.smithi115.stderr:2024-03-25T18:26:40.989+0000 7fc56b3af640 10 monclient: _renew_subs 2024-03-25T18:26:40.991 INFO:teuthology.orchestra.run.smithi115.stderr:2024-03-25T18:26:40.989+0000 7fc56b3af640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.1:6789/0 2024-03-25T18:26:40.991 INFO:teuthology.orchestra.run.smithi115.stderr:2024-03-25T18:26:40.989+0000 7fc56b3af640 1 -- 172.21.15.115:0/2365027004 --> v1:172.21.15.1:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7fc5641d5b00 con 0x7fc564070800 2024-03-25T18:26:40.991 INFO:teuthology.orchestra.run.smithi115.stderr:2024-03-25T18:26:40.989+0000 7fc56b3af640 1 librados: init done 2024-03-25T18:26:40.992 INFO:teuthology.orchestra.run.smithi115.stderr:2024-03-25T18:26:40.990+0000 7fc56b3af640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T18:26:40.992 INFO:teuthology.orchestra.run.smithi115.stderr:2024-03-25T18:26:40.990+0000 7fc56b3af640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:26:40.992 INFO:teuthology.orchestra.run.smithi115.stderr:2024-03-25T18:26:40.990+0000 7fc56b3af640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.1:6789/0 2024-03-25T18:26:40.992 INFO:teuthology.orchestra.run.smithi115.stderr:2024-03-25T18:26:40.990+0000 7fc56b3af640 1 -- 172.21.15.115:0/2365027004 --> v1:172.21.15.1:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc5640b3e40 con 0x7fc564070800 2024-03-25T18:26:41.011 INFO:tasks.ceph.mgr.x.smithi115.stderr:2024-03-25T18:26:41.010+0000 7f99dd247040 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2024-03-25T18:26:41.074 INFO:tasks.ceph.mgr.x.smithi115.stderr:2024-03-25T18:26:41.073+0000 7f99dd247040 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2024-03-25T18:26:41.093 INFO:teuthology.orchestra.run.smithi115.stderr:2024-03-25T18:26:41.092+0000 7fc559ffb640 1 -- 172.21.15.115:0/2365027004 <== mon.0 v1:172.21.15.1:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fc560003610 con 0x7fc564070800 2024-03-25T18:26:41.093 INFO:teuthology.orchestra.run.smithi115.stderr:2024-03-25T18:26:41.092+0000 7fc559ffb640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:26:41.094 INFO:teuthology.orchestra.run.smithi115.stderr:2024-03-25T18:26:41.092+0000 7fc559ffb640 1 -- 172.21.15.115:0/2365027004 <== mon.0 v1:172.21.15.1:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1586965025 0 0) 0x7fc5600039e0 con 0x7fc564070800 2024-03-25T18:26:41.094 INFO:teuthology.orchestra.run.smithi115.stderr:2024-03-25T18:26:41.092+0000 7fc559ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:26:41.094 INFO:teuthology.orchestra.run.smithi115.stderr:2024-03-25T18:26:41.092+0000 7fc559ffb640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-25T18:26:41.094 INFO:teuthology.orchestra.run.smithi115.stderr:2024-03-25T18:26:41.092+0000 7fc559ffb640 10 monclient: dump: 2024-03-25T18:26:41.094 INFO:teuthology.orchestra.run.smithi115.stderr:epoch 1 2024-03-25T18:26:41.094 INFO:teuthology.orchestra.run.smithi115.stderr:fsid c8ea72cf-9e99-4383-9b52-6c23d365fdfc 2024-03-25T18:26:41.094 INFO:teuthology.orchestra.run.smithi115.stderr:last_changed 2024-03-25T18:25:50.510519+0000 2024-03-25T18:26:41.094 INFO:teuthology.orchestra.run.smithi115.stderr:created 2024-03-25T18:25:50.510519+0000 2024-03-25T18:26:41.094 INFO:teuthology.orchestra.run.smithi115.stderr:min_mon_release 19 (squid) 2024-03-25T18:26:41.094 INFO:teuthology.orchestra.run.smithi115.stderr:election_strategy: 3 2024-03-25T18:26:41.094 INFO:teuthology.orchestra.run.smithi115.stderr:0: v1:172.21.15.1:6789/0 mon.a 2024-03-25T18:26:41.094 INFO:teuthology.orchestra.run.smithi115.stderr:1: v1:172.21.15.115:6789/0 mon.b 2024-03-25T18:26:41.094 INFO:teuthology.orchestra.run.smithi115.stderr:2: v1:172.21.15.1:6790/0 mon.c 2024-03-25T18:26:41.094 INFO:teuthology.orchestra.run.smithi115.stderr: 2024-03-25T18:26:41.095 INFO:teuthology.orchestra.run.smithi115.stderr:2024-03-25T18:26:41.092+0000 7fc559ffb640 1 -- 172.21.15.115:0/2365027004 <== mon.0 v1:172.21.15.1:6789/0 7 ==== mgrmap(e 1) v1 ==== 807+0+0 (unknown 3189685207 0 0) 0x7fc560003f90 con 0x7fc564070800 2024-03-25T18:26:41.095 INFO:teuthology.orchestra.run.smithi115.stderr:2024-03-25T18:26:41.092+0000 7fc55b7fe640 4 set_mon_vals no callback set 2024-03-25T18:26:41.095 INFO:teuthology.orchestra.run.smithi115.stderr:2024-03-25T18:26:41.093+0000 7fc559ffb640 1 -- 172.21.15.115:0/2365027004 <== mon.0 v1:172.21.15.1:6789/0 8 ==== osd_map(2..2 src has 1..2) v4 ==== 725+0+0 (unknown 2504152564 0 0) 0x7fc560006620 con 0x7fc564070800 2024-03-25T18:26:41.096 INFO:teuthology.orchestra.run.smithi115.stderr:2024-03-25T18:26:41.093+0000 7fc559ffb640 10 client.4116.objecter ms_dispatch 0x7fc564000b90 osd_map(2..2 src has 1..2) v4 2024-03-25T18:26:41.096 INFO:teuthology.orchestra.run.smithi115.stderr:2024-03-25T18:26:41.093+0000 7fc559ffb640 3 client.4116.objecter handle_osd_map got epochs [2,2] > 0 2024-03-25T18:26:41.096 INFO:teuthology.orchestra.run.smithi115.stderr:2024-03-25T18:26:41.093+0000 7fc559ffb640 3 client.4116.objecter handle_osd_map decoding full epoch 2 2024-03-25T18:26:41.096 INFO:teuthology.orchestra.run.smithi115.stderr:2024-03-25T18:26:41.093+0000 7fc559ffb640 20 client.4116.objecter dump_active .. 0 homeless 2024-03-25T18:26:41.096 INFO:teuthology.orchestra.run.smithi115.stderr:2024-03-25T18:26:41.094+0000 7fc559ffb640 1 -- 172.21.15.115:0/2365027004 <== mon.0 v1:172.21.15.1:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+73141 (unknown 1092875540 0 2031569661) 0x7fc560006970 con 0x7fc564070800 2024-03-25T18:26:41.096 INFO:teuthology.orchestra.run.smithi115.stderr:2024-03-25T18:26:41.094+0000 7fc559ffb640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:26:41.096 INFO:teuthology.orchestra.run.smithi115.stderr:2024-03-25T18:26:41.094+0000 7fc559ffb640 10 monclient: _finish_command 1 = system:0 2024-03-25T18:26:41.117 INFO:tasks.ceph.mgr.x.smithi115.stderr:2024-03-25T18:26:41.116+0000 7f99dd247040 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2024-03-25T18:26:41.167 INFO:teuthology.orchestra.run.smithi115.stderr:2024-03-25T18:26:41.166+0000 7fc56b3af640 10 start_mon_command cmd=[{"prefix": "osd new", "uuid": "b9f34922-3a84-49be-8426-807c1eceaefc", "id": 0}] 2024-03-25T18:26:41.167 INFO:teuthology.orchestra.run.smithi115.stderr:2024-03-25T18:26:41.166+0000 7fc56b3af640 10 monclient: _send_command 2 [{"prefix": "osd new", "uuid": "b9f34922-3a84-49be-8426-807c1eceaefc", "id": 0}] 2024-03-25T18:26:41.168 INFO:teuthology.orchestra.run.smithi115.stderr:2024-03-25T18:26:41.166+0000 7fc56b3af640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.1:6789/0 2024-03-25T18:26:41.168 INFO:teuthology.orchestra.run.smithi115.stderr:2024-03-25T18:26:41.166+0000 7fc56b3af640 1 -- 172.21.15.115:0/2365027004 --> v1:172.21.15.1:6789/0 -- mon_command({"prefix": "osd new", "uuid": "b9f34922-3a84-49be-8426-807c1eceaefc", "id": 0} v 0) v1 -- 0x7fc56404f730 con 0x7fc564070800 2024-03-25T18:26:41.235 INFO:tasks.ceph.mgr.x.smithi115.stderr:2024-03-25T18:26:41.234+0000 7f99dd247040 -1 mgr[py] Module rgw has missing NOTIFY_TYPES member 2024-03-25T18:26:41.256 INFO:teuthology.orchestra.run.smithi115.stderr:2024-03-25T18:26:41.255+0000 7fc559ffb640 1 -- 172.21.15.115:0/2365027004 <== mon.0 v1:172.21.15.1:6789/0 10 ==== mon_command_ack([{"prefix": "osd new", "uuid": "b9f34922-3a84-49be-8426-807c1eceaefc", "id": 0}]=0 v3) v1 ==== 112+0+2 (unknown 756026451 0 732707515) 0x7fc560006bb0 con 0x7fc564070800 2024-03-25T18:26:41.256 INFO:teuthology.orchestra.run.smithi115.stderr:2024-03-25T18:26:41.255+0000 7fc559ffb640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd new", "uuid": "b9f34922-3a84-49be-8426-807c1eceaefc", "id": 0}] 2024-03-25T18:26:41.256 INFO:teuthology.orchestra.run.smithi115.stderr:2024-03-25T18:26:41.255+0000 7fc559ffb640 10 monclient: _finish_command 2 = system:0 2024-03-25T18:26:41.256 INFO:teuthology.orchestra.run.smithi115.stdout:0 2024-03-25T18:26:41.257 INFO:teuthology.orchestra.run.smithi115.stderr:2024-03-25T18:26:41.255+0000 7fc56b3af640 10 librados: watch_flush enter 2024-03-25T18:26:41.257 INFO:teuthology.orchestra.run.smithi115.stderr:2024-03-25T18:26:41.256+0000 7fc56b3af640 10 librados: watch_flush exit 2024-03-25T18:26:41.258 INFO:teuthology.orchestra.run.smithi115.stderr:2024-03-25T18:26:41.256+0000 7fc56b3af640 20 client.4116.objecter shutdown clearing up homeless session... 2024-03-25T18:26:41.258 INFO:teuthology.orchestra.run.smithi115.stderr:2024-03-25T18:26:41.256+0000 7fc56b3af640 10 client.4116.objecter successfully canceled tick 2024-03-25T18:26:41.258 INFO:teuthology.orchestra.run.smithi115.stderr:2024-03-25T18:26:41.256+0000 7fc56b3af640 10 monclient: shutdown 2024-03-25T18:26:41.258 INFO:teuthology.orchestra.run.smithi115.stderr:2024-03-25T18:26:41.256+0000 7fc56b3af640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:26:41.258 INFO:teuthology.orchestra.run.smithi115.stderr:2024-03-25T18:26:41.256+0000 7fc56b3af640 1 -- 172.21.15.115:0/2365027004 >> v1:172.21.15.1:6789/0 conn(0x7fc564070800 legacy=0x7fc564132490 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:26:41.258 INFO:teuthology.orchestra.run.smithi115.stderr:2024-03-25T18:26:41.256+0000 7fc56b3af640 1 -- 172.21.15.115:0/2365027004 shutdown_connections 2024-03-25T18:26:41.258 INFO:teuthology.orchestra.run.smithi115.stderr:2024-03-25T18:26:41.256+0000 7fc56b3af640 1 -- 172.21.15.115:0/2365027004 >> 172.21.15.115:0/2365027004 conn(0x7fc5640b5de0 msgr2=0x7fc5640b5430 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:26:41.258 INFO:teuthology.orchestra.run.smithi115.stderr:2024-03-25T18:26:41.256+0000 7fc56b3af640 1 -- 172.21.15.115:0/2365027004 shutdown_connections 2024-03-25T18:26:41.258 INFO:teuthology.orchestra.run.smithi115.stderr:2024-03-25T18:26:41.256+0000 7fc56b3af640 1 -- 172.21.15.115:0/2365027004 wait complete. 2024-03-25T18:26:41.258 INFO:teuthology.orchestra.run.smithi115.stderr:2024-03-25T18:26:41.256+0000 7fc56b3af640 1 librados: shutdown 2024-03-25T18:26:41.268 DEBUG:teuthology.orchestra.run.smithi115:> sudo ceph --cluster ceph osd new 42bf306f-55e3-4634-9874-8b4b73ac6644 1 2024-03-25T18:26:41.306 INFO:tasks.ceph.mgr.y.smithi001.stderr:2024-03-25T18:26:41.305+0000 7faca7441040 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2024-03-25T18:26:41.310 INFO:tasks.ceph.mgr.x.smithi115.stderr:2024-03-25T18:26:41.309+0000 7f99dd247040 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2024-03-25T18:26:41.358 INFO:tasks.ceph.mgr.x.smithi115.stderr:2024-03-25T18:26:41.356+0000 7f99dd247040 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2024-03-25T18:26:41.362 INFO:tasks.ceph.mgr.y.smithi001.stderr:2024-03-25T18:26:41.361+0000 7faca7441040 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2024-03-25T18:26:41.368 INFO:teuthology.orchestra.run.smithi115.stderr:2024-03-25T18:26:41.367+0000 7f836b24a640 10 monclient: get_monmap_and_config 2024-03-25T18:26:41.368 INFO:teuthology.orchestra.run.smithi115.stderr:2024-03-25T18:26:41.367+0000 7f836b24a640 10 monclient: build_initial_monmap 2024-03-25T18:26:41.368 INFO:teuthology.orchestra.run.smithi115.stderr:2024-03-25T18:26:41.367+0000 7f836b24a640 10 monclient: monmap: 2024-03-25T18:26:41.369 INFO:teuthology.orchestra.run.smithi115.stderr:epoch 0 2024-03-25T18:26:41.369 INFO:teuthology.orchestra.run.smithi115.stderr:fsid c8ea72cf-9e99-4383-9b52-6c23d365fdfc 2024-03-25T18:26:41.369 INFO:teuthology.orchestra.run.smithi115.stderr:last_changed 2024-03-25T18:26:41.368442+0000 2024-03-25T18:26:41.369 INFO:teuthology.orchestra.run.smithi115.stderr:created 2024-03-25T18:26:41.368442+0000 2024-03-25T18:26:41.369 INFO:teuthology.orchestra.run.smithi115.stderr:min_mon_release 0 (unknown) 2024-03-25T18:26:41.369 INFO:teuthology.orchestra.run.smithi115.stderr:election_strategy: 3 2024-03-25T18:26:41.369 INFO:teuthology.orchestra.run.smithi115.stderr:0: v1:172.21.15.1:6789/0 mon.noname-a 2024-03-25T18:26:41.369 INFO:teuthology.orchestra.run.smithi115.stderr:1: v1:172.21.15.115:6789/0 mon.noname-c 2024-03-25T18:26:41.369 INFO:teuthology.orchestra.run.smithi115.stderr:2: v1:172.21.15.1:6790/0 mon.noname-b 2024-03-25T18:26:41.369 INFO:teuthology.orchestra.run.smithi115.stderr: 2024-03-25T18:26:41.371 INFO:teuthology.orchestra.run.smithi115.stderr:2024-03-25T18:26:41.368+0000 7f836b24a640 1 Processor -- start 2024-03-25T18:26:41.371 INFO:teuthology.orchestra.run.smithi115.stderr:2024-03-25T18:26:41.368+0000 7f836b24a640 1 -- start start 2024-03-25T18:26:41.371 INFO:teuthology.orchestra.run.smithi115.stderr:2024-03-25T18:26:41.368+0000 7f836b24a640 10 monclient: init 2024-03-25T18:26:41.371 INFO:teuthology.orchestra.run.smithi115.stderr:2024-03-25T18:26:41.368+0000 7f836b24a640 10 monclient: _reopen_session rank -1 2024-03-25T18:26:41.371 INFO:teuthology.orchestra.run.smithi115.stderr:2024-03-25T18:26:41.368+0000 7f836b24a640 10 monclient: _add_conns ranks=[2,0,1] 2024-03-25T18:26:41.371 INFO:teuthology.orchestra.run.smithi115.stderr:2024-03-25T18:26:41.368+0000 7f836b24a640 10 monclient(hunting): picked mon.noname-b con 0x7f83640b4b50 addr v1:172.21.15.1:6790/0 2024-03-25T18:26:41.371 INFO:teuthology.orchestra.run.smithi115.stderr:2024-03-25T18:26:41.368+0000 7f836b24a640 10 monclient(hunting): picked mon.noname-a con 0x7f836413ff20 addr v1:172.21.15.1:6789/0 2024-03-25T18:26:41.371 INFO:teuthology.orchestra.run.smithi115.stderr:2024-03-25T18:26:41.368+0000 7f836b24a640 10 monclient(hunting): picked mon.noname-c con 0x7f8364142dd0 addr v1:172.21.15.115:6789/0 2024-03-25T18:26:41.371 INFO:teuthology.orchestra.run.smithi115.stderr:2024-03-25T18:26:41.368+0000 7f836b24a640 1 -- --> v1:172.21.15.1:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f83640ab440 con 0x7f836413ff20 2024-03-25T18:26:41.372 INFO:teuthology.orchestra.run.smithi115.stderr:2024-03-25T18:26:41.369+0000 7f836b24a640 1 -- --> v1:172.21.15.115:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f83640ab6b0 con 0x7f8364142dd0 2024-03-25T18:26:41.372 INFO:teuthology.orchestra.run.smithi115.stderr:2024-03-25T18:26:41.369+0000 7f836b24a640 1 -- --> v1:172.21.15.1:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f83640a9f10 con 0x7f83640b4b50 2024-03-25T18:26:41.372 INFO:teuthology.orchestra.run.smithi115.stderr:2024-03-25T18:26:41.369+0000 7f836b24a640 10 monclient(hunting): _renew_subs 2024-03-25T18:26:41.372 INFO:teuthology.orchestra.run.smithi115.stderr:2024-03-25T18:26:41.369+0000 7f836b24a640 10 monclient(hunting): authenticate will time out at 7492.524964s 2024-03-25T18:26:41.372 INFO:teuthology.orchestra.run.smithi115.stderr:2024-03-25T18:26:41.369+0000 7f8368fe6640 1 --1- >> v1:172.21.15.1:6790/0 conn(0x7f83640b4b50 0x7f836413f810 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.1:6790/0 says I am v1:172.21.15.115:50230/0 (socket says 172.21.15.115:50230) 2024-03-25T18:26:41.372 INFO:teuthology.orchestra.run.smithi115.stderr:2024-03-25T18:26:41.369+0000 7f8368fe6640 1 -- 172.21.15.115:0/2679120163 learned_addr learned my addr 172.21.15.115:0/2679120163 (peer_addr_for_me v1:172.21.15.115:0/0) 2024-03-25T18:26:41.373 INFO:teuthology.orchestra.run.smithi115.stderr:2024-03-25T18:26:41.371+0000 7f83637fe640 1 -- 172.21.15.115:0/2679120163 <== mon.2 v1:172.21.15.1:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4202422689 0 0) 0x7f83640a9f10 con 0x7f83640b4b50 2024-03-25T18:26:41.373 INFO:teuthology.orchestra.run.smithi115.stderr:2024-03-25T18:26:41.371+0000 7f83637fe640 10 monclient(hunting): _negotiate 2024-03-25T18:26:41.373 INFO:teuthology.orchestra.run.smithi115.stderr:2024-03-25T18:26:41.371+0000 7f83637fe640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:26:41.373 INFO:teuthology.orchestra.run.smithi115.stderr:2024-03-25T18:26:41.371+0000 7f83637fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:26:41.373 INFO:teuthology.orchestra.run.smithi115.stderr:2024-03-25T18:26:41.371+0000 7f83637fe640 10 monclient(hunting): my global_id is 4118 2024-03-25T18:26:41.373 INFO:teuthology.orchestra.run.smithi115.stderr:2024-03-25T18:26:41.371+0000 7f83637fe640 1 -- 172.21.15.115:0/2679120163 --> v1:172.21.15.1:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f8354003520 con 0x7f83640b4b50 2024-03-25T18:26:41.374 INFO:teuthology.orchestra.run.smithi115.stderr:2024-03-25T18:26:41.372+0000 7f83637fe640 1 -- 172.21.15.115:0/2679120163 <== mon.1 v1:172.21.15.115:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2163182674 0 0) 0x7f83640ab6b0 con 0x7f8364142dd0 2024-03-25T18:26:41.374 INFO:teuthology.orchestra.run.smithi115.stderr:2024-03-25T18:26:41.372+0000 7f83637fe640 10 monclient(hunting): _negotiate 2024-03-25T18:26:41.374 INFO:teuthology.orchestra.run.smithi115.stderr:2024-03-25T18:26:41.372+0000 7f83637fe640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:26:41.374 INFO:teuthology.orchestra.run.smithi115.stderr:2024-03-25T18:26:41.372+0000 7f83637fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:26:41.374 INFO:teuthology.orchestra.run.smithi115.stderr:2024-03-25T18:26:41.372+0000 7f83637fe640 10 monclient(hunting): my global_id is 4114 2024-03-25T18:26:41.374 INFO:teuthology.orchestra.run.smithi115.stderr:2024-03-25T18:26:41.372+0000 7f83637fe640 1 -- 172.21.15.115:0/2679120163 --> v1:172.21.15.115:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f83640a9f10 con 0x7f8364142dd0 2024-03-25T18:26:41.374 INFO:teuthology.orchestra.run.smithi115.stderr:2024-03-25T18:26:41.372+0000 7f83637fe640 1 -- 172.21.15.115:0/2679120163 <== mon.0 v1:172.21.15.1:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2173077027 0 0) 0x7f83640ab440 con 0x7f836413ff20 2024-03-25T18:26:41.374 INFO:teuthology.orchestra.run.smithi115.stderr:2024-03-25T18:26:41.372+0000 7f83637fe640 10 monclient(hunting): _negotiate 2024-03-25T18:26:41.374 INFO:teuthology.orchestra.run.smithi115.stderr:2024-03-25T18:26:41.372+0000 7f83637fe640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:26:41.374 INFO:teuthology.orchestra.run.smithi115.stderr:2024-03-25T18:26:41.372+0000 7f83637fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:26:41.374 INFO:teuthology.orchestra.run.smithi115.stderr:2024-03-25T18:26:41.372+0000 7f83637fe640 10 monclient(hunting): my global_id is 4119 2024-03-25T18:26:41.374 INFO:teuthology.orchestra.run.smithi115.stderr:2024-03-25T18:26:41.372+0000 7f83637fe640 1 -- 172.21.15.115:0/2679120163 --> v1:172.21.15.1:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f83640ab6b0 con 0x7f836413ff20 2024-03-25T18:26:41.376 INFO:teuthology.orchestra.run.smithi115.stderr:2024-03-25T18:26:41.375+0000 7f83637fe640 1 -- 172.21.15.115:0/2679120163 <== mon.2 v1:172.21.15.1:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3316070708 0 0) 0x7f8354003520 con 0x7f83640b4b50 2024-03-25T18:26:41.376 INFO:teuthology.orchestra.run.smithi115.stderr:2024-03-25T18:26:41.375+0000 7f83637fe640 1 -- 172.21.15.115:0/2679120163 >> v1:172.21.15.115:6789/0 conn(0x7f8364142dd0 legacy=0x7f836414df50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:26:41.376 INFO:teuthology.orchestra.run.smithi115.stderr:2024-03-25T18:26:41.375+0000 7f83637fe640 1 -- 172.21.15.115:0/2679120163 >> v1:172.21.15.1:6789/0 conn(0x7f836413ff20 legacy=0x7f83641426c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:26:41.376 INFO:teuthology.orchestra.run.smithi115.stderr:2024-03-25T18:26:41.375+0000 7f83637fe640 10 monclient: _finish_hunting 0 2024-03-25T18:26:41.376 INFO:teuthology.orchestra.run.smithi115.stderr:2024-03-25T18:26:41.375+0000 7f83637fe640 1 monclient: found mon.noname-b 2024-03-25T18:26:41.377 INFO:teuthology.orchestra.run.smithi115.stderr:2024-03-25T18:26:41.375+0000 7f83637fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:26:41.377 INFO:teuthology.orchestra.run.smithi115.stderr:2024-03-25T18:26:41.375+0000 7f83637fe640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.1:6790/0 2024-03-25T18:26:41.377 INFO:teuthology.orchestra.run.smithi115.stderr:2024-03-25T18:26:41.375+0000 7f83637fe640 1 -- 172.21.15.115:0/2679120163 --> v1:172.21.15.1:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f83640aa0a0 con 0x7f83640b4b50 2024-03-25T18:26:41.377 INFO:teuthology.orchestra.run.smithi115.stderr:2024-03-25T18:26:41.375+0000 7f83637fe640 10 monclient: _finish_auth 0 2024-03-25T18:26:41.377 INFO:teuthology.orchestra.run.smithi115.stderr:2024-03-25T18:26:41.375+0000 7f83637fe640 10 monclient: _check_auth_tickets 2024-03-25T18:26:41.377 INFO:teuthology.orchestra.run.smithi115.stderr:2024-03-25T18:26:41.375+0000 7f83637fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:26:41.377 INFO:teuthology.orchestra.run.smithi115.stderr:2024-03-25T18:26:41.375+0000 7f83637fe640 1 -- 172.21.15.115:0/2679120163 <== mon.2 v1:172.21.15.1:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1586965025 0 0) 0x7f8358001690 con 0x7f83640b4b50 2024-03-25T18:26:41.377 INFO:teuthology.orchestra.run.smithi115.stderr:2024-03-25T18:26:41.375+0000 7f836b24a640 5 monclient: authenticate success, global_id 4118 2024-03-25T18:26:41.377 INFO:teuthology.orchestra.run.smithi115.stderr:2024-03-25T18:26:41.375+0000 7f83637fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:26:41.377 INFO:teuthology.orchestra.run.smithi115.stderr:2024-03-25T18:26:41.375+0000 7f83637fe640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-03-25T18:26:41.377 INFO:teuthology.orchestra.run.smithi115.stderr:2024-03-25T18:26:41.375+0000 7f83637fe640 10 monclient: dump: 2024-03-25T18:26:41.377 INFO:teuthology.orchestra.run.smithi115.stderr:epoch 1 2024-03-25T18:26:41.377 INFO:teuthology.orchestra.run.smithi115.stderr:fsid c8ea72cf-9e99-4383-9b52-6c23d365fdfc 2024-03-25T18:26:41.377 INFO:teuthology.orchestra.run.smithi115.stderr:last_changed 2024-03-25T18:25:50.510519+0000 2024-03-25T18:26:41.377 INFO:teuthology.orchestra.run.smithi115.stderr:created 2024-03-25T18:25:50.510519+0000 2024-03-25T18:26:41.378 INFO:teuthology.orchestra.run.smithi115.stderr:min_mon_release 19 (squid) 2024-03-25T18:26:41.378 INFO:teuthology.orchestra.run.smithi115.stderr:election_strategy: 3 2024-03-25T18:26:41.378 INFO:teuthology.orchestra.run.smithi115.stderr:0: v1:172.21.15.1:6789/0 mon.a 2024-03-25T18:26:41.378 INFO:teuthology.orchestra.run.smithi115.stderr:1: v1:172.21.15.115:6789/0 mon.b 2024-03-25T18:26:41.378 INFO:teuthology.orchestra.run.smithi115.stderr:2: v1:172.21.15.1:6790/0 mon.c 2024-03-25T18:26:41.378 INFO:teuthology.orchestra.run.smithi115.stderr: 2024-03-25T18:26:41.378 INFO:teuthology.orchestra.run.smithi115.stderr:2024-03-25T18:26:41.375+0000 7f836b24a640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T18:26:41.381 INFO:teuthology.orchestra.run.smithi115.stderr:2024-03-25T18:26:41.376+0000 7f83637fe640 1 -- 172.21.15.115:0/2679120163 <== mon.2 v1:172.21.15.1:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f83580017f0 con 0x7f83640b4b50 2024-03-25T18:26:41.381 INFO:teuthology.orchestra.run.smithi115.stderr:2024-03-25T18:26:41.376+0000 7f83637fe640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:26:41.382 INFO:teuthology.orchestra.run.smithi115.stderr:2024-03-25T18:26:41.376+0000 7f83637fe640 1 -- 172.21.15.115:0/2679120163 <== mon.2 v1:172.21.15.1:6790/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1586965025 0 0) 0x7f8358001b80 con 0x7f83640b4b50 2024-03-25T18:26:41.382 INFO:teuthology.orchestra.run.smithi115.stderr:2024-03-25T18:26:41.376+0000 7f83637fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:26:41.382 INFO:teuthology.orchestra.run.smithi115.stderr:2024-03-25T18:26:41.376+0000 7f83637fe640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-25T18:26:41.382 INFO:teuthology.orchestra.run.smithi115.stderr:2024-03-25T18:26:41.376+0000 7f83637fe640 10 monclient: dump: 2024-03-25T18:26:41.382 INFO:teuthology.orchestra.run.smithi115.stderr:epoch 1 2024-03-25T18:26:41.382 INFO:teuthology.orchestra.run.smithi115.stderr:fsid c8ea72cf-9e99-4383-9b52-6c23d365fdfc 2024-03-25T18:26:41.382 INFO:teuthology.orchestra.run.smithi115.stderr:last_changed 2024-03-25T18:25:50.510519+0000 2024-03-25T18:26:41.382 INFO:teuthology.orchestra.run.smithi115.stderr:created 2024-03-25T18:25:50.510519+0000 2024-03-25T18:26:41.382 INFO:teuthology.orchestra.run.smithi115.stderr:min_mon_release 19 (squid) 2024-03-25T18:26:41.382 INFO:teuthology.orchestra.run.smithi115.stderr:election_strategy: 3 2024-03-25T18:26:41.382 INFO:teuthology.orchestra.run.smithi115.stderr:0: v1:172.21.15.1:6789/0 mon.a 2024-03-25T18:26:41.382 INFO:teuthology.orchestra.run.smithi115.stderr:1: v1:172.21.15.115:6789/0 mon.b 2024-03-25T18:26:41.383 INFO:teuthology.orchestra.run.smithi115.stderr:2: v1:172.21.15.1:6790/0 mon.c 2024-03-25T18:26:41.383 INFO:teuthology.orchestra.run.smithi115.stderr: 2024-03-25T18:26:41.383 INFO:teuthology.orchestra.run.smithi115.stderr:2024-03-25T18:26:41.377+0000 7f836b24a640 10 monclient: get_monmap_and_config success 2024-03-25T18:26:41.383 INFO:teuthology.orchestra.run.smithi115.stderr:2024-03-25T18:26:41.377+0000 7f836b24a640 4 set_mon_vals no callback set 2024-03-25T18:26:41.383 INFO:teuthology.orchestra.run.smithi115.stderr:2024-03-25T18:26:41.377+0000 7f836b24a640 10 monclient: shutdown 2024-03-25T18:26:41.383 INFO:teuthology.orchestra.run.smithi115.stderr:2024-03-25T18:26:41.377+0000 7f836b24a640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:26:41.383 INFO:teuthology.orchestra.run.smithi115.stderr:2024-03-25T18:26:41.377+0000 7f836b24a640 1 -- 172.21.15.115:0/2679120163 >> v1:172.21.15.1:6790/0 conn(0x7f83640b4b50 legacy=0x7f836413f810 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:26:41.383 INFO:teuthology.orchestra.run.smithi115.stderr:2024-03-25T18:26:41.377+0000 7f836b24a640 1 -- 172.21.15.115:0/2679120163 shutdown_connections 2024-03-25T18:26:41.383 INFO:teuthology.orchestra.run.smithi115.stderr:2024-03-25T18:26:41.377+0000 7f836b24a640 1 -- 172.21.15.115:0/2679120163 >> 172.21.15.115:0/2679120163 conn(0x7f8364066920 msgr2=0x7f8364066d30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:26:41.383 INFO:teuthology.orchestra.run.smithi115.stderr:2024-03-25T18:26:41.377+0000 7f836b24a640 1 -- 172.21.15.115:0/2679120163 shutdown_connections 2024-03-25T18:26:41.383 INFO:teuthology.orchestra.run.smithi115.stderr:2024-03-25T18:26:41.377+0000 7f836b24a640 1 -- 172.21.15.115:0/2679120163 wait complete. 2024-03-25T18:26:41.383 INFO:teuthology.orchestra.run.smithi115.stderr:2024-03-25T18:26:41.378+0000 7f836b24a640 10 monclient: build_initial_monmap 2024-03-25T18:26:41.383 INFO:teuthology.orchestra.run.smithi115.stderr:2024-03-25T18:26:41.378+0000 7f836b24a640 10 monclient: monmap: 2024-03-25T18:26:41.383 INFO:teuthology.orchestra.run.smithi115.stderr:epoch 0 2024-03-25T18:26:41.383 INFO:teuthology.orchestra.run.smithi115.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T18:26:41.384 INFO:teuthology.orchestra.run.smithi115.stderr:last_changed 0.000000 2024-03-25T18:26:41.384 INFO:teuthology.orchestra.run.smithi115.stderr:created 0.000000 2024-03-25T18:26:41.384 INFO:teuthology.orchestra.run.smithi115.stderr:min_mon_release 0 (unknown) 2024-03-25T18:26:41.384 INFO:teuthology.orchestra.run.smithi115.stderr:election_strategy: 1 2024-03-25T18:26:41.384 INFO:teuthology.orchestra.run.smithi115.stderr:0: v1:172.21.15.1:6789/0 mon.noname-a 2024-03-25T18:26:41.384 INFO:teuthology.orchestra.run.smithi115.stderr:1: v1:172.21.15.115:6789/0 mon.noname-b 2024-03-25T18:26:41.384 INFO:teuthology.orchestra.run.smithi115.stderr:2: v1:172.21.15.1:6790/0 mon.noname-c 2024-03-25T18:26:41.384 INFO:teuthology.orchestra.run.smithi115.stderr: 2024-03-25T18:26:41.384 INFO:teuthology.orchestra.run.smithi115.stderr:2024-03-25T18:26:41.378+0000 7f836b24a640 1 librados: starting msgr at 2024-03-25T18:26:41.384 INFO:teuthology.orchestra.run.smithi115.stderr:2024-03-25T18:26:41.378+0000 7f836b24a640 1 librados: starting objecter 2024-03-25T18:26:41.384 INFO:teuthology.orchestra.run.smithi115.stderr:2024-03-25T18:26:41.378+0000 7f836b24a640 1 Processor -- start 2024-03-25T18:26:41.384 INFO:teuthology.orchestra.run.smithi115.stderr:2024-03-25T18:26:41.378+0000 7f836b24a640 1 -- start start 2024-03-25T18:26:41.384 INFO:teuthology.orchestra.run.smithi115.stderr:2024-03-25T18:26:41.378+0000 7f836b24a640 1 librados: setting wanted keys 2024-03-25T18:26:41.384 INFO:teuthology.orchestra.run.smithi115.stderr:2024-03-25T18:26:41.378+0000 7f836b24a640 1 librados: calling monclient init 2024-03-25T18:26:41.385 INFO:teuthology.orchestra.run.smithi115.stderr:2024-03-25T18:26:41.378+0000 7f836b24a640 10 monclient: init 2024-03-25T18:26:41.385 INFO:teuthology.orchestra.run.smithi115.stderr:2024-03-25T18:26:41.379+0000 7f836b24a640 10 monclient: _reopen_session rank -1 2024-03-25T18:26:41.385 INFO:teuthology.orchestra.run.smithi115.stderr:2024-03-25T18:26:41.379+0000 7f836b24a640 10 monclient: _add_conns ranks=[1,0,2] 2024-03-25T18:26:41.385 INFO:teuthology.orchestra.run.smithi115.stderr:2024-03-25T18:26:41.379+0000 7f836b24a640 10 monclient(hunting): picked mon.noname-b con 0x7f83640b4b50 addr v1:172.21.15.115:6789/0 2024-03-25T18:26:41.385 INFO:teuthology.orchestra.run.smithi115.stderr:2024-03-25T18:26:41.379+0000 7f836b24a640 10 monclient(hunting): picked mon.noname-a con 0x7f836413ff20 addr v1:172.21.15.1:6789/0 2024-03-25T18:26:41.385 INFO:teuthology.orchestra.run.smithi115.stderr:2024-03-25T18:26:41.379+0000 7f836b24a640 10 monclient(hunting): picked mon.noname-c con 0x7f8364142dd0 addr v1:172.21.15.1:6790/0 2024-03-25T18:26:41.385 INFO:teuthology.orchestra.run.smithi115.stderr:2024-03-25T18:26:41.379+0000 7f836b24a640 1 -- --> v1:172.21.15.1:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8364144730 con 0x7f836413ff20 2024-03-25T18:26:41.385 INFO:teuthology.orchestra.run.smithi115.stderr:2024-03-25T18:26:41.379+0000 7f836b24a640 1 -- --> v1:172.21.15.115:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8364144a10 con 0x7f83640b4b50 2024-03-25T18:26:41.385 INFO:teuthology.orchestra.run.smithi115.stderr:2024-03-25T18:26:41.379+0000 7f836b24a640 1 -- --> v1:172.21.15.1:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8364134dc0 con 0x7f8364142dd0 2024-03-25T18:26:41.385 INFO:teuthology.orchestra.run.smithi115.stderr:2024-03-25T18:26:41.379+0000 7f836b24a640 10 monclient(hunting): _renew_subs 2024-03-25T18:26:41.385 INFO:teuthology.orchestra.run.smithi115.stderr:2024-03-25T18:26:41.379+0000 7f836b24a640 10 monclient(hunting): authenticate will time out at 7492.535587s 2024-03-25T18:26:41.385 INFO:teuthology.orchestra.run.smithi115.stderr:2024-03-25T18:26:41.380+0000 7f83697e7640 1 --1- >> v1:172.21.15.1:6790/0 conn(0x7f8364142dd0 0x7f83641e2360 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.1:6790/0 says I am v1:172.21.15.115:50234/0 (socket says 172.21.15.115:50234) 2024-03-25T18:26:41.385 INFO:teuthology.orchestra.run.smithi115.stderr:2024-03-25T18:26:41.380+0000 7f83697e7640 1 -- 172.21.15.115:0/1350219835 learned_addr learned my addr 172.21.15.115:0/1350219835 (peer_addr_for_me v1:172.21.15.115:0/0) 2024-03-25T18:26:41.385 INFO:teuthology.orchestra.run.smithi115.stderr:2024-03-25T18:26:41.380+0000 7f83617fa640 10 client.?.objecter ms_handle_connect 0x7f8364142dd0 2024-03-25T18:26:41.386 INFO:teuthology.orchestra.run.smithi115.stderr:2024-03-25T18:26:41.380+0000 7f83617fa640 10 client.?.objecter resend_mon_ops 2024-03-25T18:26:41.386 INFO:teuthology.orchestra.run.smithi115.stderr:2024-03-25T18:26:41.380+0000 7f83617fa640 10 client.?.objecter ms_handle_connect 0x7f83640b4b50 2024-03-25T18:26:41.386 INFO:teuthology.orchestra.run.smithi115.stderr:2024-03-25T18:26:41.380+0000 7f83617fa640 10 client.?.objecter resend_mon_ops 2024-03-25T18:26:41.386 INFO:teuthology.orchestra.run.smithi115.stderr:2024-03-25T18:26:41.380+0000 7f83617fa640 10 client.?.objecter ms_handle_connect 0x7f836413ff20 2024-03-25T18:26:41.386 INFO:teuthology.orchestra.run.smithi115.stderr:2024-03-25T18:26:41.380+0000 7f83617fa640 10 client.?.objecter resend_mon_ops 2024-03-25T18:26:41.387 INFO:teuthology.orchestra.run.smithi115.stderr:2024-03-25T18:26:41.382+0000 7f83617fa640 1 -- 172.21.15.115:0/1350219835 <== mon.2 v1:172.21.15.1:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4011417765 0 0) 0x7f8364134dc0 con 0x7f8364142dd0 2024-03-25T18:26:41.387 INFO:teuthology.orchestra.run.smithi115.stderr:2024-03-25T18:26:41.382+0000 7f83617fa640 10 monclient(hunting): _negotiate 2024-03-25T18:26:41.387 INFO:teuthology.orchestra.run.smithi115.stderr:2024-03-25T18:26:41.382+0000 7f83617fa640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:26:41.387 INFO:teuthology.orchestra.run.smithi115.stderr:2024-03-25T18:26:41.382+0000 7f83617fa640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:26:41.387 INFO:teuthology.orchestra.run.smithi115.stderr:2024-03-25T18:26:41.382+0000 7f83617fa640 10 monclient(hunting): my global_id is 4121 2024-03-25T18:26:41.387 INFO:teuthology.orchestra.run.smithi115.stderr:2024-03-25T18:26:41.382+0000 7f83617fa640 1 -- 172.21.15.115:0/1350219835 --> v1:172.21.15.1:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f8348004eb0 con 0x7f8364142dd0 2024-03-25T18:26:41.387 INFO:teuthology.orchestra.run.smithi115.stderr:2024-03-25T18:26:41.382+0000 7f83617fa640 1 -- 172.21.15.115:0/1350219835 <== mon.1 v1:172.21.15.115:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1591563177 0 0) 0x7f8364144a10 con 0x7f83640b4b50 2024-03-25T18:26:41.387 INFO:teuthology.orchestra.run.smithi115.stderr:2024-03-25T18:26:41.382+0000 7f83617fa640 10 monclient(hunting): _negotiate 2024-03-25T18:26:41.387 INFO:teuthology.orchestra.run.smithi115.stderr:2024-03-25T18:26:41.382+0000 7f83617fa640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:26:41.388 INFO:teuthology.orchestra.run.smithi115.stderr:2024-03-25T18:26:41.382+0000 7f83617fa640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:26:41.388 INFO:teuthology.orchestra.run.smithi115.stderr:2024-03-25T18:26:41.382+0000 7f83617fa640 10 monclient(hunting): my global_id is 4117 2024-03-25T18:26:41.388 INFO:teuthology.orchestra.run.smithi115.stderr:2024-03-25T18:26:41.382+0000 7f83617fa640 1 -- 172.21.15.115:0/1350219835 --> v1:172.21.15.115:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f8364134dc0 con 0x7f83640b4b50 2024-03-25T18:26:41.388 INFO:teuthology.orchestra.run.smithi115.stderr:2024-03-25T18:26:41.383+0000 7f83617fa640 1 -- 172.21.15.115:0/1350219835 <== mon.0 v1:172.21.15.1:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1375907235 0 0) 0x7f8364144730 con 0x7f836413ff20 2024-03-25T18:26:41.388 INFO:teuthology.orchestra.run.smithi115.stderr:2024-03-25T18:26:41.383+0000 7f83617fa640 10 monclient(hunting): _negotiate 2024-03-25T18:26:41.388 INFO:teuthology.orchestra.run.smithi115.stderr:2024-03-25T18:26:41.383+0000 7f83617fa640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:26:41.388 INFO:teuthology.orchestra.run.smithi115.stderr:2024-03-25T18:26:41.383+0000 7f83617fa640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:26:41.388 INFO:teuthology.orchestra.run.smithi115.stderr:2024-03-25T18:26:41.383+0000 7f83617fa640 10 monclient(hunting): my global_id is 4122 2024-03-25T18:26:41.388 INFO:teuthology.orchestra.run.smithi115.stderr:2024-03-25T18:26:41.383+0000 7f83617fa640 1 -- 172.21.15.115:0/1350219835 --> v1:172.21.15.1:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f8364144a10 con 0x7f836413ff20 2024-03-25T18:26:41.390 INFO:teuthology.orchestra.run.smithi115.stderr:2024-03-25T18:26:41.387+0000 7f83617fa640 1 -- 172.21.15.115:0/1350219835 <== mon.2 v1:172.21.15.1:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1836974499 0 0) 0x7f8348004eb0 con 0x7f8364142dd0 2024-03-25T18:26:41.390 INFO:teuthology.orchestra.run.smithi115.stderr:2024-03-25T18:26:41.387+0000 7f83617fa640 1 -- 172.21.15.115:0/1350219835 --> v1:172.21.15.1:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f8364144730 con 0x7f8364142dd0 2024-03-25T18:26:41.390 INFO:teuthology.orchestra.run.smithi115.stderr:2024-03-25T18:26:41.387+0000 7f83617fa640 1 -- 172.21.15.115:0/1350219835 <== mon.2 v1:172.21.15.1:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1586965025 0 0) 0x7f835c001c90 con 0x7f8364142dd0 2024-03-25T18:26:41.390 INFO:teuthology.orchestra.run.smithi115.stderr:2024-03-25T18:26:41.387+0000 7f83617fa640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:26:41.390 INFO:teuthology.orchestra.run.smithi115.stderr:2024-03-25T18:26:41.387+0000 7f83617fa640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-03-25T18:26:41.390 INFO:teuthology.orchestra.run.smithi115.stderr:2024-03-25T18:26:41.387+0000 7f83617fa640 10 monclient(hunting): dump: 2024-03-25T18:26:41.390 INFO:teuthology.orchestra.run.smithi115.stderr:epoch 1 2024-03-25T18:26:41.391 INFO:teuthology.orchestra.run.smithi115.stderr:fsid c8ea72cf-9e99-4383-9b52-6c23d365fdfc 2024-03-25T18:26:41.391 INFO:teuthology.orchestra.run.smithi115.stderr:last_changed 2024-03-25T18:25:50.510519+0000 2024-03-25T18:26:41.391 INFO:teuthology.orchestra.run.smithi115.stderr:created 2024-03-25T18:25:50.510519+0000 2024-03-25T18:26:41.391 INFO:teuthology.orchestra.run.smithi115.stderr:min_mon_release 19 (squid) 2024-03-25T18:26:41.391 INFO:teuthology.orchestra.run.smithi115.stderr:election_strategy: 3 2024-03-25T18:26:41.391 INFO:teuthology.orchestra.run.smithi115.stderr:0: v1:172.21.15.1:6789/0 mon.a 2024-03-25T18:26:41.391 INFO:teuthology.orchestra.run.smithi115.stderr:1: v1:172.21.15.115:6789/0 mon.b 2024-03-25T18:26:41.391 INFO:teuthology.orchestra.run.smithi115.stderr:2: v1:172.21.15.1:6790/0 mon.c 2024-03-25T18:26:41.391 INFO:teuthology.orchestra.run.smithi115.stderr: 2024-03-25T18:26:41.391 INFO:teuthology.orchestra.run.smithi115.stderr:2024-03-25T18:26:41.387+0000 7f83617fa640 10 monclient(hunting): _finish_auth 0 2024-03-25T18:26:41.391 INFO:teuthology.orchestra.run.smithi115.stderr:2024-03-25T18:26:41.388+0000 7f83617fa640 1 -- 172.21.15.115:0/1350219835 <== mon.0 v1:172.21.15.1:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 11786361 0 0) 0x7f8364144a10 con 0x7f836413ff20 2024-03-25T18:26:41.391 INFO:teuthology.orchestra.run.smithi115.stderr:2024-03-25T18:26:41.388+0000 7f83617fa640 1 -- 172.21.15.115:0/1350219835 --> v1:172.21.15.1:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f8348004eb0 con 0x7f836413ff20 2024-03-25T18:26:41.391 INFO:teuthology.orchestra.run.smithi115.stderr:2024-03-25T18:26:41.388+0000 7f83617fa640 1 -- 172.21.15.115:0/1350219835 <== mon.0 v1:172.21.15.1:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1586965025 0 0) 0x7f83500017e0 con 0x7f836413ff20 2024-03-25T18:26:41.392 INFO:teuthology.orchestra.run.smithi115.stderr:2024-03-25T18:26:41.388+0000 7f83617fa640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:26:41.392 INFO:teuthology.orchestra.run.smithi115.stderr:2024-03-25T18:26:41.388+0000 7f83617fa640 10 monclient(hunting): got monmap 1 from mon.a (according to old e1) 2024-03-25T18:26:41.392 INFO:teuthology.orchestra.run.smithi115.stderr:2024-03-25T18:26:41.388+0000 7f83617fa640 10 monclient(hunting): dump: 2024-03-25T18:26:41.392 INFO:teuthology.orchestra.run.smithi115.stderr:epoch 1 2024-03-25T18:26:41.392 INFO:teuthology.orchestra.run.smithi115.stderr:fsid c8ea72cf-9e99-4383-9b52-6c23d365fdfc 2024-03-25T18:26:41.392 INFO:teuthology.orchestra.run.smithi115.stderr:last_changed 2024-03-25T18:25:50.510519+0000 2024-03-25T18:26:41.392 INFO:teuthology.orchestra.run.smithi115.stderr:created 2024-03-25T18:25:50.510519+0000 2024-03-25T18:26:41.392 INFO:teuthology.orchestra.run.smithi115.stderr:min_mon_release 19 (squid) 2024-03-25T18:26:41.392 INFO:teuthology.orchestra.run.smithi115.stderr:election_strategy: 3 2024-03-25T18:26:41.392 INFO:teuthology.orchestra.run.smithi115.stderr:0: v1:172.21.15.1:6789/0 mon.a 2024-03-25T18:26:41.392 INFO:teuthology.orchestra.run.smithi115.stderr:1: v1:172.21.15.115:6789/0 mon.b 2024-03-25T18:26:41.392 INFO:teuthology.orchestra.run.smithi115.stderr:2: v1:172.21.15.1:6790/0 mon.c 2024-03-25T18:26:41.393 INFO:teuthology.orchestra.run.smithi115.stderr: 2024-03-25T18:26:41.394 INFO:teuthology.orchestra.run.smithi115.stderr:2024-03-25T18:26:41.391+0000 7f83617fa640 1 -- 172.21.15.115:0/1350219835 <== mon.0 v1:172.21.15.1:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 736433149 0 0) 0x7f8348004eb0 con 0x7f836413ff20 2024-03-25T18:26:41.394 INFO:teuthology.orchestra.run.smithi115.stderr:2024-03-25T18:26:41.391+0000 7f83617fa640 1 -- 172.21.15.115:0/1350219835 >> v1:172.21.15.1:6790/0 conn(0x7f8364142dd0 legacy=0x7f83641e2360 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:26:41.395 INFO:teuthology.orchestra.run.smithi115.stderr:2024-03-25T18:26:41.391+0000 7f83617fa640 1 -- 172.21.15.115:0/1350219835 >> v1:172.21.15.115:6789/0 conn(0x7f83640b4b50 legacy=0x7f8364138840 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:26:41.395 INFO:teuthology.orchestra.run.smithi115.stderr:2024-03-25T18:26:41.391+0000 7f83617fa640 10 monclient: _finish_hunting 0 2024-03-25T18:26:41.395 INFO:teuthology.orchestra.run.smithi115.stderr:2024-03-25T18:26:41.391+0000 7f83617fa640 1 monclient: found mon.a 2024-03-25T18:26:41.395 INFO:teuthology.orchestra.run.smithi115.stderr:2024-03-25T18:26:41.391+0000 7f83617fa640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:26:41.395 INFO:teuthology.orchestra.run.smithi115.stderr:2024-03-25T18:26:41.391+0000 7f83617fa640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.1:6789/0 2024-03-25T18:26:41.395 INFO:teuthology.orchestra.run.smithi115.stderr:2024-03-25T18:26:41.391+0000 7f83617fa640 1 -- 172.21.15.115:0/1350219835 --> v1:172.21.15.1:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f83641439b0 con 0x7f836413ff20 2024-03-25T18:26:41.395 INFO:teuthology.orchestra.run.smithi115.stderr:2024-03-25T18:26:41.391+0000 7f83617fa640 10 monclient: _finish_auth 0 2024-03-25T18:26:41.395 INFO:teuthology.orchestra.run.smithi115.stderr:2024-03-25T18:26:41.391+0000 7f83617fa640 10 monclient: _check_auth_tickets 2024-03-25T18:26:41.395 INFO:teuthology.orchestra.run.smithi115.stderr:2024-03-25T18:26:41.391+0000 7f83617fa640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:26:41.395 INFO:teuthology.orchestra.run.smithi115.stderr:2024-03-25T18:26:41.391+0000 7f836b24a640 5 monclient: authenticate success, global_id 4122 2024-03-25T18:26:41.395 INFO:teuthology.orchestra.run.smithi115.stderr:2024-03-25T18:26:41.391+0000 7f836b24a640 10 monclient: _renew_subs 2024-03-25T18:26:41.395 INFO:teuthology.orchestra.run.smithi115.stderr:2024-03-25T18:26:41.391+0000 7f836b24a640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.1:6789/0 2024-03-25T18:26:41.396 INFO:teuthology.orchestra.run.smithi115.stderr:2024-03-25T18:26:41.391+0000 7f836b24a640 1 -- 172.21.15.115:0/1350219835 --> v1:172.21.15.1:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8364143c60 con 0x7f836413ff20 2024-03-25T18:26:41.396 INFO:teuthology.orchestra.run.smithi115.stderr:2024-03-25T18:26:41.391+0000 7f836b24a640 10 client.4122.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T18:26:41.396 INFO:teuthology.orchestra.run.smithi115.stderr:2024-03-25T18:26:41.391+0000 7f836b24a640 10 monclient: _renew_subs 2024-03-25T18:26:41.396 INFO:teuthology.orchestra.run.smithi115.stderr:2024-03-25T18:26:41.391+0000 7f836b24a640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.1:6789/0 2024-03-25T18:26:41.396 INFO:teuthology.orchestra.run.smithi115.stderr:2024-03-25T18:26:41.391+0000 7f836b24a640 1 -- 172.21.15.115:0/1350219835 --> v1:172.21.15.1:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f836413f0b0 con 0x7f836413ff20 2024-03-25T18:26:41.396 INFO:teuthology.orchestra.run.smithi115.stderr:2024-03-25T18:26:41.391+0000 7f836b24a640 1 librados: init done 2024-03-25T18:26:41.396 INFO:teuthology.orchestra.run.smithi115.stderr:2024-03-25T18:26:41.392+0000 7f836b24a640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T18:26:41.396 INFO:teuthology.orchestra.run.smithi115.stderr:2024-03-25T18:26:41.392+0000 7f836b24a640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:26:41.396 INFO:teuthology.orchestra.run.smithi115.stderr:2024-03-25T18:26:41.392+0000 7f836b24a640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.1:6789/0 2024-03-25T18:26:41.396 INFO:teuthology.orchestra.run.smithi115.stderr:2024-03-25T18:26:41.392+0000 7f836b24a640 1 -- 172.21.15.115:0/1350219835 --> v1:172.21.15.1:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f836404f730 con 0x7f836413ff20 2024-03-25T18:26:41.413 INFO:tasks.ceph.mgr.y.smithi001.stderr:2024-03-25T18:26:41.412+0000 7faca7441040 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2024-03-25T18:26:41.418 INFO:tasks.ceph.mgr.x.smithi115.stderr:2024-03-25T18:26:41.417+0000 7f99dd247040 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2024-03-25T18:26:41.465 INFO:tasks.ceph.mgr.x.smithi115.stderr:2024-03-25T18:26:41.464+0000 7f99dd247040 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2024-03-25T18:26:41.469 INFO:tasks.ceph.mgr.y.smithi001.stderr:2024-03-25T18:26:41.467+0000 7faca7441040 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2024-03-25T18:26:41.499 INFO:teuthology.orchestra.run.smithi115.stderr:2024-03-25T18:26:41.498+0000 7f83617fa640 1 -- 172.21.15.115:0/1350219835 <== mon.0 v1:172.21.15.1:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f8350001cf0 con 0x7f836413ff20 2024-03-25T18:26:41.500 INFO:teuthology.orchestra.run.smithi115.stderr:2024-03-25T18:26:41.498+0000 7f83617fa640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:26:41.500 INFO:teuthology.orchestra.run.smithi115.stderr:2024-03-25T18:26:41.498+0000 7f83617fa640 1 -- 172.21.15.115:0/1350219835 <== mon.0 v1:172.21.15.1:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1586965025 0 0) 0x7f83500037b0 con 0x7f836413ff20 2024-03-25T18:26:41.500 INFO:teuthology.orchestra.run.smithi115.stderr:2024-03-25T18:26:41.498+0000 7f83617fa640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:26:41.500 INFO:teuthology.orchestra.run.smithi115.stderr:2024-03-25T18:26:41.498+0000 7f83617fa640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-25T18:26:41.500 INFO:teuthology.orchestra.run.smithi115.stderr:2024-03-25T18:26:41.498+0000 7f83617fa640 10 monclient: dump: 2024-03-25T18:26:41.500 INFO:teuthology.orchestra.run.smithi115.stderr:epoch 1 2024-03-25T18:26:41.501 INFO:teuthology.orchestra.run.smithi115.stderr:fsid c8ea72cf-9e99-4383-9b52-6c23d365fdfc 2024-03-25T18:26:41.501 INFO:teuthology.orchestra.run.smithi115.stderr:last_changed 2024-03-25T18:25:50.510519+0000 2024-03-25T18:26:41.501 INFO:teuthology.orchestra.run.smithi115.stderr:created 2024-03-25T18:25:50.510519+0000 2024-03-25T18:26:41.501 INFO:teuthology.orchestra.run.smithi115.stderr:min_mon_release 19 (squid) 2024-03-25T18:26:41.501 INFO:teuthology.orchestra.run.smithi115.stderr:election_strategy: 3 2024-03-25T18:26:41.501 INFO:teuthology.orchestra.run.smithi115.stderr:0: v1:172.21.15.1:6789/0 mon.a 2024-03-25T18:26:41.502 INFO:teuthology.orchestra.run.smithi115.stderr:1: v1:172.21.15.115:6789/0 mon.b 2024-03-25T18:26:41.502 INFO:teuthology.orchestra.run.smithi115.stderr:2: v1:172.21.15.1:6790/0 mon.c 2024-03-25T18:26:41.502 INFO:teuthology.orchestra.run.smithi115.stderr: 2024-03-25T18:26:41.502 INFO:teuthology.orchestra.run.smithi115.stderr:2024-03-25T18:26:41.498+0000 7f83617fa640 1 -- 172.21.15.115:0/1350219835 <== mon.0 v1:172.21.15.1:6789/0 7 ==== mgrmap(e 1) v1 ==== 807+0+0 (unknown 3189685207 0 0) 0x7f8350003ca0 con 0x7f836413ff20 2024-03-25T18:26:41.502 INFO:teuthology.orchestra.run.smithi115.stderr:2024-03-25T18:26:41.498+0000 7f8362ffd640 4 set_mon_vals no callback set 2024-03-25T18:26:41.503 INFO:teuthology.orchestra.run.smithi115.stderr:2024-03-25T18:26:41.499+0000 7f83617fa640 1 -- 172.21.15.115:0/1350219835 <== mon.0 v1:172.21.15.1:6789/0 8 ==== osd_map(3..3 src has 1..3) v4 ==== 840+0+0 (unknown 2055477717 0 0) 0x7f8350005180 con 0x7f836413ff20 2024-03-25T18:26:41.503 INFO:teuthology.orchestra.run.smithi115.stderr:2024-03-25T18:26:41.499+0000 7f83617fa640 10 client.4122.objecter ms_dispatch 0x7f8364000b90 osd_map(3..3 src has 1..3) v4 2024-03-25T18:26:41.503 INFO:teuthology.orchestra.run.smithi115.stderr:2024-03-25T18:26:41.499+0000 7f83617fa640 3 client.4122.objecter handle_osd_map got epochs [3,3] > 0 2024-03-25T18:26:41.503 INFO:teuthology.orchestra.run.smithi115.stderr:2024-03-25T18:26:41.499+0000 7f83617fa640 3 client.4122.objecter handle_osd_map decoding full epoch 3 2024-03-25T18:26:41.503 INFO:teuthology.orchestra.run.smithi115.stderr:2024-03-25T18:26:41.499+0000 7f83617fa640 20 client.4122.objecter dump_active .. 0 homeless 2024-03-25T18:26:41.503 INFO:teuthology.orchestra.run.smithi115.stderr:2024-03-25T18:26:41.499+0000 7f83617fa640 1 -- 172.21.15.115:0/1350219835 <== mon.0 v1:172.21.15.1:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+73141 (unknown 1092875540 0 2031569661) 0x7f8350004210 con 0x7f836413ff20 2024-03-25T18:26:41.504 INFO:teuthology.orchestra.run.smithi115.stderr:2024-03-25T18:26:41.499+0000 7f83617fa640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:26:41.504 INFO:teuthology.orchestra.run.smithi115.stderr:2024-03-25T18:26:41.499+0000 7f83617fa640 10 monclient: _finish_command 1 = system:0 2024-03-25T18:26:41.514 INFO:tasks.ceph.mgr.y.smithi001.stderr:2024-03-25T18:26:41.513+0000 7faca7441040 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2024-03-25T18:26:41.572 INFO:teuthology.orchestra.run.smithi115.stderr:2024-03-25T18:26:41.571+0000 7f836b24a640 10 start_mon_command cmd=[{"prefix": "osd new", "uuid": "42bf306f-55e3-4634-9874-8b4b73ac6644", "id": 1}] 2024-03-25T18:26:41.572 INFO:teuthology.orchestra.run.smithi115.stderr:2024-03-25T18:26:41.571+0000 7f836b24a640 10 monclient: _send_command 2 [{"prefix": "osd new", "uuid": "42bf306f-55e3-4634-9874-8b4b73ac6644", "id": 1}] 2024-03-25T18:26:41.572 INFO:teuthology.orchestra.run.smithi115.stderr:2024-03-25T18:26:41.571+0000 7f836b24a640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.1:6789/0 2024-03-25T18:26:41.573 INFO:teuthology.orchestra.run.smithi115.stderr:2024-03-25T18:26:41.571+0000 7f836b24a640 1 -- 172.21.15.115:0/1350219835 --> v1:172.21.15.1:6789/0 -- mon_command({"prefix": "osd new", "uuid": "42bf306f-55e3-4634-9874-8b4b73ac6644", "id": 1} v 0) v1 -- 0x7f83640681c0 con 0x7f836413ff20 2024-03-25T18:26:41.614 INFO:teuthology.orchestra.run.smithi115.stderr:2024-03-25T18:26:41.613+0000 7f83617fa640 1 -- 172.21.15.115:0/1350219835 <== mon.0 v1:172.21.15.1:6789/0 10 ==== mon_command_ack([{"prefix": "osd new", "uuid": "42bf306f-55e3-4634-9874-8b4b73ac6644", "id": 1}]=0 v4) v1 ==== 112+0+2 (unknown 3944164976 0 940483276) 0x7f8350004520 con 0x7f836413ff20 2024-03-25T18:26:41.615 INFO:teuthology.orchestra.run.smithi115.stderr:2024-03-25T18:26:41.614+0000 7f83617fa640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd new", "uuid": "42bf306f-55e3-4634-9874-8b4b73ac6644", "id": 1}] 2024-03-25T18:26:41.615 INFO:teuthology.orchestra.run.smithi115.stderr:2024-03-25T18:26:41.614+0000 7f83617fa640 10 monclient: _finish_command 2 = system:0 2024-03-25T18:26:41.615 INFO:teuthology.orchestra.run.smithi115.stdout:1 2024-03-25T18:26:41.616 INFO:teuthology.orchestra.run.smithi115.stderr:2024-03-25T18:26:41.615+0000 7f836b24a640 10 librados: watch_flush enter 2024-03-25T18:26:41.616 INFO:teuthology.orchestra.run.smithi115.stderr:2024-03-25T18:26:41.615+0000 7f836b24a640 10 librados: watch_flush exit 2024-03-25T18:26:41.616 INFO:teuthology.orchestra.run.smithi115.stderr:2024-03-25T18:26:41.615+0000 7f836b24a640 20 client.4122.objecter shutdown clearing up homeless session... 2024-03-25T18:26:41.616 INFO:teuthology.orchestra.run.smithi115.stderr:2024-03-25T18:26:41.615+0000 7f836b24a640 10 client.4122.objecter successfully canceled tick 2024-03-25T18:26:41.616 INFO:teuthology.orchestra.run.smithi115.stderr:2024-03-25T18:26:41.615+0000 7f836b24a640 10 monclient: shutdown 2024-03-25T18:26:41.617 INFO:teuthology.orchestra.run.smithi115.stderr:2024-03-25T18:26:41.615+0000 7f836b24a640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:26:41.617 INFO:teuthology.orchestra.run.smithi115.stderr:2024-03-25T18:26:41.615+0000 7f836b24a640 1 -- 172.21.15.115:0/1350219835 >> v1:172.21.15.1:6789/0 conn(0x7f836413ff20 legacy=0x7f8364138f50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:26:41.617 INFO:teuthology.orchestra.run.smithi115.stderr:2024-03-25T18:26:41.615+0000 7f836b24a640 1 -- 172.21.15.115:0/1350219835 shutdown_connections 2024-03-25T18:26:41.617 INFO:teuthology.orchestra.run.smithi115.stderr:2024-03-25T18:26:41.615+0000 7f836b24a640 1 -- 172.21.15.115:0/1350219835 >> 172.21.15.115:0/1350219835 conn(0x7f8364066920 msgr2=0x7f836413f5e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:26:41.617 INFO:teuthology.orchestra.run.smithi115.stderr:2024-03-25T18:26:41.615+0000 7f836b24a640 1 -- 172.21.15.115:0/1350219835 shutdown_connections 2024-03-25T18:26:41.617 INFO:teuthology.orchestra.run.smithi115.stderr:2024-03-25T18:26:41.615+0000 7f836b24a640 1 -- 172.21.15.115:0/1350219835 wait complete. 2024-03-25T18:26:41.617 INFO:teuthology.orchestra.run.smithi115.stderr:2024-03-25T18:26:41.615+0000 7f836b24a640 1 librados: shutdown 2024-03-25T18:26:41.629 DEBUG:teuthology.orchestra.run.smithi115:> sudo ceph --cluster ceph osd new 292cb23a-18a9-4207-a7cc-4d5727feaec0 2 2024-03-25T18:26:41.653 INFO:tasks.ceph.mgr.y.smithi001.stderr:2024-03-25T18:26:41.651+0000 7faca7441040 -1 mgr[py] Module rgw has missing NOTIFY_TYPES member 2024-03-25T18:26:41.738 INFO:tasks.ceph.mgr.y.smithi001.stderr:2024-03-25T18:26:41.736+0000 7faca7441040 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2024-03-25T18:26:41.773 INFO:teuthology.orchestra.run.smithi115.stderr:2024-03-25T18:26:41.770+0000 7f6053532640 10 monclient: get_monmap_and_config 2024-03-25T18:26:41.773 INFO:teuthology.orchestra.run.smithi115.stderr:2024-03-25T18:26:41.770+0000 7f6053532640 10 monclient: build_initial_monmap 2024-03-25T18:26:41.773 INFO:teuthology.orchestra.run.smithi115.stderr:2024-03-25T18:26:41.770+0000 7f6053532640 10 monclient: monmap: 2024-03-25T18:26:41.773 INFO:teuthology.orchestra.run.smithi115.stderr:epoch 0 2024-03-25T18:26:41.773 INFO:teuthology.orchestra.run.smithi115.stderr:fsid c8ea72cf-9e99-4383-9b52-6c23d365fdfc 2024-03-25T18:26:41.773 INFO:teuthology.orchestra.run.smithi115.stderr:last_changed 2024-03-25T18:26:41.771086+0000 2024-03-25T18:26:41.774 INFO:teuthology.orchestra.run.smithi115.stderr:created 2024-03-25T18:26:41.771086+0000 2024-03-25T18:26:41.774 INFO:teuthology.orchestra.run.smithi115.stderr:min_mon_release 0 (unknown) 2024-03-25T18:26:41.774 INFO:teuthology.orchestra.run.smithi115.stderr:election_strategy: 3 2024-03-25T18:26:41.774 INFO:teuthology.orchestra.run.smithi115.stderr:0: v1:172.21.15.1:6789/0 mon.noname-a 2024-03-25T18:26:41.774 INFO:teuthology.orchestra.run.smithi115.stderr:1: v1:172.21.15.115:6789/0 mon.noname-c 2024-03-25T18:26:41.774 INFO:teuthology.orchestra.run.smithi115.stderr:2: v1:172.21.15.1:6790/0 mon.noname-b 2024-03-25T18:26:41.774 INFO:teuthology.orchestra.run.smithi115.stderr: 2024-03-25T18:26:41.774 INFO:teuthology.orchestra.run.smithi115.stderr:2024-03-25T18:26:41.770+0000 7f6053532640 1 Processor -- start 2024-03-25T18:26:41.774 INFO:teuthology.orchestra.run.smithi115.stderr:2024-03-25T18:26:41.770+0000 7f6053532640 1 -- start start 2024-03-25T18:26:41.774 INFO:teuthology.orchestra.run.smithi115.stderr:2024-03-25T18:26:41.770+0000 7f6053532640 10 monclient: init 2024-03-25T18:26:41.774 INFO:teuthology.orchestra.run.smithi115.stderr:2024-03-25T18:26:41.771+0000 7f6053532640 10 monclient: _reopen_session rank -1 2024-03-25T18:26:41.774 INFO:teuthology.orchestra.run.smithi115.stderr:2024-03-25T18:26:41.771+0000 7f6053532640 10 monclient: _add_conns ranks=[1,2,0] 2024-03-25T18:26:41.774 INFO:teuthology.orchestra.run.smithi115.stderr:2024-03-25T18:26:41.771+0000 7f6053532640 10 monclient(hunting): picked mon.noname-c con 0x7f604c0b4b50 addr v1:172.21.15.115:6789/0 2024-03-25T18:26:41.774 INFO:teuthology.orchestra.run.smithi115.stderr:2024-03-25T18:26:41.771+0000 7f6053532640 10 monclient(hunting): picked mon.noname-b con 0x7f604c13ff20 addr v1:172.21.15.1:6790/0 2024-03-25T18:26:41.775 INFO:teuthology.orchestra.run.smithi115.stderr:2024-03-25T18:26:41.771+0000 7f6053532640 10 monclient(hunting): picked mon.noname-a con 0x7f604c142dd0 addr v1:172.21.15.1:6789/0 2024-03-25T18:26:41.775 INFO:teuthology.orchestra.run.smithi115.stderr:2024-03-25T18:26:41.771+0000 7f6053532640 1 -- --> v1:172.21.15.1:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f604c0ab440 con 0x7f604c142dd0 2024-03-25T18:26:41.775 INFO:teuthology.orchestra.run.smithi115.stderr:2024-03-25T18:26:41.771+0000 7f6053532640 1 -- --> v1:172.21.15.115:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f604c0ab6b0 con 0x7f604c0b4b50 2024-03-25T18:26:41.775 INFO:teuthology.orchestra.run.smithi115.stderr:2024-03-25T18:26:41.771+0000 7f6053532640 1 -- --> v1:172.21.15.1:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f604c0a9f10 con 0x7f604c13ff20 2024-03-25T18:26:41.775 INFO:teuthology.orchestra.run.smithi115.stderr:2024-03-25T18:26:41.771+0000 7f6053532640 10 monclient(hunting): _renew_subs 2024-03-25T18:26:41.775 INFO:teuthology.orchestra.run.smithi115.stderr:2024-03-25T18:26:41.771+0000 7f6053532640 10 monclient(hunting): authenticate will time out at 7492.927423s 2024-03-25T18:26:41.775 INFO:teuthology.orchestra.run.smithi115.stderr:2024-03-25T18:26:41.772+0000 7f6050acd640 1 --1- >> v1:172.21.15.1:6790/0 conn(0x7f604c13ff20 0x7f604c1426c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.1:6790/0 says I am v1:172.21.15.115:47060/0 (socket says 172.21.15.115:47060) 2024-03-25T18:26:41.775 INFO:teuthology.orchestra.run.smithi115.stderr:2024-03-25T18:26:41.772+0000 7f6050acd640 1 -- 172.21.15.115:0/2774751209 learned_addr learned my addr 172.21.15.115:0/2774751209 (peer_addr_for_me v1:172.21.15.115:0/0) 2024-03-25T18:26:41.776 INFO:teuthology.orchestra.run.smithi115.stderr:2024-03-25T18:26:41.774+0000 7f6043fff640 1 -- 172.21.15.115:0/2774751209 <== mon.2 v1:172.21.15.1:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1470769983 0 0) 0x7f604c0a9f10 con 0x7f604c13ff20 2024-03-25T18:26:41.776 INFO:teuthology.orchestra.run.smithi115.stderr:2024-03-25T18:26:41.774+0000 7f6043fff640 10 monclient(hunting): _negotiate 2024-03-25T18:26:41.776 INFO:teuthology.orchestra.run.smithi115.stderr:2024-03-25T18:26:41.774+0000 7f6043fff640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:26:41.776 INFO:teuthology.orchestra.run.smithi115.stderr:2024-03-25T18:26:41.774+0000 7f6043fff640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:26:41.776 INFO:teuthology.orchestra.run.smithi115.stderr:2024-03-25T18:26:41.774+0000 7f6043fff640 10 monclient(hunting): my global_id is 4124 2024-03-25T18:26:41.776 INFO:teuthology.orchestra.run.smithi115.stderr:2024-03-25T18:26:41.774+0000 7f6043fff640 1 -- 172.21.15.115:0/2774751209 --> v1:172.21.15.1:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f6038003520 con 0x7f604c13ff20 2024-03-25T18:26:41.776 INFO:teuthology.orchestra.run.smithi115.stderr:2024-03-25T18:26:41.774+0000 7f6043fff640 1 -- 172.21.15.115:0/2774751209 <== mon.1 v1:172.21.15.115:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3450834383 0 0) 0x7f604c0ab6b0 con 0x7f604c0b4b50 2024-03-25T18:26:41.777 INFO:teuthology.orchestra.run.smithi115.stderr:2024-03-25T18:26:41.774+0000 7f6043fff640 10 monclient(hunting): _negotiate 2024-03-25T18:26:41.777 INFO:teuthology.orchestra.run.smithi115.stderr:2024-03-25T18:26:41.774+0000 7f6043fff640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:26:41.777 INFO:teuthology.orchestra.run.smithi115.stderr:2024-03-25T18:26:41.774+0000 7f6043fff640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:26:41.777 INFO:teuthology.orchestra.run.smithi115.stderr:2024-03-25T18:26:41.774+0000 7f6043fff640 10 monclient(hunting): my global_id is 4120 2024-03-25T18:26:41.777 INFO:teuthology.orchestra.run.smithi115.stderr:2024-03-25T18:26:41.774+0000 7f6043fff640 1 -- 172.21.15.115:0/2774751209 --> v1:172.21.15.115:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f604c0a9f10 con 0x7f604c0b4b50 2024-03-25T18:26:41.777 INFO:teuthology.orchestra.run.smithi115.stderr:2024-03-25T18:26:41.774+0000 7f6043fff640 1 -- 172.21.15.115:0/2774751209 <== mon.0 v1:172.21.15.1:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 195469698 0 0) 0x7f604c0ab440 con 0x7f604c142dd0 2024-03-25T18:26:41.777 INFO:teuthology.orchestra.run.smithi115.stderr:2024-03-25T18:26:41.775+0000 7f6043fff640 10 monclient(hunting): _negotiate 2024-03-25T18:26:41.777 INFO:teuthology.orchestra.run.smithi115.stderr:2024-03-25T18:26:41.775+0000 7f6043fff640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:26:41.777 INFO:teuthology.orchestra.run.smithi115.stderr:2024-03-25T18:26:41.775+0000 7f6043fff640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:26:41.777 INFO:teuthology.orchestra.run.smithi115.stderr:2024-03-25T18:26:41.775+0000 7f6043fff640 10 monclient(hunting): my global_id is 4125 2024-03-25T18:26:41.777 INFO:teuthology.orchestra.run.smithi115.stderr:2024-03-25T18:26:41.775+0000 7f6043fff640 1 -- 172.21.15.115:0/2774751209 --> v1:172.21.15.1:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f604c0ab6b0 con 0x7f604c142dd0 2024-03-25T18:26:41.779 INFO:teuthology.orchestra.run.smithi115.stderr:2024-03-25T18:26:41.777+0000 7f6043fff640 1 -- 172.21.15.115:0/2774751209 <== mon.2 v1:172.21.15.1:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 469403852 0 0) 0x7f6038003520 con 0x7f604c13ff20 2024-03-25T18:26:41.779 INFO:teuthology.orchestra.run.smithi115.stderr:2024-03-25T18:26:41.777+0000 7f6043fff640 1 -- 172.21.15.115:0/2774751209 >> v1:172.21.15.115:6789/0 conn(0x7f604c0b4b50 legacy=0x7f604c13f810 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:26:41.779 INFO:teuthology.orchestra.run.smithi115.stderr:2024-03-25T18:26:41.777+0000 7f6043fff640 1 -- 172.21.15.115:0/2774751209 >> v1:172.21.15.1:6789/0 conn(0x7f604c142dd0 legacy=0x7f604c14df50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:26:41.779 INFO:teuthology.orchestra.run.smithi115.stderr:2024-03-25T18:26:41.777+0000 7f6043fff640 10 monclient: _finish_hunting 0 2024-03-25T18:26:41.779 INFO:teuthology.orchestra.run.smithi115.stderr:2024-03-25T18:26:41.777+0000 7f6043fff640 1 monclient: found mon.noname-b 2024-03-25T18:26:41.779 INFO:teuthology.orchestra.run.smithi115.stderr:2024-03-25T18:26:41.777+0000 7f6043fff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:26:41.779 INFO:teuthology.orchestra.run.smithi115.stderr:2024-03-25T18:26:41.778+0000 7f6043fff640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.1:6790/0 2024-03-25T18:26:41.779 INFO:teuthology.orchestra.run.smithi115.stderr:2024-03-25T18:26:41.778+0000 7f6043fff640 1 -- 172.21.15.115:0/2774751209 --> v1:172.21.15.1:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f604c0aa0a0 con 0x7f604c13ff20 2024-03-25T18:26:41.779 INFO:teuthology.orchestra.run.smithi115.stderr:2024-03-25T18:26:41.778+0000 7f6043fff640 10 monclient: _finish_auth 0 2024-03-25T18:26:41.779 INFO:teuthology.orchestra.run.smithi115.stderr:2024-03-25T18:26:41.778+0000 7f6043fff640 10 monclient: _check_auth_tickets 2024-03-25T18:26:41.779 INFO:teuthology.orchestra.run.smithi115.stderr:2024-03-25T18:26:41.778+0000 7f6043fff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:26:41.780 INFO:teuthology.orchestra.run.smithi115.stderr:2024-03-25T18:26:41.778+0000 7f6053532640 5 monclient: authenticate success, global_id 4124 2024-03-25T18:26:41.780 INFO:teuthology.orchestra.run.smithi115.stderr:2024-03-25T18:26:41.778+0000 7f6043fff640 1 -- 172.21.15.115:0/2774751209 <== mon.2 v1:172.21.15.1:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1586965025 0 0) 0x7f603c001690 con 0x7f604c13ff20 2024-03-25T18:26:41.780 INFO:teuthology.orchestra.run.smithi115.stderr:2024-03-25T18:26:41.778+0000 7f6053532640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T18:26:41.780 INFO:teuthology.orchestra.run.smithi115.stderr:2024-03-25T18:26:41.778+0000 7f6043fff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:26:41.780 INFO:teuthology.orchestra.run.smithi115.stderr:2024-03-25T18:26:41.778+0000 7f6043fff640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-03-25T18:26:41.780 INFO:teuthology.orchestra.run.smithi115.stderr:2024-03-25T18:26:41.778+0000 7f6043fff640 10 monclient: dump: 2024-03-25T18:26:41.780 INFO:teuthology.orchestra.run.smithi115.stderr:epoch 1 2024-03-25T18:26:41.780 INFO:teuthology.orchestra.run.smithi115.stderr:fsid c8ea72cf-9e99-4383-9b52-6c23d365fdfc 2024-03-25T18:26:41.780 INFO:teuthology.orchestra.run.smithi115.stderr:last_changed 2024-03-25T18:25:50.510519+0000 2024-03-25T18:26:41.780 INFO:teuthology.orchestra.run.smithi115.stderr:created 2024-03-25T18:25:50.510519+0000 2024-03-25T18:26:41.780 INFO:teuthology.orchestra.run.smithi115.stderr:min_mon_release 19 (squid) 2024-03-25T18:26:41.780 INFO:teuthology.orchestra.run.smithi115.stderr:election_strategy: 3 2024-03-25T18:26:41.780 INFO:teuthology.orchestra.run.smithi115.stderr:0: v1:172.21.15.1:6789/0 mon.a 2024-03-25T18:26:41.780 INFO:teuthology.orchestra.run.smithi115.stderr:1: v1:172.21.15.115:6789/0 mon.b 2024-03-25T18:26:41.780 INFO:teuthology.orchestra.run.smithi115.stderr:2: v1:172.21.15.1:6790/0 mon.c 2024-03-25T18:26:41.781 INFO:teuthology.orchestra.run.smithi115.stderr: 2024-03-25T18:26:41.781 INFO:teuthology.orchestra.run.smithi115.stderr:2024-03-25T18:26:41.778+0000 7f6053532640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T18:26:41.784 INFO:teuthology.orchestra.run.smithi115.stderr:2024-03-25T18:26:41.779+0000 7f6043fff640 1 -- 172.21.15.115:0/2774751209 <== mon.2 v1:172.21.15.1:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f603c0017f0 con 0x7f604c13ff20 2024-03-25T18:26:41.784 INFO:teuthology.orchestra.run.smithi115.stderr:2024-03-25T18:26:41.779+0000 7f6043fff640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:26:41.784 INFO:teuthology.orchestra.run.smithi115.stderr:2024-03-25T18:26:41.779+0000 7f6053532640 10 monclient: get_monmap_and_config success 2024-03-25T18:26:41.784 INFO:teuthology.orchestra.run.smithi115.stderr:2024-03-25T18:26:41.779+0000 7f6053532640 4 set_mon_vals no callback set 2024-03-25T18:26:41.784 INFO:teuthology.orchestra.run.smithi115.stderr:2024-03-25T18:26:41.779+0000 7f6043fff640 1 -- 172.21.15.115:0/2774751209 <== mon.2 v1:172.21.15.1:6790/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1586965025 0 0) 0x7f603c001b80 con 0x7f604c13ff20 2024-03-25T18:26:41.784 INFO:teuthology.orchestra.run.smithi115.stderr:2024-03-25T18:26:41.779+0000 7f6043fff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:26:41.784 INFO:teuthology.orchestra.run.smithi115.stderr:2024-03-25T18:26:41.779+0000 7f6043fff640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-25T18:26:41.784 INFO:teuthology.orchestra.run.smithi115.stderr:2024-03-25T18:26:41.779+0000 7f6043fff640 10 monclient: dump: 2024-03-25T18:26:41.784 INFO:teuthology.orchestra.run.smithi115.stderr:epoch 1 2024-03-25T18:26:41.784 INFO:teuthology.orchestra.run.smithi115.stderr:fsid c8ea72cf-9e99-4383-9b52-6c23d365fdfc 2024-03-25T18:26:41.784 INFO:teuthology.orchestra.run.smithi115.stderr:last_changed 2024-03-25T18:25:50.510519+0000 2024-03-25T18:26:41.784 INFO:teuthology.orchestra.run.smithi115.stderr:created 2024-03-25T18:25:50.510519+0000 2024-03-25T18:26:41.785 INFO:teuthology.orchestra.run.smithi115.stderr:min_mon_release 19 (squid) 2024-03-25T18:26:41.785 INFO:teuthology.orchestra.run.smithi115.stderr:election_strategy: 3 2024-03-25T18:26:41.785 INFO:teuthology.orchestra.run.smithi115.stderr:0: v1:172.21.15.1:6789/0 mon.a 2024-03-25T18:26:41.785 INFO:teuthology.orchestra.run.smithi115.stderr:1: v1:172.21.15.115:6789/0 mon.b 2024-03-25T18:26:41.785 INFO:teuthology.orchestra.run.smithi115.stderr:2: v1:172.21.15.1:6790/0 mon.c 2024-03-25T18:26:41.785 INFO:teuthology.orchestra.run.smithi115.stderr: 2024-03-25T18:26:41.785 INFO:teuthology.orchestra.run.smithi115.stderr:2024-03-25T18:26:41.780+0000 7f6053532640 10 monclient: shutdown 2024-03-25T18:26:41.785 INFO:teuthology.orchestra.run.smithi115.stderr:2024-03-25T18:26:41.780+0000 7f6053532640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:26:41.785 INFO:teuthology.orchestra.run.smithi115.stderr:2024-03-25T18:26:41.780+0000 7f6053532640 1 -- 172.21.15.115:0/2774751209 >> v1:172.21.15.1:6790/0 conn(0x7f604c13ff20 legacy=0x7f604c1426c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:26:41.785 INFO:teuthology.orchestra.run.smithi115.stderr:2024-03-25T18:26:41.780+0000 7f6053532640 1 -- 172.21.15.115:0/2774751209 shutdown_connections 2024-03-25T18:26:41.785 INFO:teuthology.orchestra.run.smithi115.stderr:2024-03-25T18:26:41.780+0000 7f6053532640 1 -- 172.21.15.115:0/2774751209 >> 172.21.15.115:0/2774751209 conn(0x7f604c066920 msgr2=0x7f604c066d30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:26:41.785 INFO:teuthology.orchestra.run.smithi115.stderr:2024-03-25T18:26:41.780+0000 7f6053532640 1 -- 172.21.15.115:0/2774751209 shutdown_connections 2024-03-25T18:26:41.785 INFO:teuthology.orchestra.run.smithi115.stderr:2024-03-25T18:26:41.780+0000 7f6053532640 1 -- 172.21.15.115:0/2774751209 wait complete. 2024-03-25T18:26:41.785 INFO:teuthology.orchestra.run.smithi115.stderr:2024-03-25T18:26:41.780+0000 7f6053532640 10 monclient: build_initial_monmap 2024-03-25T18:26:41.785 INFO:teuthology.orchestra.run.smithi115.stderr:2024-03-25T18:26:41.780+0000 7f6053532640 10 monclient: monmap: 2024-03-25T18:26:41.785 INFO:teuthology.orchestra.run.smithi115.stderr:epoch 0 2024-03-25T18:26:41.786 INFO:teuthology.orchestra.run.smithi115.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T18:26:41.786 INFO:teuthology.orchestra.run.smithi115.stderr:last_changed 0.000000 2024-03-25T18:26:41.786 INFO:teuthology.orchestra.run.smithi115.stderr:created 0.000000 2024-03-25T18:26:41.786 INFO:teuthology.orchestra.run.smithi115.stderr:min_mon_release 0 (unknown) 2024-03-25T18:26:41.786 INFO:teuthology.orchestra.run.smithi115.stderr:election_strategy: 1 2024-03-25T18:26:41.786 INFO:teuthology.orchestra.run.smithi115.stderr:0: v1:172.21.15.1:6789/0 mon.noname-a 2024-03-25T18:26:41.786 INFO:teuthology.orchestra.run.smithi115.stderr:1: v1:172.21.15.115:6789/0 mon.noname-b 2024-03-25T18:26:41.786 INFO:teuthology.orchestra.run.smithi115.stderr:2: v1:172.21.15.1:6790/0 mon.noname-c 2024-03-25T18:26:41.786 INFO:teuthology.orchestra.run.smithi115.stderr: 2024-03-25T18:26:41.786 INFO:teuthology.orchestra.run.smithi115.stderr:2024-03-25T18:26:41.781+0000 7f6053532640 1 librados: starting msgr at 2024-03-25T18:26:41.786 INFO:teuthology.orchestra.run.smithi115.stderr:2024-03-25T18:26:41.781+0000 7f6053532640 1 librados: starting objecter 2024-03-25T18:26:41.786 INFO:teuthology.orchestra.run.smithi115.stderr:2024-03-25T18:26:41.781+0000 7f6053532640 1 Processor -- start 2024-03-25T18:26:41.786 INFO:teuthology.orchestra.run.smithi115.stderr:2024-03-25T18:26:41.781+0000 7f6053532640 1 -- start start 2024-03-25T18:26:41.786 INFO:teuthology.orchestra.run.smithi115.stderr:2024-03-25T18:26:41.781+0000 7f6053532640 1 librados: setting wanted keys 2024-03-25T18:26:41.787 INFO:teuthology.orchestra.run.smithi115.stderr:2024-03-25T18:26:41.781+0000 7f6053532640 1 librados: calling monclient init 2024-03-25T18:26:41.787 INFO:teuthology.orchestra.run.smithi115.stderr:2024-03-25T18:26:41.781+0000 7f6053532640 10 monclient: init 2024-03-25T18:26:41.787 INFO:teuthology.orchestra.run.smithi115.stderr:2024-03-25T18:26:41.781+0000 7f6053532640 10 monclient: _reopen_session rank -1 2024-03-25T18:26:41.787 INFO:teuthology.orchestra.run.smithi115.stderr:2024-03-25T18:26:41.781+0000 7f6053532640 10 monclient: _add_conns ranks=[2,0,1] 2024-03-25T18:26:41.787 INFO:teuthology.orchestra.run.smithi115.stderr:2024-03-25T18:26:41.781+0000 7f6053532640 10 monclient(hunting): picked mon.noname-c con 0x7f604c0b4b50 addr v1:172.21.15.1:6790/0 2024-03-25T18:26:41.787 INFO:teuthology.orchestra.run.smithi115.stderr:2024-03-25T18:26:41.781+0000 7f6053532640 10 monclient(hunting): picked mon.noname-a con 0x7f604c13ff20 addr v1:172.21.15.1:6789/0 2024-03-25T18:26:41.787 INFO:teuthology.orchestra.run.smithi115.stderr:2024-03-25T18:26:41.781+0000 7f6053532640 10 monclient(hunting): picked mon.noname-b con 0x7f604c142dd0 addr v1:172.21.15.115:6789/0 2024-03-25T18:26:41.787 INFO:teuthology.orchestra.run.smithi115.stderr:2024-03-25T18:26:41.781+0000 7f6053532640 1 -- --> v1:172.21.15.1:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f604c144720 con 0x7f604c13ff20 2024-03-25T18:26:41.787 INFO:teuthology.orchestra.run.smithi115.stderr:2024-03-25T18:26:41.781+0000 7f6053532640 1 -- --> v1:172.21.15.115:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f604c140430 con 0x7f604c142dd0 2024-03-25T18:26:41.787 INFO:teuthology.orchestra.run.smithi115.stderr:2024-03-25T18:26:41.781+0000 7f6053532640 1 -- --> v1:172.21.15.1:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f604c1d77b0 con 0x7f604c0b4b50 2024-03-25T18:26:41.787 INFO:teuthology.orchestra.run.smithi115.stderr:2024-03-25T18:26:41.781+0000 7f6053532640 10 monclient(hunting): _renew_subs 2024-03-25T18:26:41.787 INFO:teuthology.orchestra.run.smithi115.stderr:2024-03-25T18:26:41.781+0000 7f6053532640 10 monclient(hunting): authenticate will time out at 7492.937738s 2024-03-25T18:26:41.787 INFO:teuthology.orchestra.run.smithi115.stderr:2024-03-25T18:26:41.782+0000 7f60512ce640 1 --1- >> v1:172.21.15.1:6790/0 conn(0x7f604c0b4b50 0x7f604c1d7cd0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.1:6790/0 says I am v1:172.21.15.115:47070/0 (socket says 172.21.15.115:47070) 2024-03-25T18:26:41.787 INFO:teuthology.orchestra.run.smithi115.stderr:2024-03-25T18:26:41.782+0000 7f60512ce640 1 -- 172.21.15.115:0/4243093633 learned_addr learned my addr 172.21.15.115:0/4243093633 (peer_addr_for_me v1:172.21.15.115:0/0) 2024-03-25T18:26:41.788 INFO:teuthology.orchestra.run.smithi115.stderr:2024-03-25T18:26:41.782+0000 7f6041ffb640 10 client.?.objecter ms_handle_connect 0x7f604c142dd0 2024-03-25T18:26:41.788 INFO:teuthology.orchestra.run.smithi115.stderr:2024-03-25T18:26:41.782+0000 7f6041ffb640 10 client.?.objecter resend_mon_ops 2024-03-25T18:26:41.788 INFO:teuthology.orchestra.run.smithi115.stderr:2024-03-25T18:26:41.782+0000 7f6041ffb640 10 client.?.objecter ms_handle_connect 0x7f604c0b4b50 2024-03-25T18:26:41.788 INFO:teuthology.orchestra.run.smithi115.stderr:2024-03-25T18:26:41.782+0000 7f6041ffb640 10 client.?.objecter resend_mon_ops 2024-03-25T18:26:41.788 INFO:teuthology.orchestra.run.smithi115.stderr:2024-03-25T18:26:41.782+0000 7f6041ffb640 10 client.?.objecter ms_handle_connect 0x7f604c13ff20 2024-03-25T18:26:41.788 INFO:teuthology.orchestra.run.smithi115.stderr:2024-03-25T18:26:41.782+0000 7f6041ffb640 10 client.?.objecter resend_mon_ops 2024-03-25T18:26:41.788 INFO:tasks.ceph.mgr.y.smithi001.stderr:2024-03-25T18:26:41.787+0000 7faca7441040 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2024-03-25T18:26:41.789 INFO:teuthology.orchestra.run.smithi115.stderr:2024-03-25T18:26:41.784+0000 7f6041ffb640 1 -- 172.21.15.115:0/4243093633 <== mon.2 v1:172.21.15.1:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 565502372 0 0) 0x7f604c1d77b0 con 0x7f604c0b4b50 2024-03-25T18:26:41.789 INFO:teuthology.orchestra.run.smithi115.stderr:2024-03-25T18:26:41.784+0000 7f6041ffb640 10 monclient(hunting): _negotiate 2024-03-25T18:26:41.789 INFO:teuthology.orchestra.run.smithi115.stderr:2024-03-25T18:26:41.784+0000 7f6041ffb640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:26:41.789 INFO:teuthology.orchestra.run.smithi115.stderr:2024-03-25T18:26:41.784+0000 7f6041ffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:26:41.790 INFO:teuthology.orchestra.run.smithi115.stderr:2024-03-25T18:26:41.784+0000 7f6041ffb640 10 monclient(hunting): my global_id is 4127 2024-03-25T18:26:41.790 INFO:teuthology.orchestra.run.smithi115.stderr:2024-03-25T18:26:41.784+0000 7f6041ffb640 1 -- 172.21.15.115:0/4243093633 --> v1:172.21.15.1:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f6030004eb0 con 0x7f604c0b4b50 2024-03-25T18:26:41.790 INFO:teuthology.orchestra.run.smithi115.stderr:2024-03-25T18:26:41.784+0000 7f6041ffb640 1 -- 172.21.15.115:0/4243093633 <== mon.1 v1:172.21.15.115:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1466746578 0 0) 0x7f604c140430 con 0x7f604c142dd0 2024-03-25T18:26:41.790 INFO:teuthology.orchestra.run.smithi115.stderr:2024-03-25T18:26:41.784+0000 7f6041ffb640 10 monclient(hunting): _negotiate 2024-03-25T18:26:41.790 INFO:teuthology.orchestra.run.smithi115.stderr:2024-03-25T18:26:41.784+0000 7f6041ffb640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:26:41.790 INFO:teuthology.orchestra.run.smithi115.stderr:2024-03-25T18:26:41.784+0000 7f6041ffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:26:41.790 INFO:teuthology.orchestra.run.smithi115.stderr:2024-03-25T18:26:41.784+0000 7f6041ffb640 10 monclient(hunting): my global_id is 4123 2024-03-25T18:26:41.790 INFO:teuthology.orchestra.run.smithi115.stderr:2024-03-25T18:26:41.784+0000 7f6041ffb640 1 -- 172.21.15.115:0/4243093633 --> v1:172.21.15.115:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f604c1d77b0 con 0x7f604c142dd0 2024-03-25T18:26:41.790 INFO:teuthology.orchestra.run.smithi115.stderr:2024-03-25T18:26:41.784+0000 7f6041ffb640 1 -- 172.21.15.115:0/4243093633 <== mon.0 v1:172.21.15.1:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2362014999 0 0) 0x7f604c144720 con 0x7f604c13ff20 2024-03-25T18:26:41.790 INFO:teuthology.orchestra.run.smithi115.stderr:2024-03-25T18:26:41.784+0000 7f6041ffb640 10 monclient(hunting): _negotiate 2024-03-25T18:26:41.790 INFO:teuthology.orchestra.run.smithi115.stderr:2024-03-25T18:26:41.784+0000 7f6041ffb640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:26:41.791 INFO:teuthology.orchestra.run.smithi115.stderr:2024-03-25T18:26:41.785+0000 7f6041ffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:26:41.791 INFO:teuthology.orchestra.run.smithi115.stderr:2024-03-25T18:26:41.785+0000 7f6041ffb640 10 monclient(hunting): my global_id is 4128 2024-03-25T18:26:41.791 INFO:teuthology.orchestra.run.smithi115.stderr:2024-03-25T18:26:41.785+0000 7f6041ffb640 1 -- 172.21.15.115:0/4243093633 --> v1:172.21.15.1:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f604c140430 con 0x7f604c13ff20 2024-03-25T18:26:41.792 INFO:teuthology.orchestra.run.smithi115.stderr:2024-03-25T18:26:41.789+0000 7f6041ffb640 1 -- 172.21.15.115:0/4243093633 <== mon.1 v1:172.21.15.115:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3027582657 0 0) 0x7f604c1d77b0 con 0x7f604c142dd0 2024-03-25T18:26:41.792 INFO:teuthology.orchestra.run.smithi115.stderr:2024-03-25T18:26:41.789+0000 7f6041ffb640 1 -- 172.21.15.115:0/4243093633 --> v1:172.21.15.115:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f604c144720 con 0x7f604c142dd0 2024-03-25T18:26:41.792 INFO:teuthology.orchestra.run.smithi115.stderr:2024-03-25T18:26:41.789+0000 7f6041ffb640 1 -- 172.21.15.115:0/4243093633 <== mon.1 v1:172.21.15.115:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1586965025 0 0) 0x7f6044001b50 con 0x7f604c142dd0 2024-03-25T18:26:41.792 INFO:teuthology.orchestra.run.smithi115.stderr:2024-03-25T18:26:41.789+0000 7f6041ffb640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:26:41.793 INFO:teuthology.orchestra.run.smithi115.stderr:2024-03-25T18:26:41.789+0000 7f6041ffb640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-03-25T18:26:41.793 INFO:teuthology.orchestra.run.smithi115.stderr:2024-03-25T18:26:41.789+0000 7f6041ffb640 10 monclient(hunting): dump: 2024-03-25T18:26:41.793 INFO:teuthology.orchestra.run.smithi115.stderr:epoch 1 2024-03-25T18:26:41.793 INFO:teuthology.orchestra.run.smithi115.stderr:fsid c8ea72cf-9e99-4383-9b52-6c23d365fdfc 2024-03-25T18:26:41.793 INFO:teuthology.orchestra.run.smithi115.stderr:last_changed 2024-03-25T18:25:50.510519+0000 2024-03-25T18:26:41.793 INFO:teuthology.orchestra.run.smithi115.stderr:created 2024-03-25T18:25:50.510519+0000 2024-03-25T18:26:41.793 INFO:teuthology.orchestra.run.smithi115.stderr:min_mon_release 19 (squid) 2024-03-25T18:26:41.793 INFO:teuthology.orchestra.run.smithi115.stderr:election_strategy: 3 2024-03-25T18:26:41.793 INFO:teuthology.orchestra.run.smithi115.stderr:0: v1:172.21.15.1:6789/0 mon.a 2024-03-25T18:26:41.793 INFO:teuthology.orchestra.run.smithi115.stderr:1: v1:172.21.15.115:6789/0 mon.b 2024-03-25T18:26:41.793 INFO:teuthology.orchestra.run.smithi115.stderr:2: v1:172.21.15.1:6790/0 mon.c 2024-03-25T18:26:41.793 INFO:teuthology.orchestra.run.smithi115.stderr: 2024-03-25T18:26:41.793 INFO:teuthology.orchestra.run.smithi115.stderr:2024-03-25T18:26:41.789+0000 7f6041ffb640 10 monclient(hunting): _finish_auth 0 2024-03-25T18:26:41.793 INFO:teuthology.orchestra.run.smithi115.stderr:2024-03-25T18:26:41.789+0000 7f6041ffb640 1 -- 172.21.15.115:0/4243093633 <== mon.2 v1:172.21.15.1:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2935562111 0 0) 0x7f6030004eb0 con 0x7f604c0b4b50 2024-03-25T18:26:41.794 INFO:teuthology.orchestra.run.smithi115.stderr:2024-03-25T18:26:41.790+0000 7f6041ffb640 1 -- 172.21.15.115:0/4243093633 --> v1:172.21.15.1:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f604c1d77b0 con 0x7f604c0b4b50 2024-03-25T18:26:41.794 INFO:teuthology.orchestra.run.smithi115.stderr:2024-03-25T18:26:41.790+0000 7f6041ffb640 1 -- 172.21.15.115:0/4243093633 <== mon.2 v1:172.21.15.1:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1586965025 0 0) 0x7f6048001a30 con 0x7f604c0b4b50 2024-03-25T18:26:41.794 INFO:teuthology.orchestra.run.smithi115.stderr:2024-03-25T18:26:41.790+0000 7f6041ffb640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:26:41.794 INFO:teuthology.orchestra.run.smithi115.stderr:2024-03-25T18:26:41.790+0000 7f6041ffb640 10 monclient(hunting): got monmap 1 from mon.c (according to old e1) 2024-03-25T18:26:41.794 INFO:teuthology.orchestra.run.smithi115.stderr:2024-03-25T18:26:41.790+0000 7f6041ffb640 10 monclient(hunting): dump: 2024-03-25T18:26:41.794 INFO:teuthology.orchestra.run.smithi115.stderr:epoch 1 2024-03-25T18:26:41.794 INFO:teuthology.orchestra.run.smithi115.stderr:fsid c8ea72cf-9e99-4383-9b52-6c23d365fdfc 2024-03-25T18:26:41.794 INFO:teuthology.orchestra.run.smithi115.stderr:last_changed 2024-03-25T18:25:50.510519+0000 2024-03-25T18:26:41.794 INFO:teuthology.orchestra.run.smithi115.stderr:created 2024-03-25T18:25:50.510519+0000 2024-03-25T18:26:41.794 INFO:teuthology.orchestra.run.smithi115.stderr:min_mon_release 19 (squid) 2024-03-25T18:26:41.794 INFO:teuthology.orchestra.run.smithi115.stderr:election_strategy: 3 2024-03-25T18:26:41.794 INFO:teuthology.orchestra.run.smithi115.stderr:0: v1:172.21.15.1:6789/0 mon.a 2024-03-25T18:26:41.794 INFO:teuthology.orchestra.run.smithi115.stderr:1: v1:172.21.15.115:6789/0 mon.b 2024-03-25T18:26:41.794 INFO:teuthology.orchestra.run.smithi115.stderr:2: v1:172.21.15.1:6790/0 mon.c 2024-03-25T18:26:41.795 INFO:teuthology.orchestra.run.smithi115.stderr: 2024-03-25T18:26:41.795 INFO:teuthology.orchestra.run.smithi115.stderr:2024-03-25T18:26:41.790+0000 7f6041ffb640 1 -- 172.21.15.115:0/4243093633 <== mon.0 v1:172.21.15.1:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2218728936 0 0) 0x7f604c140430 con 0x7f604c13ff20 2024-03-25T18:26:41.795 INFO:teuthology.orchestra.run.smithi115.stderr:2024-03-25T18:26:41.790+0000 7f6041ffb640 1 -- 172.21.15.115:0/4243093633 --> v1:172.21.15.1:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f6030004eb0 con 0x7f604c13ff20 2024-03-25T18:26:41.795 INFO:teuthology.orchestra.run.smithi115.stderr:2024-03-25T18:26:41.790+0000 7f6041ffb640 1 -- 172.21.15.115:0/4243093633 <== mon.0 v1:172.21.15.1:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1586965025 0 0) 0x7f603c0013b0 con 0x7f604c13ff20 2024-03-25T18:26:41.795 INFO:teuthology.orchestra.run.smithi115.stderr:2024-03-25T18:26:41.790+0000 7f6041ffb640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:26:41.795 INFO:teuthology.orchestra.run.smithi115.stderr:2024-03-25T18:26:41.790+0000 7f6041ffb640 10 monclient(hunting): got monmap 1 from mon.a (according to old e1) 2024-03-25T18:26:41.795 INFO:teuthology.orchestra.run.smithi115.stderr:2024-03-25T18:26:41.790+0000 7f6041ffb640 10 monclient(hunting): dump: 2024-03-25T18:26:41.795 INFO:teuthology.orchestra.run.smithi115.stderr:epoch 1 2024-03-25T18:26:41.795 INFO:teuthology.orchestra.run.smithi115.stderr:fsid c8ea72cf-9e99-4383-9b52-6c23d365fdfc 2024-03-25T18:26:41.795 INFO:teuthology.orchestra.run.smithi115.stderr:last_changed 2024-03-25T18:25:50.510519+0000 2024-03-25T18:26:41.795 INFO:teuthology.orchestra.run.smithi115.stderr:created 2024-03-25T18:25:50.510519+0000 2024-03-25T18:26:41.795 INFO:teuthology.orchestra.run.smithi115.stderr:min_mon_release 19 (squid) 2024-03-25T18:26:41.795 INFO:teuthology.orchestra.run.smithi115.stderr:election_strategy: 3 2024-03-25T18:26:41.795 INFO:teuthology.orchestra.run.smithi115.stderr:0: v1:172.21.15.1:6789/0 mon.a 2024-03-25T18:26:41.795 INFO:teuthology.orchestra.run.smithi115.stderr:1: v1:172.21.15.115:6789/0 mon.b 2024-03-25T18:26:41.796 INFO:teuthology.orchestra.run.smithi115.stderr:2: v1:172.21.15.1:6790/0 mon.c 2024-03-25T18:26:41.796 INFO:teuthology.orchestra.run.smithi115.stderr: 2024-03-25T18:26:41.797 INFO:teuthology.orchestra.run.smithi115.stderr:2024-03-25T18:26:41.792+0000 7f6041ffb640 1 -- 172.21.15.115:0/4243093633 <== mon.2 v1:172.21.15.1:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 1197033133 0 0) 0x7f604c1d77b0 con 0x7f604c0b4b50 2024-03-25T18:26:41.797 INFO:teuthology.orchestra.run.smithi115.stderr:2024-03-25T18:26:41.792+0000 7f6041ffb640 1 -- 172.21.15.115:0/4243093633 >> v1:172.21.15.115:6789/0 conn(0x7f604c142dd0 legacy=0x7f604c1e2f30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:26:41.797 INFO:teuthology.orchestra.run.smithi115.stderr:2024-03-25T18:26:41.792+0000 7f6041ffb640 1 -- 172.21.15.115:0/4243093633 >> v1:172.21.15.1:6789/0 conn(0x7f604c13ff20 legacy=0x7f604c145f10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:26:41.797 INFO:teuthology.orchestra.run.smithi115.stderr:2024-03-25T18:26:41.792+0000 7f6041ffb640 10 monclient: _finish_hunting 0 2024-03-25T18:26:41.797 INFO:teuthology.orchestra.run.smithi115.stderr:2024-03-25T18:26:41.792+0000 7f6041ffb640 1 monclient: found mon.c 2024-03-25T18:26:41.797 INFO:teuthology.orchestra.run.smithi115.stderr:2024-03-25T18:26:41.792+0000 7f6041ffb640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:26:41.797 INFO:teuthology.orchestra.run.smithi115.stderr:2024-03-25T18:26:41.792+0000 7f6041ffb640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.1:6790/0 2024-03-25T18:26:41.797 INFO:teuthology.orchestra.run.smithi115.stderr:2024-03-25T18:26:41.792+0000 7f6041ffb640 1 -- 172.21.15.115:0/4243093633 --> v1:172.21.15.1:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f604c1d83e0 con 0x7f604c0b4b50 2024-03-25T18:26:41.797 INFO:teuthology.orchestra.run.smithi115.stderr:2024-03-25T18:26:41.792+0000 7f6041ffb640 10 monclient: _finish_auth 0 2024-03-25T18:26:41.797 INFO:teuthology.orchestra.run.smithi115.stderr:2024-03-25T18:26:41.792+0000 7f6041ffb640 10 monclient: _check_auth_tickets 2024-03-25T18:26:41.798 INFO:teuthology.orchestra.run.smithi115.stderr:2024-03-25T18:26:41.792+0000 7f6041ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:26:41.798 INFO:teuthology.orchestra.run.smithi115.stderr:2024-03-25T18:26:41.792+0000 7f6053532640 5 monclient: authenticate success, global_id 4127 2024-03-25T18:26:41.798 INFO:teuthology.orchestra.run.smithi115.stderr:2024-03-25T18:26:41.792+0000 7f6053532640 10 monclient: _renew_subs 2024-03-25T18:26:41.798 INFO:teuthology.orchestra.run.smithi115.stderr:2024-03-25T18:26:41.792+0000 7f6053532640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.1:6790/0 2024-03-25T18:26:41.798 INFO:teuthology.orchestra.run.smithi115.stderr:2024-03-25T18:26:41.792+0000 7f6053532640 1 -- 172.21.15.115:0/4243093633 --> v1:172.21.15.1:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f604c137ee0 con 0x7f604c0b4b50 2024-03-25T18:26:41.798 INFO:teuthology.orchestra.run.smithi115.stderr:2024-03-25T18:26:41.793+0000 7f6053532640 10 client.4127.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T18:26:41.798 INFO:teuthology.orchestra.run.smithi115.stderr:2024-03-25T18:26:41.793+0000 7f6053532640 10 monclient: _renew_subs 2024-03-25T18:26:41.798 INFO:teuthology.orchestra.run.smithi115.stderr:2024-03-25T18:26:41.793+0000 7f6053532640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.1:6790/0 2024-03-25T18:26:41.798 INFO:teuthology.orchestra.run.smithi115.stderr:2024-03-25T18:26:41.793+0000 7f6053532640 1 -- 172.21.15.115:0/4243093633 --> v1:172.21.15.1:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f604c1411d0 con 0x7f604c0b4b50 2024-03-25T18:26:41.798 INFO:teuthology.orchestra.run.smithi115.stderr:2024-03-25T18:26:41.793+0000 7f6053532640 1 librados: init done 2024-03-25T18:26:41.799 INFO:teuthology.orchestra.run.smithi115.stderr:2024-03-25T18:26:41.794+0000 7f6053532640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T18:26:41.799 INFO:teuthology.orchestra.run.smithi115.stderr:2024-03-25T18:26:41.794+0000 7f6053532640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:26:41.799 INFO:teuthology.orchestra.run.smithi115.stderr:2024-03-25T18:26:41.794+0000 7f6053532640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.1:6790/0 2024-03-25T18:26:41.799 INFO:teuthology.orchestra.run.smithi115.stderr:2024-03-25T18:26:41.794+0000 7f6053532640 1 -- 172.21.15.115:0/4243093633 --> v1:172.21.15.1:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f604c1d7610 con 0x7f604c0b4b50 2024-03-25T18:26:41.844 INFO:tasks.ceph.mgr.y.smithi001.stderr:2024-03-25T18:26:41.843+0000 7faca7441040 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2024-03-25T18:26:41.861 INFO:tasks.ceph.mgr.x.smithi115.stderr:2024-03-25T18:26:41.860+0000 7f99dd247040 -1 mgr[py] Module rook has missing NOTIFY_TYPES member 2024-03-25T18:26:41.894 INFO:tasks.ceph.mgr.y.smithi001.stderr:2024-03-25T18:26:41.892+0000 7faca7441040 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2024-03-25T18:26:41.995 INFO:teuthology.orchestra.run.smithi115.stderr:2024-03-25T18:26:41.994+0000 7f6041ffb640 1 -- 172.21.15.115:0/4243093633 <== mon.2 v1:172.21.15.1:6790/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f6048001c50 con 0x7f604c0b4b50 2024-03-25T18:26:41.995 INFO:teuthology.orchestra.run.smithi115.stderr:2024-03-25T18:26:41.994+0000 7f6041ffb640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:26:41.995 INFO:teuthology.orchestra.run.smithi115.stderr:2024-03-25T18:26:41.994+0000 7f6041ffb640 1 -- 172.21.15.115:0/4243093633 <== mon.2 v1:172.21.15.1:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1586965025 0 0) 0x7f60480038c0 con 0x7f604c0b4b50 2024-03-25T18:26:41.995 INFO:teuthology.orchestra.run.smithi115.stderr:2024-03-25T18:26:41.994+0000 7f6041ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:26:41.995 INFO:teuthology.orchestra.run.smithi115.stderr:2024-03-25T18:26:41.994+0000 7f6041ffb640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-25T18:26:41.995 INFO:teuthology.orchestra.run.smithi115.stderr:2024-03-25T18:26:41.994+0000 7f6041ffb640 10 monclient: dump: 2024-03-25T18:26:41.995 INFO:teuthology.orchestra.run.smithi115.stderr:epoch 1 2024-03-25T18:26:41.996 INFO:teuthology.orchestra.run.smithi115.stderr:fsid c8ea72cf-9e99-4383-9b52-6c23d365fdfc 2024-03-25T18:26:41.996 INFO:teuthology.orchestra.run.smithi115.stderr:last_changed 2024-03-25T18:25:50.510519+0000 2024-03-25T18:26:41.996 INFO:teuthology.orchestra.run.smithi115.stderr:created 2024-03-25T18:25:50.510519+0000 2024-03-25T18:26:41.996 INFO:teuthology.orchestra.run.smithi115.stderr:min_mon_release 19 (squid) 2024-03-25T18:26:41.996 INFO:teuthology.orchestra.run.smithi115.stderr:election_strategy: 3 2024-03-25T18:26:41.996 INFO:teuthology.orchestra.run.smithi115.stderr:0: v1:172.21.15.1:6789/0 mon.a 2024-03-25T18:26:41.996 INFO:teuthology.orchestra.run.smithi115.stderr:1: v1:172.21.15.115:6789/0 mon.b 2024-03-25T18:26:41.996 INFO:teuthology.orchestra.run.smithi115.stderr:2: v1:172.21.15.1:6790/0 mon.c 2024-03-25T18:26:41.996 INFO:teuthology.orchestra.run.smithi115.stderr: 2024-03-25T18:26:41.996 INFO:teuthology.orchestra.run.smithi115.stderr:2024-03-25T18:26:41.994+0000 7f6041ffb640 1 -- 172.21.15.115:0/4243093633 <== mon.2 v1:172.21.15.1:6790/0 7 ==== mgrmap(e 1) v1 ==== 807+0+0 (unknown 3189685207 0 0) 0x7f6048003e70 con 0x7f604c0b4b50 2024-03-25T18:26:41.996 INFO:teuthology.orchestra.run.smithi115.stderr:2024-03-25T18:26:41.994+0000 7f60437fe640 4 set_mon_vals no callback set 2024-03-25T18:26:41.997 INFO:tasks.ceph.mgr.x.smithi115.stderr:2024-03-25T18:26:41.995+0000 7f99dd247040 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2024-03-25T18:26:41.997 INFO:teuthology.orchestra.run.smithi115.stderr:2024-03-25T18:26:41.996+0000 7f6041ffb640 1 -- 172.21.15.115:0/4243093633 <== mon.2 v1:172.21.15.1:6790/0 8 ==== osd_map(4..4 src has 1..4) v4 ==== 955+0+0 (unknown 3677838266 0 0) 0x7f60480064b0 con 0x7f604c0b4b50 2024-03-25T18:26:41.997 INFO:teuthology.orchestra.run.smithi115.stderr:2024-03-25T18:26:41.996+0000 7f6041ffb640 10 client.4127.objecter ms_dispatch 0x7f604c000b90 osd_map(4..4 src has 1..4) v4 2024-03-25T18:26:41.997 INFO:teuthology.orchestra.run.smithi115.stderr:2024-03-25T18:26:41.996+0000 7f6041ffb640 3 client.4127.objecter handle_osd_map got epochs [4,4] > 0 2024-03-25T18:26:41.998 INFO:teuthology.orchestra.run.smithi115.stderr:2024-03-25T18:26:41.996+0000 7f6041ffb640 3 client.4127.objecter handle_osd_map decoding full epoch 4 2024-03-25T18:26:41.998 INFO:teuthology.orchestra.run.smithi115.stderr:2024-03-25T18:26:41.997+0000 7f6041ffb640 20 client.4127.objecter dump_active .. 0 homeless 2024-03-25T18:26:41.998 INFO:teuthology.orchestra.run.smithi115.stderr:2024-03-25T18:26:41.997+0000 7f6041ffb640 1 -- 172.21.15.115:0/4243093633 <== mon.2 v1:172.21.15.1:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+73141 (unknown 1092875540 0 2031569661) 0x7f6048004350 con 0x7f604c0b4b50 2024-03-25T18:26:41.998 INFO:teuthology.orchestra.run.smithi115.stderr:2024-03-25T18:26:41.997+0000 7f6041ffb640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:26:41.998 INFO:teuthology.orchestra.run.smithi115.stderr:2024-03-25T18:26:41.997+0000 7f6041ffb640 10 monclient: _finish_command 1 = system:0 2024-03-25T18:26:42.071 INFO:teuthology.orchestra.run.smithi115.stderr:2024-03-25T18:26:42.070+0000 7f6053532640 10 start_mon_command cmd=[{"prefix": "osd new", "uuid": "292cb23a-18a9-4207-a7cc-4d5727feaec0", "id": 2}] 2024-03-25T18:26:42.071 INFO:teuthology.orchestra.run.smithi115.stderr:2024-03-25T18:26:42.070+0000 7f6053532640 10 monclient: _send_command 2 [{"prefix": "osd new", "uuid": "292cb23a-18a9-4207-a7cc-4d5727feaec0", "id": 2}] 2024-03-25T18:26:42.071 INFO:teuthology.orchestra.run.smithi115.stderr:2024-03-25T18:26:42.070+0000 7f6053532640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.1:6790/0 2024-03-25T18:26:42.071 INFO:teuthology.orchestra.run.smithi115.stderr:2024-03-25T18:26:42.070+0000 7f6053532640 1 -- 172.21.15.115:0/4243093633 --> v1:172.21.15.1:6790/0 -- mon_command({"prefix": "osd new", "uuid": "292cb23a-18a9-4207-a7cc-4d5727feaec0", "id": 2} v 0) v1 -- 0x7f604c04f730 con 0x7f604c0b4b50 2024-03-25T18:26:42.113 INFO:tasks.ceph.mgr.x.smithi115.stderr:2024-03-25T18:26:42.112+0000 7f99dd247040 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2024-03-25T18:26:42.159 INFO:tasks.ceph.mgr.x.smithi115.stderr:2024-03-25T18:26:42.158+0000 7f99dd247040 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2024-03-25T18:26:42.183 INFO:teuthology.orchestra.run.smithi115.stderr:2024-03-25T18:26:42.182+0000 7f6041ffb640 1 -- 172.21.15.115:0/4243093633 <== mon.2 v1:172.21.15.1:6790/0 10 ==== mon_command_ack([{"prefix": "osd new", "uuid": "292cb23a-18a9-4207-a7cc-4d5727feaec0", "id": 2}]=0 v5) v1 ==== 112+0+2 (unknown 3359372401 0 216599125) 0x7f6048006160 con 0x7f604c0b4b50 2024-03-25T18:26:42.183 INFO:teuthology.orchestra.run.smithi115.stderr:2024-03-25T18:26:42.182+0000 7f6041ffb640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd new", "uuid": "292cb23a-18a9-4207-a7cc-4d5727feaec0", "id": 2}] 2024-03-25T18:26:42.183 INFO:teuthology.orchestra.run.smithi115.stderr:2024-03-25T18:26:42.182+0000 7f6041ffb640 10 monclient: _finish_command 2 = system:0 2024-03-25T18:26:42.184 INFO:teuthology.orchestra.run.smithi115.stdout:2 2024-03-25T18:26:42.185 INFO:teuthology.orchestra.run.smithi115.stderr:2024-03-25T18:26:42.183+0000 7f6053532640 10 librados: watch_flush enter 2024-03-25T18:26:42.185 INFO:teuthology.orchestra.run.smithi115.stderr:2024-03-25T18:26:42.183+0000 7f6053532640 10 librados: watch_flush exit 2024-03-25T18:26:42.185 INFO:teuthology.orchestra.run.smithi115.stderr:2024-03-25T18:26:42.183+0000 7f6053532640 20 client.4127.objecter shutdown clearing up homeless session... 2024-03-25T18:26:42.185 INFO:teuthology.orchestra.run.smithi115.stderr:2024-03-25T18:26:42.183+0000 7f6053532640 10 client.4127.objecter successfully canceled tick 2024-03-25T18:26:42.185 INFO:teuthology.orchestra.run.smithi115.stderr:2024-03-25T18:26:42.183+0000 7f6053532640 10 monclient: shutdown 2024-03-25T18:26:42.185 INFO:teuthology.orchestra.run.smithi115.stderr:2024-03-25T18:26:42.183+0000 7f6053532640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:26:42.185 INFO:teuthology.orchestra.run.smithi115.stderr:2024-03-25T18:26:42.183+0000 7f6053532640 1 -- 172.21.15.115:0/4243093633 >> v1:172.21.15.1:6790/0 conn(0x7f604c0b4b50 legacy=0x7f604c1d7cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:26:42.185 INFO:teuthology.orchestra.run.smithi115.stderr:2024-03-25T18:26:42.183+0000 7f6053532640 1 -- 172.21.15.115:0/4243093633 shutdown_connections 2024-03-25T18:26:42.186 INFO:teuthology.orchestra.run.smithi115.stderr:2024-03-25T18:26:42.183+0000 7f6053532640 1 -- 172.21.15.115:0/4243093633 >> 172.21.15.115:0/4243093633 conn(0x7f604c066920 msgr2=0x7f604c13f5e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:26:42.186 INFO:teuthology.orchestra.run.smithi115.stderr:2024-03-25T18:26:42.183+0000 7f6053532640 1 -- 172.21.15.115:0/4243093633 shutdown_connections 2024-03-25T18:26:42.186 INFO:teuthology.orchestra.run.smithi115.stderr:2024-03-25T18:26:42.183+0000 7f6053532640 1 -- 172.21.15.115:0/4243093633 wait complete. 2024-03-25T18:26:42.186 INFO:teuthology.orchestra.run.smithi115.stderr:2024-03-25T18:26:42.183+0000 7f6053532640 1 librados: shutdown 2024-03-25T18:26:42.194 DEBUG:teuthology.orchestra.run.smithi115:> sudo ceph --cluster ceph osd new 45e0e9ab-898d-4673-9590-b4117bc5e84a 3 2024-03-25T18:26:42.278 INFO:tasks.ceph.mgr.x.smithi115.stderr:2024-03-25T18:26:42.277+0000 7f99dd247040 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2024-03-25T18:26:42.294 INFO:teuthology.orchestra.run.smithi115.stderr:2024-03-25T18:26:42.292+0000 7f858bb0e640 10 monclient: get_monmap_and_config 2024-03-25T18:26:42.294 INFO:teuthology.orchestra.run.smithi115.stderr:2024-03-25T18:26:42.292+0000 7f858bb0e640 10 monclient: build_initial_monmap 2024-03-25T18:26:42.294 INFO:teuthology.orchestra.run.smithi115.stderr:2024-03-25T18:26:42.292+0000 7f858bb0e640 10 monclient: monmap: 2024-03-25T18:26:42.294 INFO:teuthology.orchestra.run.smithi115.stderr:epoch 0 2024-03-25T18:26:42.294 INFO:teuthology.orchestra.run.smithi115.stderr:fsid c8ea72cf-9e99-4383-9b52-6c23d365fdfc 2024-03-25T18:26:42.294 INFO:teuthology.orchestra.run.smithi115.stderr:last_changed 2024-03-25T18:26:42.293857+0000 2024-03-25T18:26:42.294 INFO:teuthology.orchestra.run.smithi115.stderr:created 2024-03-25T18:26:42.293857+0000 2024-03-25T18:26:42.294 INFO:teuthology.orchestra.run.smithi115.stderr:min_mon_release 0 (unknown) 2024-03-25T18:26:42.294 INFO:teuthology.orchestra.run.smithi115.stderr:election_strategy: 3 2024-03-25T18:26:42.294 INFO:teuthology.orchestra.run.smithi115.stderr:0: v1:172.21.15.1:6789/0 mon.noname-a 2024-03-25T18:26:42.295 INFO:teuthology.orchestra.run.smithi115.stderr:1: v1:172.21.15.115:6789/0 mon.noname-c 2024-03-25T18:26:42.295 INFO:teuthology.orchestra.run.smithi115.stderr:2: v1:172.21.15.1:6790/0 mon.noname-b 2024-03-25T18:26:42.295 INFO:teuthology.orchestra.run.smithi115.stderr: 2024-03-25T18:26:42.295 INFO:teuthology.orchestra.run.smithi115.stderr:2024-03-25T18:26:42.293+0000 7f858bb0e640 1 Processor -- start 2024-03-25T18:26:42.296 INFO:teuthology.orchestra.run.smithi115.stderr:2024-03-25T18:26:42.293+0000 7f858bb0e640 1 -- start start 2024-03-25T18:26:42.296 INFO:teuthology.orchestra.run.smithi115.stderr:2024-03-25T18:26:42.293+0000 7f858bb0e640 10 monclient: init 2024-03-25T18:26:42.296 INFO:teuthology.orchestra.run.smithi115.stderr:2024-03-25T18:26:42.294+0000 7f858bb0e640 10 monclient: _reopen_session rank -1 2024-03-25T18:26:42.296 INFO:teuthology.orchestra.run.smithi115.stderr:2024-03-25T18:26:42.294+0000 7f858bb0e640 10 monclient: _add_conns ranks=[0,2,1] 2024-03-25T18:26:42.296 INFO:teuthology.orchestra.run.smithi115.stderr:2024-03-25T18:26:42.294+0000 7f858bb0e640 10 monclient(hunting): picked mon.noname-a con 0x7f85840b4b50 addr v1:172.21.15.1:6789/0 2024-03-25T18:26:42.296 INFO:teuthology.orchestra.run.smithi115.stderr:2024-03-25T18:26:42.294+0000 7f858bb0e640 10 monclient(hunting): picked mon.noname-b con 0x7f858413ff20 addr v1:172.21.15.1:6790/0 2024-03-25T18:26:42.296 INFO:teuthology.orchestra.run.smithi115.stderr:2024-03-25T18:26:42.294+0000 7f858bb0e640 10 monclient(hunting): picked mon.noname-c con 0x7f8584142dd0 addr v1:172.21.15.115:6789/0 2024-03-25T18:26:42.296 INFO:teuthology.orchestra.run.smithi115.stderr:2024-03-25T18:26:42.294+0000 7f858bb0e640 1 -- --> v1:172.21.15.1:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f85840ab440 con 0x7f85840b4b50 2024-03-25T18:26:42.296 INFO:teuthology.orchestra.run.smithi115.stderr:2024-03-25T18:26:42.294+0000 7f858bb0e640 1 -- --> v1:172.21.15.115:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f85840ab6b0 con 0x7f8584142dd0 2024-03-25T18:26:42.297 INFO:teuthology.orchestra.run.smithi115.stderr:2024-03-25T18:26:42.294+0000 7f858bb0e640 1 -- --> v1:172.21.15.1:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f85840a9f10 con 0x7f858413ff20 2024-03-25T18:26:42.297 INFO:teuthology.orchestra.run.smithi115.stderr:2024-03-25T18:26:42.294+0000 7f858bb0e640 10 monclient(hunting): _renew_subs 2024-03-25T18:26:42.297 INFO:teuthology.orchestra.run.smithi115.stderr:2024-03-25T18:26:42.294+0000 7f858bb0e640 10 monclient(hunting): authenticate will time out at 7493.450156s 2024-03-25T18:26:42.297 INFO:teuthology.orchestra.run.smithi115.stderr:2024-03-25T18:26:42.294+0000 7f858a0ab640 1 --1- >> v1:172.21.15.115:6789/0 conn(0x7f8584142dd0 0x7f858414df50 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.115:6789/0 says I am v1:172.21.15.115:44598/0 (socket says 172.21.15.115:44598) 2024-03-25T18:26:42.297 INFO:teuthology.orchestra.run.smithi115.stderr:2024-03-25T18:26:42.294+0000 7f858a0ab640 1 -- 172.21.15.115:0/231228671 learned_addr learned my addr 172.21.15.115:0/231228671 (peer_addr_for_me v1:172.21.15.115:0/0) 2024-03-25T18:26:42.297 INFO:tasks.ceph.mgr.y.smithi001.stderr:2024-03-25T18:26:42.296+0000 7faca7441040 -1 mgr[py] Module rook has missing NOTIFY_TYPES member 2024-03-25T18:26:42.298 INFO:teuthology.orchestra.run.smithi115.stderr:2024-03-25T18:26:42.297+0000 7f85888a8640 1 -- 172.21.15.115:0/231228671 <== mon.1 v1:172.21.15.115:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1807138502 0 0) 0x7f85840ab6b0 con 0x7f8584142dd0 2024-03-25T18:26:42.298 INFO:teuthology.orchestra.run.smithi115.stderr:2024-03-25T18:26:42.297+0000 7f85888a8640 10 monclient(hunting): _negotiate 2024-03-25T18:26:42.298 INFO:teuthology.orchestra.run.smithi115.stderr:2024-03-25T18:26:42.297+0000 7f85888a8640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:26:42.298 INFO:teuthology.orchestra.run.smithi115.stderr:2024-03-25T18:26:42.297+0000 7f85888a8640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:26:42.298 INFO:teuthology.orchestra.run.smithi115.stderr:2024-03-25T18:26:42.297+0000 7f85888a8640 10 monclient(hunting): my global_id is 4126 2024-03-25T18:26:42.298 INFO:teuthology.orchestra.run.smithi115.stderr:2024-03-25T18:26:42.297+0000 7f85888a8640 1 -- 172.21.15.115:0/231228671 --> v1:172.21.15.115:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f8570003520 con 0x7f8584142dd0 2024-03-25T18:26:42.299 INFO:teuthology.orchestra.run.smithi115.stderr:2024-03-25T18:26:42.297+0000 7f85888a8640 1 -- 172.21.15.115:0/231228671 <== mon.0 v1:172.21.15.1:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3436532141 0 0) 0x7f85840ab440 con 0x7f85840b4b50 2024-03-25T18:26:42.299 INFO:teuthology.orchestra.run.smithi115.stderr:2024-03-25T18:26:42.297+0000 7f85888a8640 10 monclient(hunting): _negotiate 2024-03-25T18:26:42.299 INFO:teuthology.orchestra.run.smithi115.stderr:2024-03-25T18:26:42.297+0000 7f85888a8640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:26:42.299 INFO:teuthology.orchestra.run.smithi115.stderr:2024-03-25T18:26:42.297+0000 7f85888a8640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:26:42.299 INFO:teuthology.orchestra.run.smithi115.stderr:2024-03-25T18:26:42.297+0000 7f85888a8640 10 monclient(hunting): my global_id is 4131 2024-03-25T18:26:42.299 INFO:teuthology.orchestra.run.smithi115.stderr:2024-03-25T18:26:42.297+0000 7f85888a8640 1 -- 172.21.15.115:0/231228671 --> v1:172.21.15.1:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f85840ab6b0 con 0x7f85840b4b50 2024-03-25T18:26:42.299 INFO:teuthology.orchestra.run.smithi115.stderr:2024-03-25T18:26:42.298+0000 7f85888a8640 1 -- 172.21.15.115:0/231228671 <== mon.2 v1:172.21.15.1:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2580942708 0 0) 0x7f85840a9f10 con 0x7f858413ff20 2024-03-25T18:26:42.299 INFO:teuthology.orchestra.run.smithi115.stderr:2024-03-25T18:26:42.298+0000 7f85888a8640 10 monclient(hunting): _negotiate 2024-03-25T18:26:42.300 INFO:teuthology.orchestra.run.smithi115.stderr:2024-03-25T18:26:42.298+0000 7f85888a8640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:26:42.300 INFO:teuthology.orchestra.run.smithi115.stderr:2024-03-25T18:26:42.298+0000 7f85888a8640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:26:42.300 INFO:teuthology.orchestra.run.smithi115.stderr:2024-03-25T18:26:42.298+0000 7f85888a8640 10 monclient(hunting): my global_id is 4130 2024-03-25T18:26:42.300 INFO:teuthology.orchestra.run.smithi115.stderr:2024-03-25T18:26:42.298+0000 7f85888a8640 1 -- 172.21.15.115:0/231228671 --> v1:172.21.15.1:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f85840ab440 con 0x7f858413ff20 2024-03-25T18:26:42.301 INFO:teuthology.orchestra.run.smithi115.stderr:2024-03-25T18:26:42.300+0000 7f85888a8640 1 -- 172.21.15.115:0/231228671 <== mon.1 v1:172.21.15.115:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2244680820 0 0) 0x7f8570003520 con 0x7f8584142dd0 2024-03-25T18:26:42.301 INFO:teuthology.orchestra.run.smithi115.stderr:2024-03-25T18:26:42.300+0000 7f85888a8640 1 -- 172.21.15.115:0/231228671 >> v1:172.21.15.1:6790/0 conn(0x7f858413ff20 legacy=0x7f85841426c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:26:42.301 INFO:teuthology.orchestra.run.smithi115.stderr:2024-03-25T18:26:42.300+0000 7f85888a8640 1 -- 172.21.15.115:0/231228671 >> v1:172.21.15.1:6789/0 conn(0x7f85840b4b50 legacy=0x7f858413f810 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:26:42.301 INFO:teuthology.orchestra.run.smithi115.stderr:2024-03-25T18:26:42.300+0000 7f85888a8640 10 monclient: _finish_hunting 0 2024-03-25T18:26:42.301 INFO:teuthology.orchestra.run.smithi115.stderr:2024-03-25T18:26:42.300+0000 7f85888a8640 1 monclient: found mon.noname-c 2024-03-25T18:26:42.301 INFO:teuthology.orchestra.run.smithi115.stderr:2024-03-25T18:26:42.300+0000 7f85888a8640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:26:42.302 INFO:teuthology.orchestra.run.smithi115.stderr:2024-03-25T18:26:42.300+0000 7f85888a8640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.115:6789/0 2024-03-25T18:26:42.302 INFO:teuthology.orchestra.run.smithi115.stderr:2024-03-25T18:26:42.300+0000 7f85888a8640 1 -- 172.21.15.115:0/231228671 --> v1:172.21.15.115:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f85840aa0a0 con 0x7f8584142dd0 2024-03-25T18:26:42.302 INFO:teuthology.orchestra.run.smithi115.stderr:2024-03-25T18:26:42.300+0000 7f85888a8640 10 monclient: _finish_auth 0 2024-03-25T18:26:42.302 INFO:teuthology.orchestra.run.smithi115.stderr:2024-03-25T18:26:42.300+0000 7f85888a8640 10 monclient: _check_auth_tickets 2024-03-25T18:26:42.302 INFO:teuthology.orchestra.run.smithi115.stderr:2024-03-25T18:26:42.300+0000 7f85888a8640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:26:42.302 INFO:teuthology.orchestra.run.smithi115.stderr:2024-03-25T18:26:42.300+0000 7f85888a8640 1 -- 172.21.15.115:0/231228671 <== mon.1 v1:172.21.15.115:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1586965025 0 0) 0x7f857c001a10 con 0x7f8584142dd0 2024-03-25T18:26:42.302 INFO:teuthology.orchestra.run.smithi115.stderr:2024-03-25T18:26:42.300+0000 7f85888a8640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:26:42.302 INFO:teuthology.orchestra.run.smithi115.stderr:2024-03-25T18:26:42.300+0000 7f85888a8640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-03-25T18:26:42.302 INFO:teuthology.orchestra.run.smithi115.stderr:2024-03-25T18:26:42.300+0000 7f85888a8640 10 monclient: dump: 2024-03-25T18:26:42.302 INFO:teuthology.orchestra.run.smithi115.stderr:epoch 1 2024-03-25T18:26:42.302 INFO:teuthology.orchestra.run.smithi115.stderr:fsid c8ea72cf-9e99-4383-9b52-6c23d365fdfc 2024-03-25T18:26:42.302 INFO:teuthology.orchestra.run.smithi115.stderr:last_changed 2024-03-25T18:25:50.510519+0000 2024-03-25T18:26:42.302 INFO:teuthology.orchestra.run.smithi115.stderr:created 2024-03-25T18:25:50.510519+0000 2024-03-25T18:26:42.303 INFO:teuthology.orchestra.run.smithi115.stderr:min_mon_release 19 (squid) 2024-03-25T18:26:42.303 INFO:teuthology.orchestra.run.smithi115.stderr:election_strategy: 3 2024-03-25T18:26:42.303 INFO:teuthology.orchestra.run.smithi115.stderr:0: v1:172.21.15.1:6789/0 mon.a 2024-03-25T18:26:42.303 INFO:teuthology.orchestra.run.smithi115.stderr:1: v1:172.21.15.115:6789/0 mon.b 2024-03-25T18:26:42.303 INFO:teuthology.orchestra.run.smithi115.stderr:2: v1:172.21.15.1:6790/0 mon.c 2024-03-25T18:26:42.303 INFO:teuthology.orchestra.run.smithi115.stderr: 2024-03-25T18:26:42.303 INFO:teuthology.orchestra.run.smithi115.stderr:2024-03-25T18:26:42.300+0000 7f858bb0e640 5 monclient: authenticate success, global_id 4126 2024-03-25T18:26:42.303 INFO:teuthology.orchestra.run.smithi115.stderr:2024-03-25T18:26:42.300+0000 7f858bb0e640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T18:26:42.327 INFO:teuthology.orchestra.run.smithi115.stderr:2024-03-25T18:26:42.326+0000 7f85888a8640 1 -- 172.21.15.115:0/231228671 <== mon.1 v1:172.21.15.115:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f857c001bf0 con 0x7f8584142dd0 2024-03-25T18:26:42.327 INFO:teuthology.orchestra.run.smithi115.stderr:2024-03-25T18:26:42.326+0000 7f85888a8640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:26:42.327 INFO:teuthology.orchestra.run.smithi115.stderr:2024-03-25T18:26:42.326+0000 7f85888a8640 1 -- 172.21.15.115:0/231228671 <== mon.1 v1:172.21.15.115:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1586965025 0 0) 0x7f857c001f80 con 0x7f8584142dd0 2024-03-25T18:26:42.327 INFO:teuthology.orchestra.run.smithi115.stderr:2024-03-25T18:26:42.326+0000 7f858bb0e640 10 monclient: get_monmap_and_config success 2024-03-25T18:26:42.327 INFO:teuthology.orchestra.run.smithi115.stderr:2024-03-25T18:26:42.326+0000 7f858bb0e640 4 set_mon_vals no callback set 2024-03-25T18:26:42.327 INFO:teuthology.orchestra.run.smithi115.stderr:2024-03-25T18:26:42.326+0000 7f85888a8640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:26:42.327 INFO:teuthology.orchestra.run.smithi115.stderr:2024-03-25T18:26:42.326+0000 7f85888a8640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-25T18:26:42.328 INFO:teuthology.orchestra.run.smithi115.stderr:2024-03-25T18:26:42.326+0000 7f85888a8640 10 monclient: dump: 2024-03-25T18:26:42.328 INFO:teuthology.orchestra.run.smithi115.stderr:epoch 1 2024-03-25T18:26:42.328 INFO:teuthology.orchestra.run.smithi115.stderr:fsid c8ea72cf-9e99-4383-9b52-6c23d365fdfc 2024-03-25T18:26:42.328 INFO:teuthology.orchestra.run.smithi115.stderr:last_changed 2024-03-25T18:25:50.510519+0000 2024-03-25T18:26:42.328 INFO:teuthology.orchestra.run.smithi115.stderr:created 2024-03-25T18:25:50.510519+0000 2024-03-25T18:26:42.328 INFO:teuthology.orchestra.run.smithi115.stderr:min_mon_release 19 (squid) 2024-03-25T18:26:42.328 INFO:teuthology.orchestra.run.smithi115.stderr:election_strategy: 3 2024-03-25T18:26:42.328 INFO:teuthology.orchestra.run.smithi115.stderr:0: v1:172.21.15.1:6789/0 mon.a 2024-03-25T18:26:42.328 INFO:teuthology.orchestra.run.smithi115.stderr:1: v1:172.21.15.115:6789/0 mon.b 2024-03-25T18:26:42.328 INFO:teuthology.orchestra.run.smithi115.stderr:2: v1:172.21.15.1:6790/0 mon.c 2024-03-25T18:26:42.328 INFO:teuthology.orchestra.run.smithi115.stderr: 2024-03-25T18:26:42.331 INFO:teuthology.orchestra.run.smithi115.stderr:2024-03-25T18:26:42.327+0000 7f858bb0e640 10 monclient: shutdown 2024-03-25T18:26:42.331 INFO:teuthology.orchestra.run.smithi115.stderr:2024-03-25T18:26:42.327+0000 7f858bb0e640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:26:42.331 INFO:teuthology.orchestra.run.smithi115.stderr:2024-03-25T18:26:42.327+0000 7f858bb0e640 1 -- 172.21.15.115:0/231228671 >> v1:172.21.15.115:6789/0 conn(0x7f8584142dd0 legacy=0x7f858414df50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:26:42.331 INFO:teuthology.orchestra.run.smithi115.stderr:2024-03-25T18:26:42.327+0000 7f858bb0e640 1 -- 172.21.15.115:0/231228671 shutdown_connections 2024-03-25T18:26:42.331 INFO:teuthology.orchestra.run.smithi115.stderr:2024-03-25T18:26:42.327+0000 7f858bb0e640 1 -- 172.21.15.115:0/231228671 >> 172.21.15.115:0/231228671 conn(0x7f8584066920 msgr2=0x7f8584066d30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:26:42.331 INFO:teuthology.orchestra.run.smithi115.stderr:2024-03-25T18:26:42.327+0000 7f858bb0e640 1 -- 172.21.15.115:0/231228671 shutdown_connections 2024-03-25T18:26:42.331 INFO:teuthology.orchestra.run.smithi115.stderr:2024-03-25T18:26:42.327+0000 7f858bb0e640 1 -- 172.21.15.115:0/231228671 wait complete. 2024-03-25T18:26:42.331 INFO:teuthology.orchestra.run.smithi115.stderr:2024-03-25T18:26:42.328+0000 7f858bb0e640 10 monclient: build_initial_monmap 2024-03-25T18:26:42.332 INFO:teuthology.orchestra.run.smithi115.stderr:2024-03-25T18:26:42.328+0000 7f858bb0e640 10 monclient: monmap: 2024-03-25T18:26:42.332 INFO:teuthology.orchestra.run.smithi115.stderr:epoch 0 2024-03-25T18:26:42.332 INFO:teuthology.orchestra.run.smithi115.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T18:26:42.332 INFO:teuthology.orchestra.run.smithi115.stderr:last_changed 0.000000 2024-03-25T18:26:42.332 INFO:teuthology.orchestra.run.smithi115.stderr:created 0.000000 2024-03-25T18:26:42.332 INFO:teuthology.orchestra.run.smithi115.stderr:min_mon_release 0 (unknown) 2024-03-25T18:26:42.332 INFO:teuthology.orchestra.run.smithi115.stderr:election_strategy: 1 2024-03-25T18:26:42.332 INFO:teuthology.orchestra.run.smithi115.stderr:0: v1:172.21.15.1:6789/0 mon.noname-a 2024-03-25T18:26:42.332 INFO:teuthology.orchestra.run.smithi115.stderr:1: v1:172.21.15.115:6789/0 mon.noname-b 2024-03-25T18:26:42.332 INFO:teuthology.orchestra.run.smithi115.stderr:2: v1:172.21.15.1:6790/0 mon.noname-c 2024-03-25T18:26:42.332 INFO:teuthology.orchestra.run.smithi115.stderr: 2024-03-25T18:26:42.332 INFO:teuthology.orchestra.run.smithi115.stderr:2024-03-25T18:26:42.328+0000 7f858bb0e640 1 librados: starting msgr at 2024-03-25T18:26:42.332 INFO:teuthology.orchestra.run.smithi115.stderr:2024-03-25T18:26:42.328+0000 7f858bb0e640 1 librados: starting objecter 2024-03-25T18:26:42.332 INFO:teuthology.orchestra.run.smithi115.stderr:2024-03-25T18:26:42.328+0000 7f858bb0e640 1 Processor -- start 2024-03-25T18:26:42.333 INFO:teuthology.orchestra.run.smithi115.stderr:2024-03-25T18:26:42.328+0000 7f858bb0e640 1 -- start start 2024-03-25T18:26:42.333 INFO:teuthology.orchestra.run.smithi115.stderr:2024-03-25T18:26:42.328+0000 7f858bb0e640 1 librados: setting wanted keys 2024-03-25T18:26:42.333 INFO:teuthology.orchestra.run.smithi115.stderr:2024-03-25T18:26:42.328+0000 7f858bb0e640 1 librados: calling monclient init 2024-03-25T18:26:42.333 INFO:teuthology.orchestra.run.smithi115.stderr:2024-03-25T18:26:42.328+0000 7f858bb0e640 10 monclient: init 2024-03-25T18:26:42.333 INFO:teuthology.orchestra.run.smithi115.stderr:2024-03-25T18:26:42.329+0000 7f858bb0e640 10 monclient: _reopen_session rank -1 2024-03-25T18:26:42.333 INFO:teuthology.orchestra.run.smithi115.stderr:2024-03-25T18:26:42.329+0000 7f858bb0e640 10 monclient: _add_conns ranks=[0,2,1] 2024-03-25T18:26:42.333 INFO:teuthology.orchestra.run.smithi115.stderr:2024-03-25T18:26:42.329+0000 7f858bb0e640 10 monclient(hunting): picked mon.noname-a con 0x7f85840b4b50 addr v1:172.21.15.1:6789/0 2024-03-25T18:26:42.333 INFO:teuthology.orchestra.run.smithi115.stderr:2024-03-25T18:26:42.329+0000 7f858bb0e640 10 monclient(hunting): picked mon.noname-c con 0x7f858413ff20 addr v1:172.21.15.1:6790/0 2024-03-25T18:26:42.333 INFO:teuthology.orchestra.run.smithi115.stderr:2024-03-25T18:26:42.329+0000 7f858bb0e640 10 monclient(hunting): picked mon.noname-b con 0x7f8584142dd0 addr v1:172.21.15.115:6789/0 2024-03-25T18:26:42.333 INFO:teuthology.orchestra.run.smithi115.stderr:2024-03-25T18:26:42.329+0000 7f858bb0e640 1 -- --> v1:172.21.15.1:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8584144640 con 0x7f85840b4b50 2024-03-25T18:26:42.333 INFO:teuthology.orchestra.run.smithi115.stderr:2024-03-25T18:26:42.329+0000 7f858bb0e640 1 -- --> v1:172.21.15.115:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8584140430 con 0x7f8584142dd0 2024-03-25T18:26:42.333 INFO:teuthology.orchestra.run.smithi115.stderr:2024-03-25T18:26:42.329+0000 7f858bb0e640 1 -- --> v1:172.21.15.1:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8584135330 con 0x7f858413ff20 2024-03-25T18:26:42.333 INFO:teuthology.orchestra.run.smithi115.stderr:2024-03-25T18:26:42.329+0000 7f858bb0e640 10 monclient(hunting): _renew_subs 2024-03-25T18:26:42.333 INFO:teuthology.orchestra.run.smithi115.stderr:2024-03-25T18:26:42.329+0000 7f858bb0e640 10 monclient(hunting): authenticate will time out at 7493.485466s 2024-03-25T18:26:42.333 INFO:teuthology.orchestra.run.smithi115.stderr:2024-03-25T18:26:42.330+0000 7f858a0ab640 1 --1- >> v1:172.21.15.115:6789/0 conn(0x7f8584142dd0 0x7f85841e2180 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.115:6789/0 says I am v1:172.21.15.115:44614/0 (socket says 172.21.15.115:44614) 2024-03-25T18:26:42.334 INFO:teuthology.orchestra.run.smithi115.stderr:2024-03-25T18:26:42.330+0000 7f858a0ab640 1 -- 172.21.15.115:0/1137708302 learned_addr learned my addr 172.21.15.115:0/1137708302 (peer_addr_for_me v1:172.21.15.115:0/0) 2024-03-25T18:26:42.334 INFO:teuthology.orchestra.run.smithi115.stderr:2024-03-25T18:26:42.330+0000 7f857a7fc640 10 client.?.objecter ms_handle_connect 0x7f8584142dd0 2024-03-25T18:26:42.334 INFO:teuthology.orchestra.run.smithi115.stderr:2024-03-25T18:26:42.330+0000 7f857a7fc640 10 client.?.objecter resend_mon_ops 2024-03-25T18:26:42.334 INFO:teuthology.orchestra.run.smithi115.stderr:2024-03-25T18:26:42.330+0000 7f857a7fc640 10 client.?.objecter ms_handle_connect 0x7f858413ff20 2024-03-25T18:26:42.334 INFO:teuthology.orchestra.run.smithi115.stderr:2024-03-25T18:26:42.330+0000 7f857a7fc640 10 client.?.objecter resend_mon_ops 2024-03-25T18:26:42.334 INFO:teuthology.orchestra.run.smithi115.stderr:2024-03-25T18:26:42.330+0000 7f857a7fc640 10 client.?.objecter ms_handle_connect 0x7f85840b4b50 2024-03-25T18:26:42.334 INFO:teuthology.orchestra.run.smithi115.stderr:2024-03-25T18:26:42.330+0000 7f857a7fc640 10 client.?.objecter resend_mon_ops 2024-03-25T18:26:42.335 INFO:teuthology.orchestra.run.smithi115.stderr:2024-03-25T18:26:42.332+0000 7f857a7fc640 1 -- 172.21.15.115:0/1137708302 <== mon.1 v1:172.21.15.115:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2101577581 0 0) 0x7f8584140430 con 0x7f8584142dd0 2024-03-25T18:26:42.335 INFO:teuthology.orchestra.run.smithi115.stderr:2024-03-25T18:26:42.332+0000 7f857a7fc640 10 monclient(hunting): _negotiate 2024-03-25T18:26:42.335 INFO:teuthology.orchestra.run.smithi115.stderr:2024-03-25T18:26:42.332+0000 7f857a7fc640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:26:42.335 INFO:teuthology.orchestra.run.smithi115.stderr:2024-03-25T18:26:42.332+0000 7f857a7fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:26:42.335 INFO:teuthology.orchestra.run.smithi115.stderr:2024-03-25T18:26:42.332+0000 7f857a7fc640 10 monclient(hunting): my global_id is 4129 2024-03-25T18:26:42.335 INFO:teuthology.orchestra.run.smithi115.stderr:2024-03-25T18:26:42.332+0000 7f857a7fc640 1 -- 172.21.15.115:0/1137708302 --> v1:172.21.15.115:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f8568004eb0 con 0x7f8584142dd0 2024-03-25T18:26:42.335 INFO:teuthology.orchestra.run.smithi115.stderr:2024-03-25T18:26:42.332+0000 7f857a7fc640 1 -- 172.21.15.115:0/1137708302 <== mon.2 v1:172.21.15.1:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1439178721 0 0) 0x7f8584135330 con 0x7f858413ff20 2024-03-25T18:26:42.335 INFO:teuthology.orchestra.run.smithi115.stderr:2024-03-25T18:26:42.332+0000 7f857a7fc640 10 monclient(hunting): _negotiate 2024-03-25T18:26:42.336 INFO:teuthology.orchestra.run.smithi115.stderr:2024-03-25T18:26:42.332+0000 7f857a7fc640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:26:42.336 INFO:teuthology.orchestra.run.smithi115.stderr:2024-03-25T18:26:42.332+0000 7f857a7fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:26:42.336 INFO:teuthology.orchestra.run.smithi115.stderr:2024-03-25T18:26:42.332+0000 7f857a7fc640 10 monclient(hunting): my global_id is 4133 2024-03-25T18:26:42.336 INFO:teuthology.orchestra.run.smithi115.stderr:2024-03-25T18:26:42.332+0000 7f857a7fc640 1 -- 172.21.15.115:0/1137708302 --> v1:172.21.15.1:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f8584140430 con 0x7f858413ff20 2024-03-25T18:26:42.336 INFO:teuthology.orchestra.run.smithi115.stderr:2024-03-25T18:26:42.333+0000 7f857a7fc640 1 -- 172.21.15.115:0/1137708302 <== mon.0 v1:172.21.15.1:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 632670607 0 0) 0x7f8584144640 con 0x7f85840b4b50 2024-03-25T18:26:42.336 INFO:teuthology.orchestra.run.smithi115.stderr:2024-03-25T18:26:42.333+0000 7f857a7fc640 10 monclient(hunting): _negotiate 2024-03-25T18:26:42.336 INFO:teuthology.orchestra.run.smithi115.stderr:2024-03-25T18:26:42.333+0000 7f857a7fc640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:26:42.336 INFO:teuthology.orchestra.run.smithi115.stderr:2024-03-25T18:26:42.333+0000 7f857a7fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:26:42.336 INFO:teuthology.orchestra.run.smithi115.stderr:2024-03-25T18:26:42.333+0000 7f857a7fc640 10 monclient(hunting): my global_id is 4134 2024-03-25T18:26:42.336 INFO:teuthology.orchestra.run.smithi115.stderr:2024-03-25T18:26:42.333+0000 7f857a7fc640 1 -- 172.21.15.115:0/1137708302 --> v1:172.21.15.1:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f8584135330 con 0x7f85840b4b50 2024-03-25T18:26:42.337 INFO:teuthology.orchestra.run.smithi115.stderr:2024-03-25T18:26:42.336+0000 7f857a7fc640 1 -- 172.21.15.115:0/1137708302 <== mon.1 v1:172.21.15.115:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1896839895 0 0) 0x7f8568004eb0 con 0x7f8584142dd0 2024-03-25T18:26:42.337 INFO:teuthology.orchestra.run.smithi115.stderr:2024-03-25T18:26:42.336+0000 7f857a7fc640 1 -- 172.21.15.115:0/1137708302 --> v1:172.21.15.115:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f8584144640 con 0x7f8584142dd0 2024-03-25T18:26:42.337 INFO:teuthology.orchestra.run.smithi115.stderr:2024-03-25T18:26:42.336+0000 7f857a7fc640 1 -- 172.21.15.115:0/1137708302 <== mon.1 v1:172.21.15.115:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1586965025 0 0) 0x7f857c001f00 con 0x7f8584142dd0 2024-03-25T18:26:42.338 INFO:teuthology.orchestra.run.smithi115.stderr:2024-03-25T18:26:42.336+0000 7f857a7fc640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:26:42.338 INFO:teuthology.orchestra.run.smithi115.stderr:2024-03-25T18:26:42.336+0000 7f857a7fc640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-03-25T18:26:42.338 INFO:teuthology.orchestra.run.smithi115.stderr:2024-03-25T18:26:42.336+0000 7f857a7fc640 10 monclient(hunting): dump: 2024-03-25T18:26:42.338 INFO:teuthology.orchestra.run.smithi115.stderr:epoch 1 2024-03-25T18:26:42.338 INFO:teuthology.orchestra.run.smithi115.stderr:fsid c8ea72cf-9e99-4383-9b52-6c23d365fdfc 2024-03-25T18:26:42.338 INFO:teuthology.orchestra.run.smithi115.stderr:last_changed 2024-03-25T18:25:50.510519+0000 2024-03-25T18:26:42.338 INFO:teuthology.orchestra.run.smithi115.stderr:created 2024-03-25T18:25:50.510519+0000 2024-03-25T18:26:42.338 INFO:teuthology.orchestra.run.smithi115.stderr:min_mon_release 19 (squid) 2024-03-25T18:26:42.338 INFO:teuthology.orchestra.run.smithi115.stderr:election_strategy: 3 2024-03-25T18:26:42.338 INFO:teuthology.orchestra.run.smithi115.stderr:0: v1:172.21.15.1:6789/0 mon.a 2024-03-25T18:26:42.338 INFO:teuthology.orchestra.run.smithi115.stderr:1: v1:172.21.15.115:6789/0 mon.b 2024-03-25T18:26:42.338 INFO:teuthology.orchestra.run.smithi115.stderr:2: v1:172.21.15.1:6790/0 mon.c 2024-03-25T18:26:42.338 INFO:teuthology.orchestra.run.smithi115.stderr: 2024-03-25T18:26:42.338 INFO:teuthology.orchestra.run.smithi115.stderr:2024-03-25T18:26:42.336+0000 7f857a7fc640 10 monclient(hunting): _finish_auth 0 2024-03-25T18:26:42.340 INFO:teuthology.orchestra.run.smithi115.stderr:2024-03-25T18:26:42.339+0000 7f857a7fc640 1 -- 172.21.15.115:0/1137708302 <== mon.2 v1:172.21.15.1:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 130365730 0 0) 0x7f8584140430 con 0x7f858413ff20 2024-03-25T18:26:42.340 INFO:teuthology.orchestra.run.smithi115.stderr:2024-03-25T18:26:42.339+0000 7f857a7fc640 1 -- 172.21.15.115:0/1137708302 --> v1:172.21.15.1:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f8568004eb0 con 0x7f858413ff20 2024-03-25T18:26:42.340 INFO:teuthology.orchestra.run.smithi115.stderr:2024-03-25T18:26:42.339+0000 7f857a7fc640 1 -- 172.21.15.115:0/1137708302 <== mon.2 v1:172.21.15.1:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1586965025 0 0) 0x7f8580001870 con 0x7f858413ff20 2024-03-25T18:26:42.340 INFO:teuthology.orchestra.run.smithi115.stderr:2024-03-25T18:26:42.339+0000 7f857a7fc640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:26:42.340 INFO:teuthology.orchestra.run.smithi115.stderr:2024-03-25T18:26:42.339+0000 7f857a7fc640 10 monclient(hunting): got monmap 1 from mon.c (according to old e1) 2024-03-25T18:26:42.340 INFO:teuthology.orchestra.run.smithi115.stderr:2024-03-25T18:26:42.339+0000 7f857a7fc640 10 monclient(hunting): dump: 2024-03-25T18:26:42.340 INFO:teuthology.orchestra.run.smithi115.stderr:epoch 1 2024-03-25T18:26:42.340 INFO:teuthology.orchestra.run.smithi115.stderr:fsid c8ea72cf-9e99-4383-9b52-6c23d365fdfc 2024-03-25T18:26:42.340 INFO:teuthology.orchestra.run.smithi115.stderr:last_changed 2024-03-25T18:25:50.510519+0000 2024-03-25T18:26:42.340 INFO:teuthology.orchestra.run.smithi115.stderr:created 2024-03-25T18:25:50.510519+0000 2024-03-25T18:26:42.340 INFO:teuthology.orchestra.run.smithi115.stderr:min_mon_release 19 (squid) 2024-03-25T18:26:42.340 INFO:teuthology.orchestra.run.smithi115.stderr:election_strategy: 3 2024-03-25T18:26:42.341 INFO:teuthology.orchestra.run.smithi115.stderr:0: v1:172.21.15.1:6789/0 mon.a 2024-03-25T18:26:42.341 INFO:teuthology.orchestra.run.smithi115.stderr:1: v1:172.21.15.115:6789/0 mon.b 2024-03-25T18:26:42.341 INFO:teuthology.orchestra.run.smithi115.stderr:2: v1:172.21.15.1:6790/0 mon.c 2024-03-25T18:26:42.341 INFO:teuthology.orchestra.run.smithi115.stderr: 2024-03-25T18:26:42.342 INFO:teuthology.orchestra.run.smithi115.stderr:2024-03-25T18:26:42.340+0000 7f857a7fc640 1 -- 172.21.15.115:0/1137708302 <== mon.1 v1:172.21.15.115:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 69669438 0 0) 0x7f8584144640 con 0x7f8584142dd0 2024-03-25T18:26:42.342 INFO:teuthology.orchestra.run.smithi115.stderr:2024-03-25T18:26:42.340+0000 7f857a7fc640 1 -- 172.21.15.115:0/1137708302 >> v1:172.21.15.1:6790/0 conn(0x7f858413ff20 legacy=0x7f85841db130 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:26:42.342 INFO:teuthology.orchestra.run.smithi115.stderr:2024-03-25T18:26:42.340+0000 7f857a7fc640 1 -- 172.21.15.115:0/1137708302 >> v1:172.21.15.1:6789/0 conn(0x7f85840b4b50 legacy=0x7f8584141c80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:26:42.342 INFO:teuthology.orchestra.run.smithi115.stderr:2024-03-25T18:26:42.340+0000 7f857a7fc640 10 monclient: _finish_hunting 0 2024-03-25T18:26:42.342 INFO:teuthology.orchestra.run.smithi115.stderr:2024-03-25T18:26:42.340+0000 7f857a7fc640 1 monclient: found mon.b 2024-03-25T18:26:42.342 INFO:teuthology.orchestra.run.smithi115.stderr:2024-03-25T18:26:42.340+0000 7f857a7fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:26:42.343 INFO:teuthology.orchestra.run.smithi115.stderr:2024-03-25T18:26:42.340+0000 7f857a7fc640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.115:6789/0 2024-03-25T18:26:42.343 INFO:teuthology.orchestra.run.smithi115.stderr:2024-03-25T18:26:42.340+0000 7f857a7fc640 1 -- 172.21.15.115:0/1137708302 --> v1:172.21.15.115:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f85841db840 con 0x7f8584142dd0 2024-03-25T18:26:42.343 INFO:teuthology.orchestra.run.smithi115.stderr:2024-03-25T18:26:42.340+0000 7f857a7fc640 10 monclient: _finish_auth 0 2024-03-25T18:26:42.343 INFO:teuthology.orchestra.run.smithi115.stderr:2024-03-25T18:26:42.340+0000 7f857a7fc640 10 monclient: _check_auth_tickets 2024-03-25T18:26:42.343 INFO:teuthology.orchestra.run.smithi115.stderr:2024-03-25T18:26:42.340+0000 7f857a7fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:26:42.343 INFO:teuthology.orchestra.run.smithi115.stderr:2024-03-25T18:26:42.340+0000 7f858bb0e640 5 monclient: authenticate success, global_id 4129 2024-03-25T18:26:42.343 INFO:teuthology.orchestra.run.smithi115.stderr:2024-03-25T18:26:42.340+0000 7f858bb0e640 10 monclient: _renew_subs 2024-03-25T18:26:42.343 INFO:teuthology.orchestra.run.smithi115.stderr:2024-03-25T18:26:42.340+0000 7f858bb0e640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.115:6789/0 2024-03-25T18:26:42.343 INFO:teuthology.orchestra.run.smithi115.stderr:2024-03-25T18:26:42.340+0000 7f858bb0e640 1 -- 172.21.15.115:0/1137708302 --> v1:172.21.15.115:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8584141060 con 0x7f8584142dd0 2024-03-25T18:26:42.343 INFO:teuthology.orchestra.run.smithi115.stderr:2024-03-25T18:26:42.340+0000 7f858bb0e640 10 client.4129.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T18:26:42.343 INFO:teuthology.orchestra.run.smithi115.stderr:2024-03-25T18:26:42.340+0000 7f858bb0e640 10 monclient: _renew_subs 2024-03-25T18:26:42.343 INFO:teuthology.orchestra.run.smithi115.stderr:2024-03-25T18:26:42.340+0000 7f858bb0e640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.115:6789/0 2024-03-25T18:26:42.343 INFO:teuthology.orchestra.run.smithi115.stderr:2024-03-25T18:26:42.340+0000 7f858bb0e640 1 -- 172.21.15.115:0/1137708302 --> v1:172.21.15.115:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f8584141400 con 0x7f8584142dd0 2024-03-25T18:26:42.343 INFO:teuthology.orchestra.run.smithi115.stderr:2024-03-25T18:26:42.340+0000 7f858bb0e640 1 librados: init done 2024-03-25T18:26:42.344 INFO:teuthology.orchestra.run.smithi115.stderr:2024-03-25T18:26:42.341+0000 7f858bb0e640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T18:26:42.344 INFO:teuthology.orchestra.run.smithi115.stderr:2024-03-25T18:26:42.341+0000 7f858bb0e640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:26:42.344 INFO:teuthology.orchestra.run.smithi115.stderr:2024-03-25T18:26:42.341+0000 7f858bb0e640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.115:6789/0 2024-03-25T18:26:42.344 INFO:teuthology.orchestra.run.smithi115.stderr:2024-03-25T18:26:42.341+0000 7f858bb0e640 1 -- 172.21.15.115:0/1137708302 --> v1:172.21.15.115:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f85841daf80 con 0x7f8584142dd0 2024-03-25T18:26:42.364 INFO:teuthology.orchestra.run.smithi115.stderr:2024-03-25T18:26:42.363+0000 7f857a7fc640 1 -- 172.21.15.115:0/1137708302 <== mon.1 v1:172.21.15.115:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f857c001570 con 0x7f8584142dd0 2024-03-25T18:26:42.364 INFO:teuthology.orchestra.run.smithi115.stderr:2024-03-25T18:26:42.363+0000 7f857a7fc640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:26:42.364 INFO:teuthology.orchestra.run.smithi115.stderr:2024-03-25T18:26:42.363+0000 7f857a7fc640 1 -- 172.21.15.115:0/1137708302 <== mon.1 v1:172.21.15.115:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1586965025 0 0) 0x7f857c004fb0 con 0x7f8584142dd0 2024-03-25T18:26:42.364 INFO:teuthology.orchestra.run.smithi115.stderr:2024-03-25T18:26:42.363+0000 7f857a7fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:26:42.364 INFO:teuthology.orchestra.run.smithi115.stderr:2024-03-25T18:26:42.363+0000 7f857a7fc640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-25T18:26:42.364 INFO:teuthology.orchestra.run.smithi115.stderr:2024-03-25T18:26:42.363+0000 7f857a7fc640 10 monclient: dump: 2024-03-25T18:26:42.364 INFO:teuthology.orchestra.run.smithi115.stderr:epoch 1 2024-03-25T18:26:42.364 INFO:teuthology.orchestra.run.smithi115.stderr:fsid c8ea72cf-9e99-4383-9b52-6c23d365fdfc 2024-03-25T18:26:42.365 INFO:teuthology.orchestra.run.smithi115.stderr:last_changed 2024-03-25T18:25:50.510519+0000 2024-03-25T18:26:42.365 INFO:teuthology.orchestra.run.smithi115.stderr:created 2024-03-25T18:25:50.510519+0000 2024-03-25T18:26:42.365 INFO:teuthology.orchestra.run.smithi115.stderr:min_mon_release 19 (squid) 2024-03-25T18:26:42.365 INFO:teuthology.orchestra.run.smithi115.stderr:election_strategy: 3 2024-03-25T18:26:42.365 INFO:teuthology.orchestra.run.smithi115.stderr:0: v1:172.21.15.1:6789/0 mon.a 2024-03-25T18:26:42.365 INFO:teuthology.orchestra.run.smithi115.stderr:1: v1:172.21.15.115:6789/0 mon.b 2024-03-25T18:26:42.365 INFO:teuthology.orchestra.run.smithi115.stderr:2: v1:172.21.15.1:6790/0 mon.c 2024-03-25T18:26:42.365 INFO:teuthology.orchestra.run.smithi115.stderr: 2024-03-25T18:26:42.365 INFO:teuthology.orchestra.run.smithi115.stderr:2024-03-25T18:26:42.363+0000 7f857bfff640 4 set_mon_vals no callback set 2024-03-25T18:26:42.365 INFO:teuthology.orchestra.run.smithi115.stderr:2024-03-25T18:26:42.363+0000 7f857a7fc640 1 -- 172.21.15.115:0/1137708302 <== mon.1 v1:172.21.15.115:6789/0 7 ==== mgrmap(e 1) v1 ==== 807+0+0 (unknown 3189685207 0 0) 0x7f857c0054a0 con 0x7f8584142dd0 2024-03-25T18:26:42.365 INFO:teuthology.orchestra.run.smithi115.stderr:2024-03-25T18:26:42.363+0000 7f857a7fc640 1 -- 172.21.15.115:0/1137708302 <== mon.1 v1:172.21.15.115:6789/0 8 ==== osd_map(5..5 src has 1..5) v4 ==== 1070+0+0 (unknown 1939002360 0 0) 0x7f857c006a80 con 0x7f8584142dd0 2024-03-25T18:26:42.365 INFO:teuthology.orchestra.run.smithi115.stderr:2024-03-25T18:26:42.363+0000 7f857a7fc640 10 client.4129.objecter ms_dispatch 0x7f8584000b90 osd_map(5..5 src has 1..5) v4 2024-03-25T18:26:42.365 INFO:teuthology.orchestra.run.smithi115.stderr:2024-03-25T18:26:42.363+0000 7f857a7fc640 3 client.4129.objecter handle_osd_map got epochs [5,5] > 0 2024-03-25T18:26:42.365 INFO:teuthology.orchestra.run.smithi115.stderr:2024-03-25T18:26:42.363+0000 7f857a7fc640 3 client.4129.objecter handle_osd_map decoding full epoch 5 2024-03-25T18:26:42.365 INFO:teuthology.orchestra.run.smithi115.stderr:2024-03-25T18:26:42.363+0000 7f857a7fc640 20 client.4129.objecter dump_active .. 0 homeless 2024-03-25T18:26:42.379 INFO:tasks.ceph.mgr.x.smithi115.stderr:2024-03-25T18:26:42.378+0000 7f99dd247040 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2024-03-25T18:26:42.442 INFO:tasks.ceph.mgr.y.smithi001.stderr:2024-03-25T18:26:42.440+0000 7faca7441040 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2024-03-25T18:26:42.533 INFO:teuthology.orchestra.run.smithi115.stderr:2024-03-25T18:26:42.532+0000 7f857a7fc640 1 -- 172.21.15.115:0/1137708302 <== mon.1 v1:172.21.15.115:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+73141 (unknown 1092875540 0 2031569661) 0x7f857c006700 con 0x7f8584142dd0 2024-03-25T18:26:42.533 INFO:teuthology.orchestra.run.smithi115.stderr:2024-03-25T18:26:42.532+0000 7f857a7fc640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:26:42.534 INFO:teuthology.orchestra.run.smithi115.stderr:2024-03-25T18:26:42.532+0000 7f857a7fc640 10 monclient: _finish_command 1 = system:0 2024-03-25T18:26:42.534 INFO:tasks.ceph.mgr.x.smithi115.stderr:2024-03-25T18:26:42.534+0000 7f99dd247040 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2024-03-25T18:26:42.553 INFO:tasks.ceph.mgr.y.smithi001.stderr:2024-03-25T18:26:42.552+0000 7faca7441040 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2024-03-25T18:26:42.578 INFO:tasks.ceph.mgr.x.smithi115.stderr:2024-03-25T18:26:42.578+0000 7f99dd247040 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2024-03-25T18:26:42.601 INFO:tasks.ceph.mgr.y.smithi001.stderr:2024-03-25T18:26:42.599+0000 7faca7441040 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2024-03-25T18:26:42.607 INFO:teuthology.orchestra.run.smithi115.stderr:2024-03-25T18:26:42.607+0000 7f858bb0e640 10 start_mon_command cmd=[{"prefix": "osd new", "uuid": "45e0e9ab-898d-4673-9590-b4117bc5e84a", "id": 3}] 2024-03-25T18:26:42.608 INFO:teuthology.orchestra.run.smithi115.stderr:2024-03-25T18:26:42.607+0000 7f858bb0e640 10 monclient: _send_command 2 [{"prefix": "osd new", "uuid": "45e0e9ab-898d-4673-9590-b4117bc5e84a", "id": 3}] 2024-03-25T18:26:42.608 INFO:teuthology.orchestra.run.smithi115.stderr:2024-03-25T18:26:42.607+0000 7f858bb0e640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.115:6789/0 2024-03-25T18:26:42.608 INFO:teuthology.orchestra.run.smithi115.stderr:2024-03-25T18:26:42.607+0000 7f858bb0e640 1 -- 172.21.15.115:0/1137708302 --> v1:172.21.15.115:6789/0 -- mon_command({"prefix": "osd new", "uuid": "45e0e9ab-898d-4673-9590-b4117bc5e84a", "id": 3} v 0) v1 -- 0x7f858404f730 con 0x7f8584142dd0 2024-03-25T18:26:42.722 INFO:tasks.ceph.mgr.y.smithi001.stderr:2024-03-25T18:26:42.721+0000 7faca7441040 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2024-03-25T18:26:42.736 INFO:tasks.ceph.mgr.x.smithi115.stderr:2024-03-25T18:26:42.735+0000 7f99dd247040 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2024-03-25T18:26:42.741 INFO:teuthology.orchestra.run.smithi115.stderr:2024-03-25T18:26:42.740+0000 7f857a7fc640 1 -- 172.21.15.115:0/1137708302 <== mon.1 v1:172.21.15.115:6789/0 10 ==== mon_command_ack([{"prefix": "osd new", "uuid": "45e0e9ab-898d-4673-9590-b4117bc5e84a", "id": 3}]=0 v6) v1 ==== 112+0+2 (unknown 2252032917 0 525046306) 0x7f857c005750 con 0x7f8584142dd0 2024-03-25T18:26:42.741 INFO:teuthology.orchestra.run.smithi115.stderr:2024-03-25T18:26:42.740+0000 7f857a7fc640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd new", "uuid": "45e0e9ab-898d-4673-9590-b4117bc5e84a", "id": 3}] 2024-03-25T18:26:42.741 INFO:teuthology.orchestra.run.smithi115.stderr:2024-03-25T18:26:42.740+0000 7f857a7fc640 10 monclient: _finish_command 2 = system:0 2024-03-25T18:26:42.741 INFO:teuthology.orchestra.run.smithi115.stdout:3 2024-03-25T18:26:42.743 INFO:teuthology.orchestra.run.smithi115.stderr:2024-03-25T18:26:42.741+0000 7f858bb0e640 10 librados: watch_flush enter 2024-03-25T18:26:42.743 INFO:teuthology.orchestra.run.smithi115.stderr:2024-03-25T18:26:42.741+0000 7f858bb0e640 10 librados: watch_flush exit 2024-03-25T18:26:42.743 INFO:teuthology.orchestra.run.smithi115.stderr:2024-03-25T18:26:42.741+0000 7f858bb0e640 20 client.4129.objecter shutdown clearing up homeless session... 2024-03-25T18:26:42.743 INFO:teuthology.orchestra.run.smithi115.stderr:2024-03-25T18:26:42.741+0000 7f858bb0e640 10 client.4129.objecter successfully canceled tick 2024-03-25T18:26:42.743 INFO:teuthology.orchestra.run.smithi115.stderr:2024-03-25T18:26:42.741+0000 7f858bb0e640 10 monclient: shutdown 2024-03-25T18:26:42.743 INFO:teuthology.orchestra.run.smithi115.stderr:2024-03-25T18:26:42.741+0000 7f858bb0e640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:26:42.743 INFO:teuthology.orchestra.run.smithi115.stderr:2024-03-25T18:26:42.741+0000 7f858bb0e640 1 -- 172.21.15.115:0/1137708302 >> v1:172.21.15.115:6789/0 conn(0x7f8584142dd0 legacy=0x7f85841e2180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:26:42.743 INFO:teuthology.orchestra.run.smithi115.stderr:2024-03-25T18:26:42.741+0000 7f858bb0e640 1 -- 172.21.15.115:0/1137708302 shutdown_connections 2024-03-25T18:26:42.743 INFO:teuthology.orchestra.run.smithi115.stderr:2024-03-25T18:26:42.741+0000 7f858bb0e640 1 -- 172.21.15.115:0/1137708302 >> 172.21.15.115:0/1137708302 conn(0x7f8584066920 msgr2=0x7f85840a9660 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:26:42.743 INFO:teuthology.orchestra.run.smithi115.stderr:2024-03-25T18:26:42.741+0000 7f858bb0e640 1 -- 172.21.15.115:0/1137708302 shutdown_connections 2024-03-25T18:26:42.743 INFO:teuthology.orchestra.run.smithi115.stderr:2024-03-25T18:26:42.741+0000 7f858bb0e640 1 -- 172.21.15.115:0/1137708302 wait complete. 2024-03-25T18:26:42.743 INFO:teuthology.orchestra.run.smithi115.stderr:2024-03-25T18:26:42.741+0000 7f858bb0e640 1 librados: shutdown 2024-03-25T18:26:42.752 DEBUG:teuthology.orchestra.run.smithi115:> sudo ceph --cluster ceph osd new 831a5a09-73ab-4a61-9872-088e76bdbc66 4 2024-03-25T18:26:42.825 INFO:tasks.ceph.mgr.y.smithi001.stderr:2024-03-25T18:26:42.824+0000 7faca7441040 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2024-03-25T18:26:42.853 INFO:teuthology.orchestra.run.smithi115.stderr:2024-03-25T18:26:42.851+0000 7f05d2cc8640 10 monclient: get_monmap_and_config 2024-03-25T18:26:42.853 INFO:teuthology.orchestra.run.smithi115.stderr:2024-03-25T18:26:42.851+0000 7f05d2cc8640 10 monclient: build_initial_monmap 2024-03-25T18:26:42.853 INFO:teuthology.orchestra.run.smithi115.stderr:2024-03-25T18:26:42.852+0000 7f05d2cc8640 10 monclient: monmap: 2024-03-25T18:26:42.853 INFO:teuthology.orchestra.run.smithi115.stderr:epoch 0 2024-03-25T18:26:42.853 INFO:teuthology.orchestra.run.smithi115.stderr:fsid c8ea72cf-9e99-4383-9b52-6c23d365fdfc 2024-03-25T18:26:42.853 INFO:teuthology.orchestra.run.smithi115.stderr:last_changed 2024-03-25T18:26:42.852930+0000 2024-03-25T18:26:42.853 INFO:teuthology.orchestra.run.smithi115.stderr:created 2024-03-25T18:26:42.852930+0000 2024-03-25T18:26:42.853 INFO:teuthology.orchestra.run.smithi115.stderr:min_mon_release 0 (unknown) 2024-03-25T18:26:42.853 INFO:teuthology.orchestra.run.smithi115.stderr:election_strategy: 3 2024-03-25T18:26:42.853 INFO:teuthology.orchestra.run.smithi115.stderr:0: v1:172.21.15.1:6789/0 mon.noname-a 2024-03-25T18:26:42.853 INFO:teuthology.orchestra.run.smithi115.stderr:1: v1:172.21.15.115:6789/0 mon.noname-c 2024-03-25T18:26:42.854 INFO:teuthology.orchestra.run.smithi115.stderr:2: v1:172.21.15.1:6790/0 mon.noname-b 2024-03-25T18:26:42.854 INFO:teuthology.orchestra.run.smithi115.stderr: 2024-03-25T18:26:42.855 INFO:teuthology.orchestra.run.smithi115.stderr:2024-03-25T18:26:42.852+0000 7f05d2cc8640 1 Processor -- start 2024-03-25T18:26:42.855 INFO:teuthology.orchestra.run.smithi115.stderr:2024-03-25T18:26:42.852+0000 7f05d2cc8640 1 -- start start 2024-03-25T18:26:42.855 INFO:teuthology.orchestra.run.smithi115.stderr:2024-03-25T18:26:42.852+0000 7f05d2cc8640 10 monclient: init 2024-03-25T18:26:42.855 INFO:teuthology.orchestra.run.smithi115.stderr:2024-03-25T18:26:42.853+0000 7f05d2cc8640 10 monclient: _reopen_session rank -1 2024-03-25T18:26:42.855 INFO:teuthology.orchestra.run.smithi115.stderr:2024-03-25T18:26:42.853+0000 7f05d2cc8640 10 monclient: _add_conns ranks=[2,0,1] 2024-03-25T18:26:42.855 INFO:teuthology.orchestra.run.smithi115.stderr:2024-03-25T18:26:42.853+0000 7f05d2cc8640 10 monclient(hunting): picked mon.noname-b con 0x7f05cc0b4b50 addr v1:172.21.15.1:6790/0 2024-03-25T18:26:42.855 INFO:teuthology.orchestra.run.smithi115.stderr:2024-03-25T18:26:42.853+0000 7f05d2cc8640 10 monclient(hunting): picked mon.noname-a con 0x7f05cc13ff20 addr v1:172.21.15.1:6789/0 2024-03-25T18:26:42.855 INFO:teuthology.orchestra.run.smithi115.stderr:2024-03-25T18:26:42.853+0000 7f05d2cc8640 10 monclient(hunting): picked mon.noname-c con 0x7f05cc142dd0 addr v1:172.21.15.115:6789/0 2024-03-25T18:26:42.856 INFO:teuthology.orchestra.run.smithi115.stderr:2024-03-25T18:26:42.853+0000 7f05d2cc8640 1 -- --> v1:172.21.15.1:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f05cc0ab440 con 0x7f05cc13ff20 2024-03-25T18:26:42.856 INFO:teuthology.orchestra.run.smithi115.stderr:2024-03-25T18:26:42.853+0000 7f05d2cc8640 1 -- --> v1:172.21.15.115:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f05cc0ab6b0 con 0x7f05cc142dd0 2024-03-25T18:26:42.856 INFO:teuthology.orchestra.run.smithi115.stderr:2024-03-25T18:26:42.853+0000 7f05d2cc8640 1 -- --> v1:172.21.15.1:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f05cc0a9f10 con 0x7f05cc0b4b50 2024-03-25T18:26:42.856 INFO:teuthology.orchestra.run.smithi115.stderr:2024-03-25T18:26:42.853+0000 7f05d2cc8640 10 monclient(hunting): _renew_subs 2024-03-25T18:26:42.856 INFO:teuthology.orchestra.run.smithi115.stderr:2024-03-25T18:26:42.853+0000 7f05d2cc8640 10 monclient(hunting): authenticate will time out at 7494.009327s 2024-03-25T18:26:42.856 INFO:teuthology.orchestra.run.smithi115.stderr:2024-03-25T18:26:42.853+0000 7f05cbfff640 1 --1- >> v1:172.21.15.1:6789/0 conn(0x7f05cc13ff20 0x7f05cc1426c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.1:6789/0 says I am v1:172.21.15.115:55614/0 (socket says 172.21.15.115:55614) 2024-03-25T18:26:42.856 INFO:teuthology.orchestra.run.smithi115.stderr:2024-03-25T18:26:42.853+0000 7f05cbfff640 1 -- 172.21.15.115:0/3080203549 learned_addr learned my addr 172.21.15.115:0/3080203549 (peer_addr_for_me v1:172.21.15.115:0/0) 2024-03-25T18:26:42.857 INFO:teuthology.orchestra.run.smithi115.stderr:2024-03-25T18:26:42.856+0000 7f05cb7fe640 1 -- 172.21.15.115:0/3080203549 <== mon.2 v1:172.21.15.1:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1858586879 0 0) 0x7f05cc0a9f10 con 0x7f05cc0b4b50 2024-03-25T18:26:42.857 INFO:teuthology.orchestra.run.smithi115.stderr:2024-03-25T18:26:42.856+0000 7f05cb7fe640 10 monclient(hunting): _negotiate 2024-03-25T18:26:42.857 INFO:teuthology.orchestra.run.smithi115.stderr:2024-03-25T18:26:42.856+0000 7f05cb7fe640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:26:42.857 INFO:teuthology.orchestra.run.smithi115.stderr:2024-03-25T18:26:42.856+0000 7f05cb7fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:26:42.857 INFO:teuthology.orchestra.run.smithi115.stderr:2024-03-25T18:26:42.856+0000 7f05cb7fe640 10 monclient(hunting): my global_id is 4136 2024-03-25T18:26:42.857 INFO:teuthology.orchestra.run.smithi115.stderr:2024-03-25T18:26:42.856+0000 7f05cb7fe640 1 -- 172.21.15.115:0/3080203549 --> v1:172.21.15.1:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f05bc003520 con 0x7f05cc0b4b50 2024-03-25T18:26:42.858 INFO:teuthology.orchestra.run.smithi115.stderr:2024-03-25T18:26:42.856+0000 7f05cb7fe640 1 -- 172.21.15.115:0/3080203549 <== mon.1 v1:172.21.15.115:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1241772576 0 0) 0x7f05cc0ab6b0 con 0x7f05cc142dd0 2024-03-25T18:26:42.858 INFO:teuthology.orchestra.run.smithi115.stderr:2024-03-25T18:26:42.856+0000 7f05cb7fe640 10 monclient(hunting): _negotiate 2024-03-25T18:26:42.858 INFO:teuthology.orchestra.run.smithi115.stderr:2024-03-25T18:26:42.856+0000 7f05cb7fe640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:26:42.858 INFO:teuthology.orchestra.run.smithi115.stderr:2024-03-25T18:26:42.856+0000 7f05cb7fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:26:42.858 INFO:teuthology.orchestra.run.smithi115.stderr:2024-03-25T18:26:42.856+0000 7f05cb7fe640 10 monclient(hunting): my global_id is 4132 2024-03-25T18:26:42.858 INFO:teuthology.orchestra.run.smithi115.stderr:2024-03-25T18:26:42.856+0000 7f05cb7fe640 1 -- 172.21.15.115:0/3080203549 --> v1:172.21.15.115:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f05cc0a9f10 con 0x7f05cc142dd0 2024-03-25T18:26:42.859 INFO:teuthology.orchestra.run.smithi115.stderr:2024-03-25T18:26:42.857+0000 7f05cb7fe640 1 -- 172.21.15.115:0/3080203549 <== mon.0 v1:172.21.15.1:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3154515345 0 0) 0x7f05cc0ab440 con 0x7f05cc13ff20 2024-03-25T18:26:42.859 INFO:teuthology.orchestra.run.smithi115.stderr:2024-03-25T18:26:42.857+0000 7f05cb7fe640 10 monclient(hunting): _negotiate 2024-03-25T18:26:42.859 INFO:teuthology.orchestra.run.smithi115.stderr:2024-03-25T18:26:42.857+0000 7f05cb7fe640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:26:42.859 INFO:teuthology.orchestra.run.smithi115.stderr:2024-03-25T18:26:42.857+0000 7f05cb7fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:26:42.859 INFO:teuthology.orchestra.run.smithi115.stderr:2024-03-25T18:26:42.857+0000 7f05cb7fe640 10 monclient(hunting): my global_id is 4137 2024-03-25T18:26:42.859 INFO:teuthology.orchestra.run.smithi115.stderr:2024-03-25T18:26:42.857+0000 7f05cb7fe640 1 -- 172.21.15.115:0/3080203549 --> v1:172.21.15.1:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f05cc0ab6b0 con 0x7f05cc13ff20 2024-03-25T18:26:42.860 INFO:teuthology.orchestra.run.smithi115.stderr:2024-03-25T18:26:42.859+0000 7f05cb7fe640 1 -- 172.21.15.115:0/3080203549 <== mon.2 v1:172.21.15.1:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2500994192 0 0) 0x7f05bc003520 con 0x7f05cc0b4b50 2024-03-25T18:26:42.861 INFO:teuthology.orchestra.run.smithi115.stderr:2024-03-25T18:26:42.859+0000 7f05cb7fe640 1 -- 172.21.15.115:0/3080203549 >> v1:172.21.15.115:6789/0 conn(0x7f05cc142dd0 legacy=0x7f05cc14df50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:26:42.861 INFO:teuthology.orchestra.run.smithi115.stderr:2024-03-25T18:26:42.859+0000 7f05cb7fe640 1 -- 172.21.15.115:0/3080203549 >> v1:172.21.15.1:6789/0 conn(0x7f05cc13ff20 legacy=0x7f05cc1426c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:26:42.861 INFO:teuthology.orchestra.run.smithi115.stderr:2024-03-25T18:26:42.859+0000 7f05cb7fe640 10 monclient: _finish_hunting 0 2024-03-25T18:26:42.861 INFO:teuthology.orchestra.run.smithi115.stderr:2024-03-25T18:26:42.859+0000 7f05cb7fe640 1 monclient: found mon.noname-b 2024-03-25T18:26:42.861 INFO:teuthology.orchestra.run.smithi115.stderr:2024-03-25T18:26:42.859+0000 7f05cb7fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:26:42.861 INFO:teuthology.orchestra.run.smithi115.stderr:2024-03-25T18:26:42.859+0000 7f05cb7fe640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.1:6790/0 2024-03-25T18:26:42.861 INFO:teuthology.orchestra.run.smithi115.stderr:2024-03-25T18:26:42.859+0000 7f05cb7fe640 1 -- 172.21.15.115:0/3080203549 --> v1:172.21.15.1:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f05cc0aa0a0 con 0x7f05cc0b4b50 2024-03-25T18:26:42.861 INFO:teuthology.orchestra.run.smithi115.stderr:2024-03-25T18:26:42.859+0000 7f05cb7fe640 10 monclient: _finish_auth 0 2024-03-25T18:26:42.861 INFO:teuthology.orchestra.run.smithi115.stderr:2024-03-25T18:26:42.859+0000 7f05cb7fe640 10 monclient: _check_auth_tickets 2024-03-25T18:26:42.861 INFO:teuthology.orchestra.run.smithi115.stderr:2024-03-25T18:26:42.859+0000 7f05cb7fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:26:42.861 INFO:teuthology.orchestra.run.smithi115.stderr:2024-03-25T18:26:42.859+0000 7f05d2cc8640 5 monclient: authenticate success, global_id 4136 2024-03-25T18:26:42.861 INFO:teuthology.orchestra.run.smithi115.stderr:2024-03-25T18:26:42.859+0000 7f05cb7fe640 1 -- 172.21.15.115:0/3080203549 <== mon.2 v1:172.21.15.1:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1586965025 0 0) 0x7f05c0001690 con 0x7f05cc0b4b50 2024-03-25T18:26:42.861 INFO:teuthology.orchestra.run.smithi115.stderr:2024-03-25T18:26:42.859+0000 7f05cb7fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:26:42.861 INFO:teuthology.orchestra.run.smithi115.stderr:2024-03-25T18:26:42.859+0000 7f05cb7fe640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-03-25T18:26:42.862 INFO:teuthology.orchestra.run.smithi115.stderr:2024-03-25T18:26:42.859+0000 7f05cb7fe640 10 monclient: dump: 2024-03-25T18:26:42.862 INFO:teuthology.orchestra.run.smithi115.stderr:epoch 1 2024-03-25T18:26:42.862 INFO:teuthology.orchestra.run.smithi115.stderr:fsid c8ea72cf-9e99-4383-9b52-6c23d365fdfc 2024-03-25T18:26:42.862 INFO:teuthology.orchestra.run.smithi115.stderr:last_changed 2024-03-25T18:25:50.510519+0000 2024-03-25T18:26:42.862 INFO:teuthology.orchestra.run.smithi115.stderr:created 2024-03-25T18:25:50.510519+0000 2024-03-25T18:26:42.862 INFO:teuthology.orchestra.run.smithi115.stderr:min_mon_release 19 (squid) 2024-03-25T18:26:42.862 INFO:teuthology.orchestra.run.smithi115.stderr:election_strategy: 3 2024-03-25T18:26:42.862 INFO:teuthology.orchestra.run.smithi115.stderr:0: v1:172.21.15.1:6789/0 mon.a 2024-03-25T18:26:42.862 INFO:teuthology.orchestra.run.smithi115.stderr:1: v1:172.21.15.115:6789/0 mon.b 2024-03-25T18:26:42.862 INFO:teuthology.orchestra.run.smithi115.stderr:2: v1:172.21.15.1:6790/0 mon.c 2024-03-25T18:26:42.862 INFO:teuthology.orchestra.run.smithi115.stderr: 2024-03-25T18:26:42.863 INFO:teuthology.orchestra.run.smithi115.stderr:2024-03-25T18:26:42.859+0000 7f05d2cc8640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T18:26:42.865 INFO:teuthology.orchestra.run.smithi115.stderr:2024-03-25T18:26:42.861+0000 7f05cb7fe640 1 -- 172.21.15.115:0/3080203549 <== mon.2 v1:172.21.15.1:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f05c00017f0 con 0x7f05cc0b4b50 2024-03-25T18:26:42.865 INFO:teuthology.orchestra.run.smithi115.stderr:2024-03-25T18:26:42.861+0000 7f05cb7fe640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:26:42.866 INFO:teuthology.orchestra.run.smithi115.stderr:2024-03-25T18:26:42.861+0000 7f05cb7fe640 1 -- 172.21.15.115:0/3080203549 <== mon.2 v1:172.21.15.1:6790/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1586965025 0 0) 0x7f05c0001b80 con 0x7f05cc0b4b50 2024-03-25T18:26:42.866 INFO:teuthology.orchestra.run.smithi115.stderr:2024-03-25T18:26:42.861+0000 7f05cb7fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:26:42.866 INFO:teuthology.orchestra.run.smithi115.stderr:2024-03-25T18:26:42.861+0000 7f05cb7fe640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-25T18:26:42.866 INFO:teuthology.orchestra.run.smithi115.stderr:2024-03-25T18:26:42.861+0000 7f05cb7fe640 10 monclient: dump: 2024-03-25T18:26:42.866 INFO:teuthology.orchestra.run.smithi115.stderr:epoch 1 2024-03-25T18:26:42.866 INFO:teuthology.orchestra.run.smithi115.stderr:fsid c8ea72cf-9e99-4383-9b52-6c23d365fdfc 2024-03-25T18:26:42.866 INFO:teuthology.orchestra.run.smithi115.stderr:last_changed 2024-03-25T18:25:50.510519+0000 2024-03-25T18:26:42.866 INFO:teuthology.orchestra.run.smithi115.stderr:created 2024-03-25T18:25:50.510519+0000 2024-03-25T18:26:42.866 INFO:teuthology.orchestra.run.smithi115.stderr:min_mon_release 19 (squid) 2024-03-25T18:26:42.866 INFO:teuthology.orchestra.run.smithi115.stderr:election_strategy: 3 2024-03-25T18:26:42.866 INFO:teuthology.orchestra.run.smithi115.stderr:0: v1:172.21.15.1:6789/0 mon.a 2024-03-25T18:26:42.866 INFO:teuthology.orchestra.run.smithi115.stderr:1: v1:172.21.15.115:6789/0 mon.b 2024-03-25T18:26:42.866 INFO:teuthology.orchestra.run.smithi115.stderr:2: v1:172.21.15.1:6790/0 mon.c 2024-03-25T18:26:42.866 INFO:teuthology.orchestra.run.smithi115.stderr: 2024-03-25T18:26:42.866 INFO:teuthology.orchestra.run.smithi115.stderr:2024-03-25T18:26:42.861+0000 7f05d2cc8640 10 monclient: get_monmap_and_config success 2024-03-25T18:26:42.867 INFO:teuthology.orchestra.run.smithi115.stderr:2024-03-25T18:26:42.861+0000 7f05d2cc8640 4 set_mon_vals no callback set 2024-03-25T18:26:42.867 INFO:teuthology.orchestra.run.smithi115.stderr:2024-03-25T18:26:42.862+0000 7f05d2cc8640 10 monclient: shutdown 2024-03-25T18:26:42.867 INFO:teuthology.orchestra.run.smithi115.stderr:2024-03-25T18:26:42.862+0000 7f05d2cc8640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:26:42.867 INFO:teuthology.orchestra.run.smithi115.stderr:2024-03-25T18:26:42.862+0000 7f05d2cc8640 1 -- 172.21.15.115:0/3080203549 >> v1:172.21.15.1:6790/0 conn(0x7f05cc0b4b50 legacy=0x7f05cc13f810 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:26:42.867 INFO:teuthology.orchestra.run.smithi115.stderr:2024-03-25T18:26:42.862+0000 7f05d2cc8640 1 -- 172.21.15.115:0/3080203549 shutdown_connections 2024-03-25T18:26:42.867 INFO:teuthology.orchestra.run.smithi115.stderr:2024-03-25T18:26:42.862+0000 7f05d2cc8640 1 -- 172.21.15.115:0/3080203549 >> 172.21.15.115:0/3080203549 conn(0x7f05cc066920 msgr2=0x7f05cc066d30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:26:42.867 INFO:teuthology.orchestra.run.smithi115.stderr:2024-03-25T18:26:42.862+0000 7f05d2cc8640 1 -- 172.21.15.115:0/3080203549 shutdown_connections 2024-03-25T18:26:42.867 INFO:teuthology.orchestra.run.smithi115.stderr:2024-03-25T18:26:42.862+0000 7f05d2cc8640 1 -- 172.21.15.115:0/3080203549 wait complete. 2024-03-25T18:26:42.867 INFO:teuthology.orchestra.run.smithi115.stderr:2024-03-25T18:26:42.862+0000 7f05d2cc8640 10 monclient: build_initial_monmap 2024-03-25T18:26:42.867 INFO:teuthology.orchestra.run.smithi115.stderr:2024-03-25T18:26:42.862+0000 7f05d2cc8640 10 monclient: monmap: 2024-03-25T18:26:42.867 INFO:teuthology.orchestra.run.smithi115.stderr:epoch 0 2024-03-25T18:26:42.867 INFO:teuthology.orchestra.run.smithi115.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T18:26:42.867 INFO:teuthology.orchestra.run.smithi115.stderr:last_changed 0.000000 2024-03-25T18:26:42.867 INFO:teuthology.orchestra.run.smithi115.stderr:created 0.000000 2024-03-25T18:26:42.868 INFO:teuthology.orchestra.run.smithi115.stderr:min_mon_release 0 (unknown) 2024-03-25T18:26:42.868 INFO:teuthology.orchestra.run.smithi115.stderr:election_strategy: 1 2024-03-25T18:26:42.868 INFO:teuthology.orchestra.run.smithi115.stderr:0: v1:172.21.15.1:6789/0 mon.noname-a 2024-03-25T18:26:42.868 INFO:teuthology.orchestra.run.smithi115.stderr:1: v1:172.21.15.115:6789/0 mon.noname-b 2024-03-25T18:26:42.868 INFO:teuthology.orchestra.run.smithi115.stderr:2: v1:172.21.15.1:6790/0 mon.noname-c 2024-03-25T18:26:42.868 INFO:teuthology.orchestra.run.smithi115.stderr: 2024-03-25T18:26:42.868 INFO:teuthology.orchestra.run.smithi115.stderr:2024-03-25T18:26:42.863+0000 7f05d2cc8640 1 librados: starting msgr at 2024-03-25T18:26:42.868 INFO:teuthology.orchestra.run.smithi115.stderr:2024-03-25T18:26:42.863+0000 7f05d2cc8640 1 librados: starting objecter 2024-03-25T18:26:42.868 INFO:teuthology.orchestra.run.smithi115.stderr:2024-03-25T18:26:42.863+0000 7f05d2cc8640 1 Processor -- start 2024-03-25T18:26:42.868 INFO:teuthology.orchestra.run.smithi115.stderr:2024-03-25T18:26:42.863+0000 7f05d2cc8640 1 -- start start 2024-03-25T18:26:42.868 INFO:teuthology.orchestra.run.smithi115.stderr:2024-03-25T18:26:42.863+0000 7f05d2cc8640 1 librados: setting wanted keys 2024-03-25T18:26:42.868 INFO:teuthology.orchestra.run.smithi115.stderr:2024-03-25T18:26:42.863+0000 7f05d2cc8640 1 librados: calling monclient init 2024-03-25T18:26:42.868 INFO:teuthology.orchestra.run.smithi115.stderr:2024-03-25T18:26:42.863+0000 7f05d2cc8640 10 monclient: init 2024-03-25T18:26:42.868 INFO:teuthology.orchestra.run.smithi115.stderr:2024-03-25T18:26:42.864+0000 7f05d2cc8640 10 monclient: _reopen_session rank -1 2024-03-25T18:26:42.868 INFO:teuthology.orchestra.run.smithi115.stderr:2024-03-25T18:26:42.864+0000 7f05d2cc8640 10 monclient: _add_conns ranks=[2,1,0] 2024-03-25T18:26:42.868 INFO:teuthology.orchestra.run.smithi115.stderr:2024-03-25T18:26:42.864+0000 7f05d2cc8640 10 monclient(hunting): picked mon.noname-c con 0x7f05cc0b4b50 addr v1:172.21.15.1:6790/0 2024-03-25T18:26:42.869 INFO:teuthology.orchestra.run.smithi115.stderr:2024-03-25T18:26:42.864+0000 7f05d2cc8640 10 monclient(hunting): picked mon.noname-b con 0x7f05cc13ff20 addr v1:172.21.15.115:6789/0 2024-03-25T18:26:42.869 INFO:teuthology.orchestra.run.smithi115.stderr:2024-03-25T18:26:42.864+0000 7f05d2cc8640 10 monclient(hunting): picked mon.noname-a con 0x7f05cc142dd0 addr v1:172.21.15.1:6789/0 2024-03-25T18:26:42.869 INFO:teuthology.orchestra.run.smithi115.stderr:2024-03-25T18:26:42.864+0000 7f05d2cc8640 1 -- --> v1:172.21.15.1:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f05cc144710 con 0x7f05cc142dd0 2024-03-25T18:26:42.869 INFO:teuthology.orchestra.run.smithi115.stderr:2024-03-25T18:26:42.864+0000 7f05d2cc8640 1 -- --> v1:172.21.15.115:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f05cc140430 con 0x7f05cc13ff20 2024-03-25T18:26:42.869 INFO:teuthology.orchestra.run.smithi115.stderr:2024-03-25T18:26:42.864+0000 7f05d2cc8640 1 -- --> v1:172.21.15.1:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f05cc1412e0 con 0x7f05cc0b4b50 2024-03-25T18:26:42.869 INFO:teuthology.orchestra.run.smithi115.stderr:2024-03-25T18:26:42.864+0000 7f05d2cc8640 10 monclient(hunting): _renew_subs 2024-03-25T18:26:42.869 INFO:teuthology.orchestra.run.smithi115.stderr:2024-03-25T18:26:42.864+0000 7f05d2cc8640 10 monclient(hunting): authenticate will time out at 7494.020022s 2024-03-25T18:26:42.869 INFO:teuthology.orchestra.run.smithi115.stderr:2024-03-25T18:26:42.864+0000 7f05cbfff640 1 --1- >> v1:172.21.15.115:6789/0 conn(0x7f05cc13ff20 0x7f05cc144e10 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.115:6789/0 says I am v1:172.21.15.115:44632/0 (socket says 172.21.15.115:44632) 2024-03-25T18:26:42.869 INFO:teuthology.orchestra.run.smithi115.stderr:2024-03-25T18:26:42.864+0000 7f05cbfff640 1 -- 172.21.15.115:0/1577004584 learned_addr learned my addr 172.21.15.115:0/1577004584 (peer_addr_for_me v1:172.21.15.115:0/0) 2024-03-25T18:26:42.869 INFO:teuthology.orchestra.run.smithi115.stderr:2024-03-25T18:26:42.864+0000 7f05c97fa640 10 client.?.objecter ms_handle_connect 0x7f05cc142dd0 2024-03-25T18:26:42.869 INFO:teuthology.orchestra.run.smithi115.stderr:2024-03-25T18:26:42.864+0000 7f05c97fa640 10 client.?.objecter resend_mon_ops 2024-03-25T18:26:42.869 INFO:teuthology.orchestra.run.smithi115.stderr:2024-03-25T18:26:42.864+0000 7f05c97fa640 10 client.?.objecter ms_handle_connect 0x7f05cc0b4b50 2024-03-25T18:26:42.869 INFO:teuthology.orchestra.run.smithi115.stderr:2024-03-25T18:26:42.864+0000 7f05c97fa640 10 client.?.objecter resend_mon_ops 2024-03-25T18:26:42.869 INFO:teuthology.orchestra.run.smithi115.stderr:2024-03-25T18:26:42.865+0000 7f05c97fa640 10 client.?.objecter ms_handle_connect 0x7f05cc13ff20 2024-03-25T18:26:42.869 INFO:teuthology.orchestra.run.smithi115.stderr:2024-03-25T18:26:42.865+0000 7f05c97fa640 10 client.?.objecter resend_mon_ops 2024-03-25T18:26:42.870 INFO:teuthology.orchestra.run.smithi115.stderr:2024-03-25T18:26:42.866+0000 7f05c97fa640 1 -- 172.21.15.115:0/1577004584 <== mon.2 v1:172.21.15.1:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2168153382 0 0) 0x7f05cc1412e0 con 0x7f05cc0b4b50 2024-03-25T18:26:42.870 INFO:teuthology.orchestra.run.smithi115.stderr:2024-03-25T18:26:42.866+0000 7f05c97fa640 10 monclient(hunting): _negotiate 2024-03-25T18:26:42.870 INFO:teuthology.orchestra.run.smithi115.stderr:2024-03-25T18:26:42.866+0000 7f05c97fa640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:26:42.871 INFO:teuthology.orchestra.run.smithi115.stderr:2024-03-25T18:26:42.866+0000 7f05c97fa640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:26:42.871 INFO:teuthology.orchestra.run.smithi115.stderr:2024-03-25T18:26:42.866+0000 7f05c97fa640 10 monclient(hunting): my global_id is 4139 2024-03-25T18:26:42.871 INFO:teuthology.orchestra.run.smithi115.stderr:2024-03-25T18:26:42.866+0000 7f05c97fa640 1 -- 172.21.15.115:0/1577004584 --> v1:172.21.15.1:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f05b0004eb0 con 0x7f05cc0b4b50 2024-03-25T18:26:42.871 INFO:teuthology.orchestra.run.smithi115.stderr:2024-03-25T18:26:42.866+0000 7f05c97fa640 1 -- 172.21.15.115:0/1577004584 <== mon.1 v1:172.21.15.115:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1510521575 0 0) 0x7f05cc140430 con 0x7f05cc13ff20 2024-03-25T18:26:42.871 INFO:teuthology.orchestra.run.smithi115.stderr:2024-03-25T18:26:42.866+0000 7f05c97fa640 10 monclient(hunting): _negotiate 2024-03-25T18:26:42.871 INFO:teuthology.orchestra.run.smithi115.stderr:2024-03-25T18:26:42.866+0000 7f05c97fa640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:26:42.871 INFO:teuthology.orchestra.run.smithi115.stderr:2024-03-25T18:26:42.866+0000 7f05c97fa640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:26:42.871 INFO:teuthology.orchestra.run.smithi115.stderr:2024-03-25T18:26:42.866+0000 7f05c97fa640 10 monclient(hunting): my global_id is 4135 2024-03-25T18:26:42.871 INFO:teuthology.orchestra.run.smithi115.stderr:2024-03-25T18:26:42.866+0000 7f05c97fa640 1 -- 172.21.15.115:0/1577004584 --> v1:172.21.15.115:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f05cc1412e0 con 0x7f05cc13ff20 2024-03-25T18:26:42.871 INFO:teuthology.orchestra.run.smithi115.stderr:2024-03-25T18:26:42.867+0000 7f05c97fa640 1 -- 172.21.15.115:0/1577004584 <== mon.0 v1:172.21.15.1:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1274331070 0 0) 0x7f05cc144710 con 0x7f05cc142dd0 2024-03-25T18:26:42.871 INFO:teuthology.orchestra.run.smithi115.stderr:2024-03-25T18:26:42.867+0000 7f05c97fa640 10 monclient(hunting): _negotiate 2024-03-25T18:26:42.871 INFO:teuthology.orchestra.run.smithi115.stderr:2024-03-25T18:26:42.867+0000 7f05c97fa640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:26:42.871 INFO:teuthology.orchestra.run.smithi115.stderr:2024-03-25T18:26:42.867+0000 7f05c97fa640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:26:42.871 INFO:teuthology.orchestra.run.smithi115.stderr:2024-03-25T18:26:42.867+0000 7f05c97fa640 10 monclient(hunting): my global_id is 4140 2024-03-25T18:26:42.871 INFO:teuthology.orchestra.run.smithi115.stderr:2024-03-25T18:26:42.867+0000 7f05c97fa640 1 -- 172.21.15.115:0/1577004584 --> v1:172.21.15.1:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f05cc140430 con 0x7f05cc142dd0 2024-03-25T18:26:42.872 INFO:teuthology.orchestra.run.smithi115.stderr:2024-03-25T18:26:42.871+0000 7f05c97fa640 1 -- 172.21.15.115:0/1577004584 <== mon.2 v1:172.21.15.1:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1857070515 0 0) 0x7f05b0004eb0 con 0x7f05cc0b4b50 2024-03-25T18:26:42.872 INFO:teuthology.orchestra.run.smithi115.stderr:2024-03-25T18:26:42.871+0000 7f05c97fa640 1 -- 172.21.15.115:0/1577004584 --> v1:172.21.15.1:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f05cc144710 con 0x7f05cc0b4b50 2024-03-25T18:26:42.872 INFO:teuthology.orchestra.run.smithi115.stderr:2024-03-25T18:26:42.871+0000 7f05c97fa640 1 -- 172.21.15.115:0/1577004584 <== mon.2 v1:172.21.15.1:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1586965025 0 0) 0x7f05c0001b00 con 0x7f05cc0b4b50 2024-03-25T18:26:42.872 INFO:teuthology.orchestra.run.smithi115.stderr:2024-03-25T18:26:42.871+0000 7f05c97fa640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:26:42.872 INFO:teuthology.orchestra.run.smithi115.stderr:2024-03-25T18:26:42.871+0000 7f05c97fa640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-03-25T18:26:42.872 INFO:teuthology.orchestra.run.smithi115.stderr:2024-03-25T18:26:42.871+0000 7f05c97fa640 10 monclient(hunting): dump: 2024-03-25T18:26:42.872 INFO:teuthology.orchestra.run.smithi115.stderr:epoch 1 2024-03-25T18:26:42.872 INFO:teuthology.orchestra.run.smithi115.stderr:fsid c8ea72cf-9e99-4383-9b52-6c23d365fdfc 2024-03-25T18:26:42.873 INFO:teuthology.orchestra.run.smithi115.stderr:last_changed 2024-03-25T18:25:50.510519+0000 2024-03-25T18:26:42.873 INFO:teuthology.orchestra.run.smithi115.stderr:created 2024-03-25T18:25:50.510519+0000 2024-03-25T18:26:42.873 INFO:teuthology.orchestra.run.smithi115.stderr:min_mon_release 19 (squid) 2024-03-25T18:26:42.873 INFO:teuthology.orchestra.run.smithi115.stderr:election_strategy: 3 2024-03-25T18:26:42.873 INFO:teuthology.orchestra.run.smithi115.stderr:0: v1:172.21.15.1:6789/0 mon.a 2024-03-25T18:26:42.873 INFO:teuthology.orchestra.run.smithi115.stderr:1: v1:172.21.15.115:6789/0 mon.b 2024-03-25T18:26:42.873 INFO:teuthology.orchestra.run.smithi115.stderr:2: v1:172.21.15.1:6790/0 mon.c 2024-03-25T18:26:42.873 INFO:teuthology.orchestra.run.smithi115.stderr: 2024-03-25T18:26:42.873 INFO:teuthology.orchestra.run.smithi115.stderr:2024-03-25T18:26:42.871+0000 7f05c97fa640 10 monclient(hunting): _finish_auth 0 2024-03-25T18:26:42.874 INFO:teuthology.orchestra.run.smithi115.stderr:2024-03-25T18:26:42.872+0000 7f05c97fa640 1 -- 172.21.15.115:0/1577004584 <== mon.1 v1:172.21.15.115:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1931518134 0 0) 0x7f05cc1412e0 con 0x7f05cc13ff20 2024-03-25T18:26:42.874 INFO:teuthology.orchestra.run.smithi115.stderr:2024-03-25T18:26:42.872+0000 7f05c97fa640 1 -- 172.21.15.115:0/1577004584 --> v1:172.21.15.115:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f05b0004eb0 con 0x7f05cc13ff20 2024-03-25T18:26:42.874 INFO:teuthology.orchestra.run.smithi115.stderr:2024-03-25T18:26:42.872+0000 7f05c97fa640 1 -- 172.21.15.115:0/1577004584 <== mon.1 v1:172.21.15.115:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1586965025 0 0) 0x7f05b80018f0 con 0x7f05cc13ff20 2024-03-25T18:26:42.874 INFO:teuthology.orchestra.run.smithi115.stderr:2024-03-25T18:26:42.872+0000 7f05c97fa640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:26:42.874 INFO:teuthology.orchestra.run.smithi115.stderr:2024-03-25T18:26:42.872+0000 7f05c97fa640 10 monclient(hunting): got monmap 1 from mon.b (according to old e1) 2024-03-25T18:26:42.874 INFO:teuthology.orchestra.run.smithi115.stderr:2024-03-25T18:26:42.872+0000 7f05c97fa640 10 monclient(hunting): dump: 2024-03-25T18:26:42.874 INFO:teuthology.orchestra.run.smithi115.stderr:epoch 1 2024-03-25T18:26:42.874 INFO:teuthology.orchestra.run.smithi115.stderr:fsid c8ea72cf-9e99-4383-9b52-6c23d365fdfc 2024-03-25T18:26:42.874 INFO:teuthology.orchestra.run.smithi115.stderr:last_changed 2024-03-25T18:25:50.510519+0000 2024-03-25T18:26:42.874 INFO:teuthology.orchestra.run.smithi115.stderr:created 2024-03-25T18:25:50.510519+0000 2024-03-25T18:26:42.874 INFO:teuthology.orchestra.run.smithi115.stderr:min_mon_release 19 (squid) 2024-03-25T18:26:42.874 INFO:teuthology.orchestra.run.smithi115.stderr:election_strategy: 3 2024-03-25T18:26:42.875 INFO:teuthology.orchestra.run.smithi115.stderr:0: v1:172.21.15.1:6789/0 mon.a 2024-03-25T18:26:42.875 INFO:teuthology.orchestra.run.smithi115.stderr:1: v1:172.21.15.115:6789/0 mon.b 2024-03-25T18:26:42.875 INFO:teuthology.orchestra.run.smithi115.stderr:2: v1:172.21.15.1:6790/0 mon.c 2024-03-25T18:26:42.875 INFO:teuthology.orchestra.run.smithi115.stderr: 2024-03-25T18:26:42.876 INFO:teuthology.orchestra.run.smithi115.stderr:2024-03-25T18:26:42.874+0000 7f05c97fa640 1 -- 172.21.15.115:0/1577004584 <== mon.0 v1:172.21.15.1:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3131128661 0 0) 0x7f05cc140430 con 0x7f05cc142dd0 2024-03-25T18:26:42.877 INFO:teuthology.orchestra.run.smithi115.stderr:2024-03-25T18:26:42.874+0000 7f05c97fa640 1 -- 172.21.15.115:0/1577004584 --> v1:172.21.15.1:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f05cc1412e0 con 0x7f05cc142dd0 2024-03-25T18:26:42.877 INFO:teuthology.orchestra.run.smithi115.stderr:2024-03-25T18:26:42.874+0000 7f05c97fa640 1 -- 172.21.15.115:0/1577004584 <== mon.0 v1:172.21.15.1:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1586965025 0 0) 0x7f05c4001bd0 con 0x7f05cc142dd0 2024-03-25T18:26:42.877 INFO:teuthology.orchestra.run.smithi115.stderr:2024-03-25T18:26:42.874+0000 7f05c97fa640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:26:42.877 INFO:teuthology.orchestra.run.smithi115.stderr:2024-03-25T18:26:42.874+0000 7f05c97fa640 10 monclient(hunting): got monmap 1 from mon.a (according to old e1) 2024-03-25T18:26:42.877 INFO:teuthology.orchestra.run.smithi115.stderr:2024-03-25T18:26:42.874+0000 7f05c97fa640 10 monclient(hunting): dump: 2024-03-25T18:26:42.877 INFO:teuthology.orchestra.run.smithi115.stderr:epoch 1 2024-03-25T18:26:42.877 INFO:teuthology.orchestra.run.smithi115.stderr:fsid c8ea72cf-9e99-4383-9b52-6c23d365fdfc 2024-03-25T18:26:42.877 INFO:teuthology.orchestra.run.smithi115.stderr:last_changed 2024-03-25T18:25:50.510519+0000 2024-03-25T18:26:42.877 INFO:teuthology.orchestra.run.smithi115.stderr:created 2024-03-25T18:25:50.510519+0000 2024-03-25T18:26:42.877 INFO:teuthology.orchestra.run.smithi115.stderr:min_mon_release 19 (squid) 2024-03-25T18:26:42.877 INFO:teuthology.orchestra.run.smithi115.stderr:election_strategy: 3 2024-03-25T18:26:42.877 INFO:teuthology.orchestra.run.smithi115.stderr:0: v1:172.21.15.1:6789/0 mon.a 2024-03-25T18:26:42.877 INFO:teuthology.orchestra.run.smithi115.stderr:1: v1:172.21.15.115:6789/0 mon.b 2024-03-25T18:26:42.877 INFO:teuthology.orchestra.run.smithi115.stderr:2: v1:172.21.15.1:6790/0 mon.c 2024-03-25T18:26:42.878 INFO:teuthology.orchestra.run.smithi115.stderr: 2024-03-25T18:26:42.878 INFO:teuthology.orchestra.run.smithi115.stderr:2024-03-25T18:26:42.875+0000 7f05c97fa640 1 -- 172.21.15.115:0/1577004584 <== mon.2 v1:172.21.15.1:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 2286534852 0 0) 0x7f05cc144710 con 0x7f05cc0b4b50 2024-03-25T18:26:42.878 INFO:teuthology.orchestra.run.smithi115.stderr:2024-03-25T18:26:42.875+0000 7f05c97fa640 1 -- 172.21.15.115:0/1577004584 >> v1:172.21.15.115:6789/0 conn(0x7f05cc13ff20 legacy=0x7f05cc144e10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:26:42.878 INFO:teuthology.orchestra.run.smithi115.stderr:2024-03-25T18:26:42.875+0000 7f05c97fa640 1 -- 172.21.15.115:0/1577004584 >> v1:172.21.15.1:6789/0 conn(0x7f05cc142dd0 legacy=0x7f05cc145520 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:26:42.878 INFO:teuthology.orchestra.run.smithi115.stderr:2024-03-25T18:26:42.875+0000 7f05c97fa640 10 monclient: _finish_hunting 0 2024-03-25T18:26:42.878 INFO:teuthology.orchestra.run.smithi115.stderr:2024-03-25T18:26:42.875+0000 7f05c97fa640 1 monclient: found mon.c 2024-03-25T18:26:42.878 INFO:teuthology.orchestra.run.smithi115.stderr:2024-03-25T18:26:42.875+0000 7f05c97fa640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:26:42.878 INFO:teuthology.orchestra.run.smithi115.stderr:2024-03-25T18:26:42.875+0000 7f05c97fa640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.1:6790/0 2024-03-25T18:26:42.878 INFO:teuthology.orchestra.run.smithi115.stderr:2024-03-25T18:26:42.875+0000 7f05c97fa640 1 -- 172.21.15.115:0/1577004584 --> v1:172.21.15.1:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f05cc141470 con 0x7f05cc0b4b50 2024-03-25T18:26:42.878 INFO:teuthology.orchestra.run.smithi115.stderr:2024-03-25T18:26:42.875+0000 7f05c97fa640 10 monclient: _finish_auth 0 2024-03-25T18:26:42.878 INFO:teuthology.orchestra.run.smithi115.stderr:2024-03-25T18:26:42.875+0000 7f05c97fa640 10 monclient: _check_auth_tickets 2024-03-25T18:26:42.878 INFO:teuthology.orchestra.run.smithi115.stderr:2024-03-25T18:26:42.875+0000 7f05c97fa640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:26:42.878 INFO:teuthology.orchestra.run.smithi115.stderr:2024-03-25T18:26:42.875+0000 7f05d2cc8640 5 monclient: authenticate success, global_id 4139 2024-03-25T18:26:42.878 INFO:teuthology.orchestra.run.smithi115.stderr:2024-03-25T18:26:42.875+0000 7f05d2cc8640 10 monclient: _renew_subs 2024-03-25T18:26:42.878 INFO:teuthology.orchestra.run.smithi115.stderr:2024-03-25T18:26:42.875+0000 7f05d2cc8640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.1:6790/0 2024-03-25T18:26:42.879 INFO:teuthology.orchestra.run.smithi115.stderr:2024-03-25T18:26:42.875+0000 7f05d2cc8640 1 -- 172.21.15.115:0/1577004584 --> v1:172.21.15.1:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f05cc134a20 con 0x7f05cc0b4b50 2024-03-25T18:26:42.879 INFO:teuthology.orchestra.run.smithi115.stderr:2024-03-25T18:26:42.875+0000 7f05d2cc8640 10 client.4139.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T18:26:42.879 INFO:teuthology.orchestra.run.smithi115.stderr:2024-03-25T18:26:42.875+0000 7f05d2cc8640 10 monclient: _renew_subs 2024-03-25T18:26:42.879 INFO:teuthology.orchestra.run.smithi115.stderr:2024-03-25T18:26:42.875+0000 7f05d2cc8640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.1:6790/0 2024-03-25T18:26:42.879 INFO:teuthology.orchestra.run.smithi115.stderr:2024-03-25T18:26:42.875+0000 7f05d2cc8640 1 -- 172.21.15.115:0/1577004584 --> v1:172.21.15.1:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f05cc141900 con 0x7f05cc0b4b50 2024-03-25T18:26:42.879 INFO:teuthology.orchestra.run.smithi115.stderr:2024-03-25T18:26:42.875+0000 7f05d2cc8640 1 librados: init done 2024-03-25T18:26:42.879 INFO:teuthology.orchestra.run.smithi115.stderr:2024-03-25T18:26:42.876+0000 7f05d2cc8640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T18:26:42.879 INFO:teuthology.orchestra.run.smithi115.stderr:2024-03-25T18:26:42.876+0000 7f05d2cc8640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:26:42.879 INFO:teuthology.orchestra.run.smithi115.stderr:2024-03-25T18:26:42.876+0000 7f05d2cc8640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.1:6790/0 2024-03-25T18:26:42.879 INFO:teuthology.orchestra.run.smithi115.stderr:2024-03-25T18:26:42.876+0000 7f05d2cc8640 1 -- 172.21.15.115:0/1577004584 --> v1:172.21.15.1:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f05cc141660 con 0x7f05cc0b4b50 2024-03-25T18:26:42.880 INFO:teuthology.orchestra.run.smithi115.stderr:2024-03-25T18:26:42.879+0000 7f05c97fa640 1 -- 172.21.15.115:0/1577004584 <== mon.2 v1:172.21.15.1:6790/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f05c0001120 con 0x7f05cc0b4b50 2024-03-25T18:26:42.880 INFO:teuthology.orchestra.run.smithi115.stderr:2024-03-25T18:26:42.879+0000 7f05c97fa640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:26:42.880 INFO:teuthology.orchestra.run.smithi115.stderr:2024-03-25T18:26:42.879+0000 7f05c97fa640 1 -- 172.21.15.115:0/1577004584 <== mon.2 v1:172.21.15.1:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1586965025 0 0) 0x7f05c0003a40 con 0x7f05cc0b4b50 2024-03-25T18:26:42.880 INFO:teuthology.orchestra.run.smithi115.stderr:2024-03-25T18:26:42.879+0000 7f05c97fa640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:26:42.880 INFO:teuthology.orchestra.run.smithi115.stderr:2024-03-25T18:26:42.879+0000 7f05c97fa640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-25T18:26:42.880 INFO:teuthology.orchestra.run.smithi115.stderr:2024-03-25T18:26:42.879+0000 7f05c97fa640 10 monclient: dump: 2024-03-25T18:26:42.880 INFO:teuthology.orchestra.run.smithi115.stderr:epoch 1 2024-03-25T18:26:42.880 INFO:teuthology.orchestra.run.smithi115.stderr:fsid c8ea72cf-9e99-4383-9b52-6c23d365fdfc 2024-03-25T18:26:42.880 INFO:teuthology.orchestra.run.smithi115.stderr:last_changed 2024-03-25T18:25:50.510519+0000 2024-03-25T18:26:42.881 INFO:teuthology.orchestra.run.smithi115.stderr:created 2024-03-25T18:25:50.510519+0000 2024-03-25T18:26:42.881 INFO:teuthology.orchestra.run.smithi115.stderr:min_mon_release 19 (squid) 2024-03-25T18:26:42.881 INFO:teuthology.orchestra.run.smithi115.stderr:election_strategy: 3 2024-03-25T18:26:42.881 INFO:teuthology.orchestra.run.smithi115.stderr:0: v1:172.21.15.1:6789/0 mon.a 2024-03-25T18:26:42.881 INFO:teuthology.orchestra.run.smithi115.stderr:1: v1:172.21.15.115:6789/0 mon.b 2024-03-25T18:26:42.881 INFO:teuthology.orchestra.run.smithi115.stderr:2: v1:172.21.15.1:6790/0 mon.c 2024-03-25T18:26:42.881 INFO:teuthology.orchestra.run.smithi115.stderr: 2024-03-25T18:26:42.881 INFO:teuthology.orchestra.run.smithi115.stderr:2024-03-25T18:26:42.879+0000 7f05caffd640 4 set_mon_vals no callback set 2024-03-25T18:26:42.881 INFO:teuthology.orchestra.run.smithi115.stderr:2024-03-25T18:26:42.879+0000 7f05c97fa640 1 -- 172.21.15.115:0/1577004584 <== mon.2 v1:172.21.15.1:6790/0 7 ==== mgrmap(e 1) v1 ==== 807+0+0 (unknown 3189685207 0 0) 0x7f05c0003f30 con 0x7f05cc0b4b50 2024-03-25T18:26:42.881 INFO:teuthology.orchestra.run.smithi115.stderr:2024-03-25T18:26:42.879+0000 7f05c97fa640 1 -- 172.21.15.115:0/1577004584 <== mon.2 v1:172.21.15.1:6790/0 8 ==== osd_map(6..6 src has 1..6) v4 ==== 1185+0+0 (unknown 876495579 0 0) 0x7f05c0005500 con 0x7f05cc0b4b50 2024-03-25T18:26:42.881 INFO:teuthology.orchestra.run.smithi115.stderr:2024-03-25T18:26:42.879+0000 7f05c97fa640 10 client.4139.objecter ms_dispatch 0x7f05cc000b90 osd_map(6..6 src has 1..6) v4 2024-03-25T18:26:42.881 INFO:teuthology.orchestra.run.smithi115.stderr:2024-03-25T18:26:42.879+0000 7f05c97fa640 3 client.4139.objecter handle_osd_map got epochs [6,6] > 0 2024-03-25T18:26:42.881 INFO:teuthology.orchestra.run.smithi115.stderr:2024-03-25T18:26:42.879+0000 7f05c97fa640 3 client.4139.objecter handle_osd_map decoding full epoch 6 2024-03-25T18:26:42.881 INFO:teuthology.orchestra.run.smithi115.stderr:2024-03-25T18:26:42.879+0000 7f05c97fa640 20 client.4139.objecter dump_active .. 0 homeless 2024-03-25T18:26:42.982 INFO:teuthology.orchestra.run.smithi115.stderr:2024-03-25T18:26:42.981+0000 7f05c97fa640 1 -- 172.21.15.115:0/1577004584 <== mon.2 v1:172.21.15.1:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+73141 (unknown 1092875540 0 2031569661) 0x7f05c0005100 con 0x7f05cc0b4b50 2024-03-25T18:26:42.982 INFO:teuthology.orchestra.run.smithi115.stderr:2024-03-25T18:26:42.981+0000 7f05c97fa640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:26:42.982 INFO:teuthology.orchestra.run.smithi115.stderr:2024-03-25T18:26:42.981+0000 7f05c97fa640 10 monclient: _finish_command 1 = system:0 2024-03-25T18:26:42.994 INFO:tasks.ceph.mgr.y.smithi001.stderr:2024-03-25T18:26:42.993+0000 7faca7441040 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2024-03-25T18:26:43.038 INFO:tasks.ceph.mgr.y.smithi001.stderr:2024-03-25T18:26:43.037+0000 7faca7441040 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2024-03-25T18:26:43.054 INFO:teuthology.orchestra.run.smithi115.stderr:2024-03-25T18:26:43.053+0000 7f05d2cc8640 10 start_mon_command cmd=[{"prefix": "osd new", "uuid": "831a5a09-73ab-4a61-9872-088e76bdbc66", "id": 4}] 2024-03-25T18:26:43.054 INFO:teuthology.orchestra.run.smithi115.stderr:2024-03-25T18:26:43.053+0000 7f05d2cc8640 10 monclient: _send_command 2 [{"prefix": "osd new", "uuid": "831a5a09-73ab-4a61-9872-088e76bdbc66", "id": 4}] 2024-03-25T18:26:43.055 INFO:teuthology.orchestra.run.smithi115.stderr:2024-03-25T18:26:43.053+0000 7f05d2cc8640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.1:6790/0 2024-03-25T18:26:43.055 INFO:teuthology.orchestra.run.smithi115.stderr:2024-03-25T18:26:43.053+0000 7f05d2cc8640 1 -- 172.21.15.115:0/1577004584 --> v1:172.21.15.1:6790/0 -- mon_command({"prefix": "osd new", "uuid": "831a5a09-73ab-4a61-9872-088e76bdbc66", "id": 4} v 0) v1 -- 0x7f05cc04f730 con 0x7f05cc0b4b50 2024-03-25T18:26:43.135 INFO:teuthology.orchestra.run.smithi115.stderr:2024-03-25T18:26:43.134+0000 7f05c97fa640 1 -- 172.21.15.115:0/1577004584 <== mon.2 v1:172.21.15.1:6790/0 10 ==== mon_command_ack([{"prefix": "osd new", "uuid": "831a5a09-73ab-4a61-9872-088e76bdbc66", "id": 4}]=0 v7) v1 ==== 112+0+2 (unknown 903419026 0 1697012583) 0x7f05c0005710 con 0x7f05cc0b4b50 2024-03-25T18:26:43.135 INFO:teuthology.orchestra.run.smithi115.stderr:2024-03-25T18:26:43.134+0000 7f05c97fa640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd new", "uuid": "831a5a09-73ab-4a61-9872-088e76bdbc66", "id": 4}] 2024-03-25T18:26:43.135 INFO:teuthology.orchestra.run.smithi115.stderr:2024-03-25T18:26:43.134+0000 7f05c97fa640 10 monclient: _finish_command 2 = system:0 2024-03-25T18:26:43.135 INFO:teuthology.orchestra.run.smithi115.stdout:4 2024-03-25T18:26:43.137 INFO:teuthology.orchestra.run.smithi115.stderr:2024-03-25T18:26:43.135+0000 7f05d2cc8640 10 librados: watch_flush enter 2024-03-25T18:26:43.137 INFO:teuthology.orchestra.run.smithi115.stderr:2024-03-25T18:26:43.135+0000 7f05d2cc8640 10 librados: watch_flush exit 2024-03-25T18:26:43.137 INFO:teuthology.orchestra.run.smithi115.stderr:2024-03-25T18:26:43.135+0000 7f05d2cc8640 20 client.4139.objecter shutdown clearing up homeless session... 2024-03-25T18:26:43.137 INFO:teuthology.orchestra.run.smithi115.stderr:2024-03-25T18:26:43.135+0000 7f05d2cc8640 10 client.4139.objecter successfully canceled tick 2024-03-25T18:26:43.137 INFO:teuthology.orchestra.run.smithi115.stderr:2024-03-25T18:26:43.135+0000 7f05d2cc8640 10 monclient: shutdown 2024-03-25T18:26:43.137 INFO:teuthology.orchestra.run.smithi115.stderr:2024-03-25T18:26:43.135+0000 7f05d2cc8640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:26:43.137 INFO:teuthology.orchestra.run.smithi115.stderr:2024-03-25T18:26:43.135+0000 7f05d2cc8640 1 -- 172.21.15.115:0/1577004584 >> v1:172.21.15.1:6790/0 conn(0x7f05cc0b4b50 legacy=0x7f05cc134310 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:26:43.137 INFO:teuthology.orchestra.run.smithi115.stderr:2024-03-25T18:26:43.135+0000 7f05d2cc8640 1 -- 172.21.15.115:0/1577004584 shutdown_connections 2024-03-25T18:26:43.137 INFO:teuthology.orchestra.run.smithi115.stderr:2024-03-25T18:26:43.135+0000 7f05d2cc8640 1 -- 172.21.15.115:0/1577004584 >> 172.21.15.115:0/1577004584 conn(0x7f05cc066920 msgr2=0x7f05cc13f5e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:26:43.138 INFO:teuthology.orchestra.run.smithi115.stderr:2024-03-25T18:26:43.136+0000 7f05d2cc8640 1 -- 172.21.15.115:0/1577004584 shutdown_connections 2024-03-25T18:26:43.138 INFO:teuthology.orchestra.run.smithi115.stderr:2024-03-25T18:26:43.136+0000 7f05d2cc8640 1 -- 172.21.15.115:0/1577004584 wait complete. 2024-03-25T18:26:43.138 INFO:teuthology.orchestra.run.smithi115.stderr:2024-03-25T18:26:43.136+0000 7f05d2cc8640 1 librados: shutdown 2024-03-25T18:26:43.150 DEBUG:teuthology.orchestra.run.smithi115:> sudo ceph --cluster ceph osd new 88cd903d-3b55-4840-9396-6e3cfe005282 5 2024-03-25T18:26:43.202 INFO:tasks.ceph.mgr.y.smithi001.stderr:2024-03-25T18:26:43.201+0000 7faca7441040 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2024-03-25T18:26:43.249 INFO:teuthology.orchestra.run.smithi115.stderr:2024-03-25T18:26:43.248+0000 7f89af4ba640 10 monclient: get_monmap_and_config 2024-03-25T18:26:43.249 INFO:teuthology.orchestra.run.smithi115.stderr:2024-03-25T18:26:43.248+0000 7f89af4ba640 10 monclient: build_initial_monmap 2024-03-25T18:26:43.249 INFO:teuthology.orchestra.run.smithi115.stderr:2024-03-25T18:26:43.248+0000 7f89af4ba640 10 monclient: monmap: 2024-03-25T18:26:43.249 INFO:teuthology.orchestra.run.smithi115.stderr:epoch 0 2024-03-25T18:26:43.249 INFO:teuthology.orchestra.run.smithi115.stderr:fsid c8ea72cf-9e99-4383-9b52-6c23d365fdfc 2024-03-25T18:26:43.249 INFO:teuthology.orchestra.run.smithi115.stderr:last_changed 2024-03-25T18:26:43.249028+0000 2024-03-25T18:26:43.249 INFO:teuthology.orchestra.run.smithi115.stderr:created 2024-03-25T18:26:43.249028+0000 2024-03-25T18:26:43.249 INFO:teuthology.orchestra.run.smithi115.stderr:min_mon_release 0 (unknown) 2024-03-25T18:26:43.249 INFO:teuthology.orchestra.run.smithi115.stderr:election_strategy: 3 2024-03-25T18:26:43.249 INFO:teuthology.orchestra.run.smithi115.stderr:0: v1:172.21.15.1:6789/0 mon.noname-a 2024-03-25T18:26:43.249 INFO:teuthology.orchestra.run.smithi115.stderr:1: v1:172.21.15.115:6789/0 mon.noname-c 2024-03-25T18:26:43.249 INFO:teuthology.orchestra.run.smithi115.stderr:2: v1:172.21.15.1:6790/0 mon.noname-b 2024-03-25T18:26:43.250 INFO:teuthology.orchestra.run.smithi115.stderr: 2024-03-25T18:26:43.251 INFO:teuthology.orchestra.run.smithi115.stderr:2024-03-25T18:26:43.248+0000 7f89af4ba640 1 Processor -- start 2024-03-25T18:26:43.251 INFO:teuthology.orchestra.run.smithi115.stderr:2024-03-25T18:26:43.248+0000 7f89af4ba640 1 -- start start 2024-03-25T18:26:43.251 INFO:teuthology.orchestra.run.smithi115.stderr:2024-03-25T18:26:43.248+0000 7f89af4ba640 10 monclient: init 2024-03-25T18:26:43.251 INFO:teuthology.orchestra.run.smithi115.stderr:2024-03-25T18:26:43.249+0000 7f89af4ba640 10 monclient: _reopen_session rank -1 2024-03-25T18:26:43.251 INFO:teuthology.orchestra.run.smithi115.stderr:2024-03-25T18:26:43.249+0000 7f89af4ba640 10 monclient: _add_conns ranks=[0,1,2] 2024-03-25T18:26:43.251 INFO:teuthology.orchestra.run.smithi115.stderr:2024-03-25T18:26:43.249+0000 7f89af4ba640 10 monclient(hunting): picked mon.noname-a con 0x7f89a80b4b50 addr v1:172.21.15.1:6789/0 2024-03-25T18:26:43.251 INFO:teuthology.orchestra.run.smithi115.stderr:2024-03-25T18:26:43.249+0000 7f89af4ba640 10 monclient(hunting): picked mon.noname-c con 0x7f89a813ff20 addr v1:172.21.15.115:6789/0 2024-03-25T18:26:43.251 INFO:teuthology.orchestra.run.smithi115.stderr:2024-03-25T18:26:43.249+0000 7f89af4ba640 10 monclient(hunting): picked mon.noname-b con 0x7f89a8142dd0 addr v1:172.21.15.1:6790/0 2024-03-25T18:26:43.251 INFO:teuthology.orchestra.run.smithi115.stderr:2024-03-25T18:26:43.249+0000 7f89af4ba640 1 -- --> v1:172.21.15.1:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f89a80ab440 con 0x7f89a80b4b50 2024-03-25T18:26:43.251 INFO:teuthology.orchestra.run.smithi115.stderr:2024-03-25T18:26:43.249+0000 7f89af4ba640 1 -- --> v1:172.21.15.115:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f89a80ab6b0 con 0x7f89a813ff20 2024-03-25T18:26:43.251 INFO:teuthology.orchestra.run.smithi115.stderr:2024-03-25T18:26:43.249+0000 7f89af4ba640 1 -- --> v1:172.21.15.1:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f89a80a9f10 con 0x7f89a8142dd0 2024-03-25T18:26:43.252 INFO:teuthology.orchestra.run.smithi115.stderr:2024-03-25T18:26:43.249+0000 7f89af4ba640 10 monclient(hunting): _renew_subs 2024-03-25T18:26:43.252 INFO:teuthology.orchestra.run.smithi115.stderr:2024-03-25T18:26:43.249+0000 7f89af4ba640 10 monclient(hunting): authenticate will time out at 7494.405351s 2024-03-25T18:26:43.252 INFO:teuthology.orchestra.run.smithi115.stderr:2024-03-25T18:26:43.250+0000 7f89ad256640 1 --1- >> v1:172.21.15.1:6789/0 conn(0x7f89a80b4b50 0x7f89a813f810 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.1:6789/0 says I am v1:172.21.15.115:55624/0 (socket says 172.21.15.115:55624) 2024-03-25T18:26:43.252 INFO:teuthology.orchestra.run.smithi115.stderr:2024-03-25T18:26:43.250+0000 7f89ad256640 1 -- 172.21.15.115:0/3179955640 learned_addr learned my addr 172.21.15.115:0/3179955640 (peer_addr_for_me v1:172.21.15.115:0/0) 2024-03-25T18:26:43.253 INFO:teuthology.orchestra.run.smithi115.stderr:2024-03-25T18:26:43.252+0000 7f899ffff640 1 -- 172.21.15.115:0/3179955640 <== mon.1 v1:172.21.15.115:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4285095432 0 0) 0x7f89a80ab6b0 con 0x7f89a813ff20 2024-03-25T18:26:43.253 INFO:teuthology.orchestra.run.smithi115.stderr:2024-03-25T18:26:43.252+0000 7f899ffff640 10 monclient(hunting): _negotiate 2024-03-25T18:26:43.253 INFO:teuthology.orchestra.run.smithi115.stderr:2024-03-25T18:26:43.252+0000 7f899ffff640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:26:43.253 INFO:teuthology.orchestra.run.smithi115.stderr:2024-03-25T18:26:43.252+0000 7f899ffff640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:26:43.253 INFO:teuthology.orchestra.run.smithi115.stderr:2024-03-25T18:26:43.252+0000 7f899ffff640 10 monclient(hunting): my global_id is 4138 2024-03-25T18:26:43.253 INFO:teuthology.orchestra.run.smithi115.stderr:2024-03-25T18:26:43.252+0000 7f899ffff640 1 -- 172.21.15.115:0/3179955640 --> v1:172.21.15.115:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f8994003520 con 0x7f89a813ff20 2024-03-25T18:26:43.253 INFO:teuthology.orchestra.run.smithi115.stderr:2024-03-25T18:26:43.252+0000 7f899ffff640 1 -- 172.21.15.115:0/3179955640 <== mon.2 v1:172.21.15.1:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2170154833 0 0) 0x7f89a80a9f10 con 0x7f89a8142dd0 2024-03-25T18:26:43.253 INFO:teuthology.orchestra.run.smithi115.stderr:2024-03-25T18:26:43.252+0000 7f899ffff640 10 monclient(hunting): _negotiate 2024-03-25T18:26:43.254 INFO:teuthology.orchestra.run.smithi115.stderr:2024-03-25T18:26:43.252+0000 7f899ffff640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:26:43.254 INFO:teuthology.orchestra.run.smithi115.stderr:2024-03-25T18:26:43.252+0000 7f899ffff640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:26:43.254 INFO:teuthology.orchestra.run.smithi115.stderr:2024-03-25T18:26:43.252+0000 7f899ffff640 10 monclient(hunting): my global_id is 4142 2024-03-25T18:26:43.254 INFO:teuthology.orchestra.run.smithi115.stderr:2024-03-25T18:26:43.252+0000 7f899ffff640 1 -- 172.21.15.115:0/3179955640 --> v1:172.21.15.1:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f89a80ab6b0 con 0x7f89a8142dd0 2024-03-25T18:26:43.254 INFO:teuthology.orchestra.run.smithi115.stderr:2024-03-25T18:26:43.253+0000 7f899ffff640 1 -- 172.21.15.115:0/3179955640 <== mon.0 v1:172.21.15.1:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1069452356 0 0) 0x7f89a80ab440 con 0x7f89a80b4b50 2024-03-25T18:26:43.255 INFO:teuthology.orchestra.run.smithi115.stderr:2024-03-25T18:26:43.253+0000 7f899ffff640 10 monclient(hunting): _negotiate 2024-03-25T18:26:43.255 INFO:teuthology.orchestra.run.smithi115.stderr:2024-03-25T18:26:43.253+0000 7f899ffff640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:26:43.255 INFO:teuthology.orchestra.run.smithi115.stderr:2024-03-25T18:26:43.253+0000 7f899ffff640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:26:43.255 INFO:teuthology.orchestra.run.smithi115.stderr:2024-03-25T18:26:43.253+0000 7f899ffff640 10 monclient(hunting): my global_id is 4143 2024-03-25T18:26:43.255 INFO:teuthology.orchestra.run.smithi115.stderr:2024-03-25T18:26:43.253+0000 7f899ffff640 1 -- 172.21.15.115:0/3179955640 --> v1:172.21.15.1:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f89a80a9f10 con 0x7f89a80b4b50 2024-03-25T18:26:43.256 INFO:teuthology.orchestra.run.smithi115.stderr:2024-03-25T18:26:43.255+0000 7f899ffff640 1 -- 172.21.15.115:0/3179955640 <== mon.1 v1:172.21.15.115:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 4159882912 0 0) 0x7f8994003520 con 0x7f89a813ff20 2024-03-25T18:26:43.256 INFO:teuthology.orchestra.run.smithi115.stderr:2024-03-25T18:26:43.255+0000 7f899ffff640 1 -- 172.21.15.115:0/3179955640 >> v1:172.21.15.1:6790/0 conn(0x7f89a8142dd0 legacy=0x7f89a814df50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:26:43.256 INFO:teuthology.orchestra.run.smithi115.stderr:2024-03-25T18:26:43.255+0000 7f899ffff640 1 -- 172.21.15.115:0/3179955640 >> v1:172.21.15.1:6789/0 conn(0x7f89a80b4b50 legacy=0x7f89a813f810 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:26:43.256 INFO:teuthology.orchestra.run.smithi115.stderr:2024-03-25T18:26:43.255+0000 7f899ffff640 10 monclient: _finish_hunting 0 2024-03-25T18:26:43.256 INFO:teuthology.orchestra.run.smithi115.stderr:2024-03-25T18:26:43.255+0000 7f899ffff640 1 monclient: found mon.noname-c 2024-03-25T18:26:43.256 INFO:teuthology.orchestra.run.smithi115.stderr:2024-03-25T18:26:43.255+0000 7f899ffff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:26:43.256 INFO:teuthology.orchestra.run.smithi115.stderr:2024-03-25T18:26:43.255+0000 7f899ffff640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.115:6789/0 2024-03-25T18:26:43.256 INFO:teuthology.orchestra.run.smithi115.stderr:2024-03-25T18:26:43.255+0000 7f899ffff640 1 -- 172.21.15.115:0/3179955640 --> v1:172.21.15.115:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f89a80aa0a0 con 0x7f89a813ff20 2024-03-25T18:26:43.257 INFO:teuthology.orchestra.run.smithi115.stderr:2024-03-25T18:26:43.255+0000 7f899ffff640 10 monclient: _finish_auth 0 2024-03-25T18:26:43.257 INFO:teuthology.orchestra.run.smithi115.stderr:2024-03-25T18:26:43.255+0000 7f899ffff640 10 monclient: _check_auth_tickets 2024-03-25T18:26:43.257 INFO:teuthology.orchestra.run.smithi115.stderr:2024-03-25T18:26:43.255+0000 7f899ffff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:26:43.257 INFO:teuthology.orchestra.run.smithi115.stderr:2024-03-25T18:26:43.255+0000 7f899ffff640 1 -- 172.21.15.115:0/3179955640 <== mon.1 v1:172.21.15.115:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1586965025 0 0) 0x7f8998001690 con 0x7f89a813ff20 2024-03-25T18:26:43.257 INFO:teuthology.orchestra.run.smithi115.stderr:2024-03-25T18:26:43.255+0000 7f89af4ba640 5 monclient: authenticate success, global_id 4138 2024-03-25T18:26:43.257 INFO:teuthology.orchestra.run.smithi115.stderr:2024-03-25T18:26:43.255+0000 7f89af4ba640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T18:26:43.257 INFO:teuthology.orchestra.run.smithi115.stderr:2024-03-25T18:26:43.255+0000 7f899ffff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:26:43.257 INFO:teuthology.orchestra.run.smithi115.stderr:2024-03-25T18:26:43.255+0000 7f899ffff640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-03-25T18:26:43.257 INFO:teuthology.orchestra.run.smithi115.stderr:2024-03-25T18:26:43.255+0000 7f899ffff640 10 monclient: dump: 2024-03-25T18:26:43.257 INFO:teuthology.orchestra.run.smithi115.stderr:epoch 1 2024-03-25T18:26:43.257 INFO:teuthology.orchestra.run.smithi115.stderr:fsid c8ea72cf-9e99-4383-9b52-6c23d365fdfc 2024-03-25T18:26:43.257 INFO:teuthology.orchestra.run.smithi115.stderr:last_changed 2024-03-25T18:25:50.510519+0000 2024-03-25T18:26:43.257 INFO:teuthology.orchestra.run.smithi115.stderr:created 2024-03-25T18:25:50.510519+0000 2024-03-25T18:26:43.257 INFO:teuthology.orchestra.run.smithi115.stderr:min_mon_release 19 (squid) 2024-03-25T18:26:43.257 INFO:teuthology.orchestra.run.smithi115.stderr:election_strategy: 3 2024-03-25T18:26:43.258 INFO:teuthology.orchestra.run.smithi115.stderr:0: v1:172.21.15.1:6789/0 mon.a 2024-03-25T18:26:43.258 INFO:teuthology.orchestra.run.smithi115.stderr:1: v1:172.21.15.115:6789/0 mon.b 2024-03-25T18:26:43.258 INFO:teuthology.orchestra.run.smithi115.stderr:2: v1:172.21.15.1:6790/0 mon.c 2024-03-25T18:26:43.258 INFO:teuthology.orchestra.run.smithi115.stderr: 2024-03-25T18:26:43.258 INFO:teuthology.orchestra.run.smithi115.stderr:2024-03-25T18:26:43.255+0000 7f89af4ba640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T18:26:43.261 INFO:teuthology.orchestra.run.smithi115.stderr:2024-03-25T18:26:43.256+0000 7f899ffff640 1 -- 172.21.15.115:0/3179955640 <== mon.1 v1:172.21.15.115:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f89980017f0 con 0x7f89a813ff20 2024-03-25T18:26:43.261 INFO:teuthology.orchestra.run.smithi115.stderr:2024-03-25T18:26:43.256+0000 7f899ffff640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:26:43.261 INFO:teuthology.orchestra.run.smithi115.stderr:2024-03-25T18:26:43.256+0000 7f899ffff640 1 -- 172.21.15.115:0/3179955640 <== mon.1 v1:172.21.15.115:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1586965025 0 0) 0x7f8998001b80 con 0x7f89a813ff20 2024-03-25T18:26:43.261 INFO:teuthology.orchestra.run.smithi115.stderr:2024-03-25T18:26:43.256+0000 7f899ffff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:26:43.261 INFO:teuthology.orchestra.run.smithi115.stderr:2024-03-25T18:26:43.257+0000 7f899ffff640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-25T18:26:43.261 INFO:teuthology.orchestra.run.smithi115.stderr:2024-03-25T18:26:43.257+0000 7f899ffff640 10 monclient: dump: 2024-03-25T18:26:43.261 INFO:teuthology.orchestra.run.smithi115.stderr:epoch 1 2024-03-25T18:26:43.261 INFO:teuthology.orchestra.run.smithi115.stderr:fsid c8ea72cf-9e99-4383-9b52-6c23d365fdfc 2024-03-25T18:26:43.261 INFO:teuthology.orchestra.run.smithi115.stderr:last_changed 2024-03-25T18:25:50.510519+0000 2024-03-25T18:26:43.261 INFO:teuthology.orchestra.run.smithi115.stderr:created 2024-03-25T18:25:50.510519+0000 2024-03-25T18:26:43.262 INFO:teuthology.orchestra.run.smithi115.stderr:min_mon_release 19 (squid) 2024-03-25T18:26:43.262 INFO:teuthology.orchestra.run.smithi115.stderr:election_strategy: 3 2024-03-25T18:26:43.262 INFO:teuthology.orchestra.run.smithi115.stderr:0: v1:172.21.15.1:6789/0 mon.a 2024-03-25T18:26:43.262 INFO:teuthology.orchestra.run.smithi115.stderr:1: v1:172.21.15.115:6789/0 mon.b 2024-03-25T18:26:43.262 INFO:teuthology.orchestra.run.smithi115.stderr:2: v1:172.21.15.1:6790/0 mon.c 2024-03-25T18:26:43.262 INFO:teuthology.orchestra.run.smithi115.stderr: 2024-03-25T18:26:43.262 INFO:teuthology.orchestra.run.smithi115.stderr:2024-03-25T18:26:43.257+0000 7f89af4ba640 10 monclient: get_monmap_and_config success 2024-03-25T18:26:43.262 INFO:teuthology.orchestra.run.smithi115.stderr:2024-03-25T18:26:43.257+0000 7f89af4ba640 4 set_mon_vals no callback set 2024-03-25T18:26:43.262 INFO:teuthology.orchestra.run.smithi115.stderr:2024-03-25T18:26:43.257+0000 7f89af4ba640 10 monclient: shutdown 2024-03-25T18:26:43.262 INFO:teuthology.orchestra.run.smithi115.stderr:2024-03-25T18:26:43.257+0000 7f89af4ba640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:26:43.262 INFO:teuthology.orchestra.run.smithi115.stderr:2024-03-25T18:26:43.257+0000 7f89af4ba640 1 -- 172.21.15.115:0/3179955640 >> v1:172.21.15.115:6789/0 conn(0x7f89a813ff20 legacy=0x7f89a81426c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:26:43.262 INFO:teuthology.orchestra.run.smithi115.stderr:2024-03-25T18:26:43.257+0000 7f89af4ba640 1 -- 172.21.15.115:0/3179955640 shutdown_connections 2024-03-25T18:26:43.262 INFO:teuthology.orchestra.run.smithi115.stderr:2024-03-25T18:26:43.257+0000 7f89af4ba640 1 -- 172.21.15.115:0/3179955640 >> 172.21.15.115:0/3179955640 conn(0x7f89a8066920 msgr2=0x7f89a8066d30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:26:43.262 INFO:teuthology.orchestra.run.smithi115.stderr:2024-03-25T18:26:43.257+0000 7f89af4ba640 1 -- 172.21.15.115:0/3179955640 shutdown_connections 2024-03-25T18:26:43.263 INFO:teuthology.orchestra.run.smithi115.stderr:2024-03-25T18:26:43.257+0000 7f89af4ba640 1 -- 172.21.15.115:0/3179955640 wait complete. 2024-03-25T18:26:43.263 INFO:teuthology.orchestra.run.smithi115.stderr:2024-03-25T18:26:43.258+0000 7f89af4ba640 10 monclient: build_initial_monmap 2024-03-25T18:26:43.263 INFO:teuthology.orchestra.run.smithi115.stderr:2024-03-25T18:26:43.258+0000 7f89af4ba640 10 monclient: monmap: 2024-03-25T18:26:43.263 INFO:teuthology.orchestra.run.smithi115.stderr:epoch 0 2024-03-25T18:26:43.263 INFO:teuthology.orchestra.run.smithi115.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T18:26:43.263 INFO:teuthology.orchestra.run.smithi115.stderr:last_changed 0.000000 2024-03-25T18:26:43.263 INFO:teuthology.orchestra.run.smithi115.stderr:created 0.000000 2024-03-25T18:26:43.263 INFO:teuthology.orchestra.run.smithi115.stderr:min_mon_release 0 (unknown) 2024-03-25T18:26:43.263 INFO:teuthology.orchestra.run.smithi115.stderr:election_strategy: 1 2024-03-25T18:26:43.263 INFO:teuthology.orchestra.run.smithi115.stderr:0: v1:172.21.15.1:6789/0 mon.noname-a 2024-03-25T18:26:43.263 INFO:teuthology.orchestra.run.smithi115.stderr:1: v1:172.21.15.115:6789/0 mon.noname-b 2024-03-25T18:26:43.263 INFO:teuthology.orchestra.run.smithi115.stderr:2: v1:172.21.15.1:6790/0 mon.noname-c 2024-03-25T18:26:43.263 INFO:teuthology.orchestra.run.smithi115.stderr: 2024-03-25T18:26:43.263 INFO:teuthology.orchestra.run.smithi115.stderr:2024-03-25T18:26:43.258+0000 7f89af4ba640 1 librados: starting msgr at 2024-03-25T18:26:43.263 INFO:teuthology.orchestra.run.smithi115.stderr:2024-03-25T18:26:43.258+0000 7f89af4ba640 1 librados: starting objecter 2024-03-25T18:26:43.264 INFO:teuthology.orchestra.run.smithi115.stderr:2024-03-25T18:26:43.258+0000 7f89af4ba640 1 Processor -- start 2024-03-25T18:26:43.264 INFO:teuthology.orchestra.run.smithi115.stderr:2024-03-25T18:26:43.258+0000 7f89af4ba640 1 -- start start 2024-03-25T18:26:43.264 INFO:teuthology.orchestra.run.smithi115.stderr:2024-03-25T18:26:43.258+0000 7f89af4ba640 1 librados: setting wanted keys 2024-03-25T18:26:43.264 INFO:teuthology.orchestra.run.smithi115.stderr:2024-03-25T18:26:43.258+0000 7f89af4ba640 1 librados: calling monclient init 2024-03-25T18:26:43.264 INFO:teuthology.orchestra.run.smithi115.stderr:2024-03-25T18:26:43.258+0000 7f89af4ba640 10 monclient: init 2024-03-25T18:26:43.264 INFO:teuthology.orchestra.run.smithi115.stderr:2024-03-25T18:26:43.259+0000 7f89af4ba640 10 monclient: _reopen_session rank -1 2024-03-25T18:26:43.264 INFO:teuthology.orchestra.run.smithi115.stderr:2024-03-25T18:26:43.259+0000 7f89af4ba640 10 monclient: _add_conns ranks=[1,2,0] 2024-03-25T18:26:43.264 INFO:teuthology.orchestra.run.smithi115.stderr:2024-03-25T18:26:43.259+0000 7f89af4ba640 10 monclient(hunting): picked mon.noname-b con 0x7f89a80b4b50 addr v1:172.21.15.115:6789/0 2024-03-25T18:26:43.264 INFO:teuthology.orchestra.run.smithi115.stderr:2024-03-25T18:26:43.259+0000 7f89af4ba640 10 monclient(hunting): picked mon.noname-c con 0x7f89a813ff20 addr v1:172.21.15.1:6790/0 2024-03-25T18:26:43.264 INFO:teuthology.orchestra.run.smithi115.stderr:2024-03-25T18:26:43.259+0000 7f89af4ba640 10 monclient(hunting): picked mon.noname-a con 0x7f89a8142dd0 addr v1:172.21.15.1:6789/0 2024-03-25T18:26:43.264 INFO:teuthology.orchestra.run.smithi115.stderr:2024-03-25T18:26:43.259+0000 7f89af4ba640 1 -- --> v1:172.21.15.1:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f89a8138e40 con 0x7f89a8142dd0 2024-03-25T18:26:43.264 INFO:teuthology.orchestra.run.smithi115.stderr:2024-03-25T18:26:43.259+0000 7f89af4ba640 1 -- --> v1:172.21.15.115:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f89a8146520 con 0x7f89a80b4b50 2024-03-25T18:26:43.264 INFO:teuthology.orchestra.run.smithi115.stderr:2024-03-25T18:26:43.259+0000 7f89af4ba640 1 -- --> v1:172.21.15.1:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f89a813aec0 con 0x7f89a813ff20 2024-03-25T18:26:43.264 INFO:teuthology.orchestra.run.smithi115.stderr:2024-03-25T18:26:43.259+0000 7f89af4ba640 10 monclient(hunting): _renew_subs 2024-03-25T18:26:43.264 INFO:teuthology.orchestra.run.smithi115.stderr:2024-03-25T18:26:43.259+0000 7f89af4ba640 10 monclient(hunting): authenticate will time out at 7494.415235s 2024-03-25T18:26:43.265 INFO:teuthology.orchestra.run.smithi115.stderr:2024-03-25T18:26:43.259+0000 7f89ad256640 1 --1- >> v1:172.21.15.115:6789/0 conn(0x7f89a80b4b50 0x7f89a813a7b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.115:6789/0 says I am v1:172.21.15.115:44638/0 (socket says 172.21.15.115:44638) 2024-03-25T18:26:43.265 INFO:teuthology.orchestra.run.smithi115.stderr:2024-03-25T18:26:43.259+0000 7f89ad256640 1 -- 172.21.15.115:0/2946476143 learned_addr learned my addr 172.21.15.115:0/2946476143 (peer_addr_for_me v1:172.21.15.115:0/0) 2024-03-25T18:26:43.265 INFO:teuthology.orchestra.run.smithi115.stderr:2024-03-25T18:26:43.259+0000 7f89ada57640 1 --1- 172.21.15.115:0/2946476143 >> v1:172.21.15.1:6789/0 conn(0x7f89a8142dd0 0x7f89a81e2f90 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.1:6789/0 says I am v1:172.21.15.115:55636/0 (socket says 172.21.15.115:55636) 2024-03-25T18:26:43.265 INFO:teuthology.orchestra.run.smithi115.stderr:2024-03-25T18:26:43.260+0000 7f899dffb640 10 client.?.objecter ms_handle_connect 0x7f89a80b4b50 2024-03-25T18:26:43.265 INFO:teuthology.orchestra.run.smithi115.stderr:2024-03-25T18:26:43.260+0000 7f899dffb640 10 client.?.objecter resend_mon_ops 2024-03-25T18:26:43.265 INFO:teuthology.orchestra.run.smithi115.stderr:2024-03-25T18:26:43.260+0000 7f899dffb640 10 client.?.objecter ms_handle_connect 0x7f89a8142dd0 2024-03-25T18:26:43.265 INFO:teuthology.orchestra.run.smithi115.stderr:2024-03-25T18:26:43.260+0000 7f899dffb640 10 client.?.objecter resend_mon_ops 2024-03-25T18:26:43.265 INFO:teuthology.orchestra.run.smithi115.stderr:2024-03-25T18:26:43.260+0000 7f899dffb640 10 client.?.objecter ms_handle_connect 0x7f89a813ff20 2024-03-25T18:26:43.265 INFO:teuthology.orchestra.run.smithi115.stderr:2024-03-25T18:26:43.260+0000 7f899dffb640 10 client.?.objecter resend_mon_ops 2024-03-25T18:26:43.267 INFO:teuthology.orchestra.run.smithi115.stderr:2024-03-25T18:26:43.261+0000 7f899dffb640 1 -- 172.21.15.115:0/2946476143 <== mon.1 v1:172.21.15.115:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3460322375 0 0) 0x7f89a8146520 con 0x7f89a80b4b50 2024-03-25T18:26:43.268 INFO:teuthology.orchestra.run.smithi115.stderr:2024-03-25T18:26:43.261+0000 7f899dffb640 10 monclient(hunting): _negotiate 2024-03-25T18:26:43.268 INFO:teuthology.orchestra.run.smithi115.stderr:2024-03-25T18:26:43.261+0000 7f899dffb640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:26:43.268 INFO:teuthology.orchestra.run.smithi115.stderr:2024-03-25T18:26:43.261+0000 7f899dffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:26:43.268 INFO:teuthology.orchestra.run.smithi115.stderr:2024-03-25T18:26:43.261+0000 7f899dffb640 10 monclient(hunting): my global_id is 4141 2024-03-25T18:26:43.268 INFO:teuthology.orchestra.run.smithi115.stderr:2024-03-25T18:26:43.261+0000 7f899dffb640 1 -- 172.21.15.115:0/2946476143 --> v1:172.21.15.115:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f898c004eb0 con 0x7f89a80b4b50 2024-03-25T18:26:43.268 INFO:teuthology.orchestra.run.smithi115.stderr:2024-03-25T18:26:43.262+0000 7f899dffb640 1 -- 172.21.15.115:0/2946476143 <== mon.2 v1:172.21.15.1:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4125349086 0 0) 0x7f89a813aec0 con 0x7f89a813ff20 2024-03-25T18:26:43.268 INFO:teuthology.orchestra.run.smithi115.stderr:2024-03-25T18:26:43.262+0000 7f899dffb640 10 monclient(hunting): _negotiate 2024-03-25T18:26:43.268 INFO:teuthology.orchestra.run.smithi115.stderr:2024-03-25T18:26:43.262+0000 7f899dffb640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:26:43.268 INFO:teuthology.orchestra.run.smithi115.stderr:2024-03-25T18:26:43.262+0000 7f899dffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:26:43.268 INFO:teuthology.orchestra.run.smithi115.stderr:2024-03-25T18:26:43.262+0000 7f899dffb640 10 monclient(hunting): my global_id is 4145 2024-03-25T18:26:43.268 INFO:teuthology.orchestra.run.smithi115.stderr:2024-03-25T18:26:43.262+0000 7f899dffb640 1 -- 172.21.15.115:0/2946476143 --> v1:172.21.15.1:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f89a8146520 con 0x7f89a813ff20 2024-03-25T18:26:43.268 INFO:teuthology.orchestra.run.smithi115.stderr:2024-03-25T18:26:43.262+0000 7f899dffb640 1 -- 172.21.15.115:0/2946476143 <== mon.0 v1:172.21.15.1:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 679336224 0 0) 0x7f89a8138e40 con 0x7f89a8142dd0 2024-03-25T18:26:43.268 INFO:teuthology.orchestra.run.smithi115.stderr:2024-03-25T18:26:43.262+0000 7f899dffb640 10 monclient(hunting): _negotiate 2024-03-25T18:26:43.269 INFO:teuthology.orchestra.run.smithi115.stderr:2024-03-25T18:26:43.262+0000 7f899dffb640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:26:43.269 INFO:teuthology.orchestra.run.smithi115.stderr:2024-03-25T18:26:43.262+0000 7f899dffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:26:43.269 INFO:teuthology.orchestra.run.smithi115.stderr:2024-03-25T18:26:43.262+0000 7f899dffb640 10 monclient(hunting): my global_id is 4146 2024-03-25T18:26:43.269 INFO:teuthology.orchestra.run.smithi115.stderr:2024-03-25T18:26:43.262+0000 7f899dffb640 1 -- 172.21.15.115:0/2946476143 --> v1:172.21.15.1:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f89a813aec0 con 0x7f89a8142dd0 2024-03-25T18:26:43.269 INFO:teuthology.orchestra.run.smithi115.stderr:2024-03-25T18:26:43.266+0000 7f899dffb640 1 -- 172.21.15.115:0/2946476143 <== mon.1 v1:172.21.15.115:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2231055590 0 0) 0x7f898c004eb0 con 0x7f89a80b4b50 2024-03-25T18:26:43.269 INFO:teuthology.orchestra.run.smithi115.stderr:2024-03-25T18:26:43.266+0000 7f899dffb640 1 -- 172.21.15.115:0/2946476143 --> v1:172.21.15.115:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f89a8138e40 con 0x7f89a80b4b50 2024-03-25T18:26:43.269 INFO:teuthology.orchestra.run.smithi115.stderr:2024-03-25T18:26:43.266+0000 7f899dffb640 1 -- 172.21.15.115:0/2946476143 <== mon.1 v1:172.21.15.115:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1586965025 0 0) 0x7f89a0001c90 con 0x7f89a80b4b50 2024-03-25T18:26:43.269 INFO:teuthology.orchestra.run.smithi115.stderr:2024-03-25T18:26:43.266+0000 7f899dffb640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:26:43.269 INFO:teuthology.orchestra.run.smithi115.stderr:2024-03-25T18:26:43.266+0000 7f899dffb640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-03-25T18:26:43.269 INFO:teuthology.orchestra.run.smithi115.stderr:2024-03-25T18:26:43.266+0000 7f899dffb640 10 monclient(hunting): dump: 2024-03-25T18:26:43.269 INFO:teuthology.orchestra.run.smithi115.stderr:epoch 1 2024-03-25T18:26:43.269 INFO:teuthology.orchestra.run.smithi115.stderr:fsid c8ea72cf-9e99-4383-9b52-6c23d365fdfc 2024-03-25T18:26:43.269 INFO:teuthology.orchestra.run.smithi115.stderr:last_changed 2024-03-25T18:25:50.510519+0000 2024-03-25T18:26:43.269 INFO:teuthology.orchestra.run.smithi115.stderr:created 2024-03-25T18:25:50.510519+0000 2024-03-25T18:26:43.270 INFO:teuthology.orchestra.run.smithi115.stderr:min_mon_release 19 (squid) 2024-03-25T18:26:43.270 INFO:teuthology.orchestra.run.smithi115.stderr:election_strategy: 3 2024-03-25T18:26:43.270 INFO:teuthology.orchestra.run.smithi115.stderr:0: v1:172.21.15.1:6789/0 mon.a 2024-03-25T18:26:43.270 INFO:teuthology.orchestra.run.smithi115.stderr:1: v1:172.21.15.115:6789/0 mon.b 2024-03-25T18:26:43.270 INFO:teuthology.orchestra.run.smithi115.stderr:2: v1:172.21.15.1:6790/0 mon.c 2024-03-25T18:26:43.270 INFO:teuthology.orchestra.run.smithi115.stderr: 2024-03-25T18:26:43.270 INFO:teuthology.orchestra.run.smithi115.stderr:2024-03-25T18:26:43.266+0000 7f899dffb640 10 monclient(hunting): _finish_auth 0 2024-03-25T18:26:43.270 INFO:teuthology.orchestra.run.smithi115.stderr:2024-03-25T18:26:43.266+0000 7f899dffb640 1 -- 172.21.15.115:0/2946476143 <== mon.2 v1:172.21.15.1:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2940260263 0 0) 0x7f89a8146520 con 0x7f89a813ff20 2024-03-25T18:26:43.270 INFO:teuthology.orchestra.run.smithi115.stderr:2024-03-25T18:26:43.266+0000 7f899dffb640 1 -- 172.21.15.115:0/2946476143 --> v1:172.21.15.1:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f898c004eb0 con 0x7f89a813ff20 2024-03-25T18:26:43.270 INFO:teuthology.orchestra.run.smithi115.stderr:2024-03-25T18:26:43.266+0000 7f899dffb640 1 -- 172.21.15.115:0/2946476143 <== mon.2 v1:172.21.15.1:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1586965025 0 0) 0x7f8998001b00 con 0x7f89a813ff20 2024-03-25T18:26:43.270 INFO:teuthology.orchestra.run.smithi115.stderr:2024-03-25T18:26:43.266+0000 7f899dffb640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:26:43.270 INFO:teuthology.orchestra.run.smithi115.stderr:2024-03-25T18:26:43.266+0000 7f899dffb640 10 monclient(hunting): got monmap 1 from mon.c (according to old e1) 2024-03-25T18:26:43.270 INFO:teuthology.orchestra.run.smithi115.stderr:2024-03-25T18:26:43.266+0000 7f899dffb640 10 monclient(hunting): dump: 2024-03-25T18:26:43.270 INFO:teuthology.orchestra.run.smithi115.stderr:epoch 1 2024-03-25T18:26:43.271 INFO:teuthology.orchestra.run.smithi115.stderr:fsid c8ea72cf-9e99-4383-9b52-6c23d365fdfc 2024-03-25T18:26:43.271 INFO:teuthology.orchestra.run.smithi115.stderr:last_changed 2024-03-25T18:25:50.510519+0000 2024-03-25T18:26:43.271 INFO:teuthology.orchestra.run.smithi115.stderr:created 2024-03-25T18:25:50.510519+0000 2024-03-25T18:26:43.271 INFO:teuthology.orchestra.run.smithi115.stderr:min_mon_release 19 (squid) 2024-03-25T18:26:43.271 INFO:teuthology.orchestra.run.smithi115.stderr:election_strategy: 3 2024-03-25T18:26:43.271 INFO:teuthology.orchestra.run.smithi115.stderr:0: v1:172.21.15.1:6789/0 mon.a 2024-03-25T18:26:43.271 INFO:teuthology.orchestra.run.smithi115.stderr:1: v1:172.21.15.115:6789/0 mon.b 2024-03-25T18:26:43.271 INFO:teuthology.orchestra.run.smithi115.stderr:2: v1:172.21.15.1:6790/0 mon.c 2024-03-25T18:26:43.271 INFO:teuthology.orchestra.run.smithi115.stderr: 2024-03-25T18:26:43.273 INFO:teuthology.orchestra.run.smithi115.stderr:2024-03-25T18:26:43.269+0000 7f899dffb640 1 -- 172.21.15.115:0/2946476143 <== mon.0 v1:172.21.15.1:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3374584266 0 0) 0x7f89a813aec0 con 0x7f89a8142dd0 2024-03-25T18:26:43.273 INFO:teuthology.orchestra.run.smithi115.stderr:2024-03-25T18:26:43.269+0000 7f899dffb640 1 -- 172.21.15.115:0/2946476143 --> v1:172.21.15.1:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f89a8146520 con 0x7f89a8142dd0 2024-03-25T18:26:43.273 INFO:teuthology.orchestra.run.smithi115.stderr:2024-03-25T18:26:43.269+0000 7f899dffb640 1 -- 172.21.15.115:0/2946476143 <== mon.0 v1:172.21.15.1:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1586965025 0 0) 0x7f89a40017d0 con 0x7f89a8142dd0 2024-03-25T18:26:43.273 INFO:teuthology.orchestra.run.smithi115.stderr:2024-03-25T18:26:43.269+0000 7f899dffb640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:26:43.273 INFO:teuthology.orchestra.run.smithi115.stderr:2024-03-25T18:26:43.269+0000 7f899dffb640 10 monclient(hunting): got monmap 1 from mon.a (according to old e1) 2024-03-25T18:26:43.273 INFO:teuthology.orchestra.run.smithi115.stderr:2024-03-25T18:26:43.269+0000 7f899dffb640 10 monclient(hunting): dump: 2024-03-25T18:26:43.273 INFO:teuthology.orchestra.run.smithi115.stderr:epoch 1 2024-03-25T18:26:43.273 INFO:teuthology.orchestra.run.smithi115.stderr:fsid c8ea72cf-9e99-4383-9b52-6c23d365fdfc 2024-03-25T18:26:43.273 INFO:teuthology.orchestra.run.smithi115.stderr:last_changed 2024-03-25T18:25:50.510519+0000 2024-03-25T18:26:43.273 INFO:teuthology.orchestra.run.smithi115.stderr:created 2024-03-25T18:25:50.510519+0000 2024-03-25T18:26:43.274 INFO:teuthology.orchestra.run.smithi115.stderr:min_mon_release 19 (squid) 2024-03-25T18:26:43.274 INFO:teuthology.orchestra.run.smithi115.stderr:election_strategy: 3 2024-03-25T18:26:43.274 INFO:teuthology.orchestra.run.smithi115.stderr:0: v1:172.21.15.1:6789/0 mon.a 2024-03-25T18:26:43.274 INFO:teuthology.orchestra.run.smithi115.stderr:1: v1:172.21.15.115:6789/0 mon.b 2024-03-25T18:26:43.274 INFO:teuthology.orchestra.run.smithi115.stderr:2: v1:172.21.15.1:6790/0 mon.c 2024-03-25T18:26:43.274 INFO:teuthology.orchestra.run.smithi115.stderr: 2024-03-25T18:26:43.274 INFO:teuthology.orchestra.run.smithi115.stderr:2024-03-25T18:26:43.269+0000 7f899dffb640 1 -- 172.21.15.115:0/2946476143 <== mon.1 v1:172.21.15.115:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 3747803690 0 0) 0x7f89a8138e40 con 0x7f89a80b4b50 2024-03-25T18:26:43.274 INFO:teuthology.orchestra.run.smithi115.stderr:2024-03-25T18:26:43.269+0000 7f899dffb640 1 -- 172.21.15.115:0/2946476143 >> v1:172.21.15.1:6790/0 conn(0x7f89a813ff20 legacy=0x7f89a81dac10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:26:43.274 INFO:teuthology.orchestra.run.smithi115.stderr:2024-03-25T18:26:43.269+0000 7f899dffb640 1 -- 172.21.15.115:0/2946476143 >> v1:172.21.15.1:6789/0 conn(0x7f89a8142dd0 legacy=0x7f89a81e2f90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:26:43.274 INFO:teuthology.orchestra.run.smithi115.stderr:2024-03-25T18:26:43.269+0000 7f899dffb640 10 monclient: _finish_hunting 0 2024-03-25T18:26:43.274 INFO:teuthology.orchestra.run.smithi115.stderr:2024-03-25T18:26:43.269+0000 7f899dffb640 1 monclient: found mon.b 2024-03-25T18:26:43.274 INFO:teuthology.orchestra.run.smithi115.stderr:2024-03-25T18:26:43.269+0000 7f899dffb640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:26:43.275 INFO:teuthology.orchestra.run.smithi115.stderr:2024-03-25T18:26:43.269+0000 7f899dffb640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.115:6789/0 2024-03-25T18:26:43.275 INFO:teuthology.orchestra.run.smithi115.stderr:2024-03-25T18:26:43.269+0000 7f899dffb640 1 -- 172.21.15.115:0/2946476143 --> v1:172.21.15.115:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f89a8145830 con 0x7f89a80b4b50 2024-03-25T18:26:43.275 INFO:teuthology.orchestra.run.smithi115.stderr:2024-03-25T18:26:43.269+0000 7f899dffb640 10 monclient: _finish_auth 0 2024-03-25T18:26:43.275 INFO:teuthology.orchestra.run.smithi115.stderr:2024-03-25T18:26:43.269+0000 7f899dffb640 10 monclient: _check_auth_tickets 2024-03-25T18:26:43.275 INFO:teuthology.orchestra.run.smithi115.stderr:2024-03-25T18:26:43.269+0000 7f899dffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:26:43.275 INFO:teuthology.orchestra.run.smithi115.stderr:2024-03-25T18:26:43.269+0000 7f89af4ba640 5 monclient: authenticate success, global_id 4141 2024-03-25T18:26:43.275 INFO:teuthology.orchestra.run.smithi115.stderr:2024-03-25T18:26:43.269+0000 7f89af4ba640 10 monclient: _renew_subs 2024-03-25T18:26:43.275 INFO:teuthology.orchestra.run.smithi115.stderr:2024-03-25T18:26:43.269+0000 7f89af4ba640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.115:6789/0 2024-03-25T18:26:43.275 INFO:teuthology.orchestra.run.smithi115.stderr:2024-03-25T18:26:43.269+0000 7f89af4ba640 1 -- 172.21.15.115:0/2946476143 --> v1:172.21.15.115:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f89a8145a80 con 0x7f89a80b4b50 2024-03-25T18:26:43.275 INFO:teuthology.orchestra.run.smithi115.stderr:2024-03-25T18:26:43.269+0000 7f89af4ba640 10 client.4141.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T18:26:43.275 INFO:teuthology.orchestra.run.smithi115.stderr:2024-03-25T18:26:43.269+0000 7f89af4ba640 10 monclient: _renew_subs 2024-03-25T18:26:43.275 INFO:teuthology.orchestra.run.smithi115.stderr:2024-03-25T18:26:43.269+0000 7f89af4ba640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.115:6789/0 2024-03-25T18:26:43.275 INFO:teuthology.orchestra.run.smithi115.stderr:2024-03-25T18:26:43.269+0000 7f89af4ba640 1 -- 172.21.15.115:0/2946476143 --> v1:172.21.15.115:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f89a8141cb0 con 0x7f89a80b4b50 2024-03-25T18:26:43.275 INFO:teuthology.orchestra.run.smithi115.stderr:2024-03-25T18:26:43.269+0000 7f89af4ba640 1 librados: init done 2024-03-25T18:26:43.276 INFO:teuthology.orchestra.run.smithi115.stderr:2024-03-25T18:26:43.270+0000 7f89af4ba640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T18:26:43.276 INFO:teuthology.orchestra.run.smithi115.stderr:2024-03-25T18:26:43.270+0000 7f89af4ba640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:26:43.276 INFO:teuthology.orchestra.run.smithi115.stderr:2024-03-25T18:26:43.270+0000 7f89af4ba640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.115:6789/0 2024-03-25T18:26:43.276 INFO:teuthology.orchestra.run.smithi115.stderr:2024-03-25T18:26:43.270+0000 7f89af4ba640 1 -- 172.21.15.115:0/2946476143 --> v1:172.21.15.115:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f89a804f730 con 0x7f89a80b4b50 2024-03-25T18:26:43.276 INFO:teuthology.orchestra.run.smithi115.stderr:2024-03-25T18:26:43.273+0000 7f899dffb640 1 -- 172.21.15.115:0/2946476143 <== mon.1 v1:172.21.15.115:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f89a0004b60 con 0x7f89a80b4b50 2024-03-25T18:26:43.277 INFO:teuthology.orchestra.run.smithi115.stderr:2024-03-25T18:26:43.273+0000 7f899dffb640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:26:43.277 INFO:teuthology.orchestra.run.smithi115.stderr:2024-03-25T18:26:43.273+0000 7f899dffb640 1 -- 172.21.15.115:0/2946476143 <== mon.1 v1:172.21.15.115:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1586965025 0 0) 0x7f89a0004f70 con 0x7f89a80b4b50 2024-03-25T18:26:43.277 INFO:teuthology.orchestra.run.smithi115.stderr:2024-03-25T18:26:43.273+0000 7f899dffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:26:43.277 INFO:teuthology.orchestra.run.smithi115.stderr:2024-03-25T18:26:43.273+0000 7f899dffb640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-25T18:26:43.277 INFO:teuthology.orchestra.run.smithi115.stderr:2024-03-25T18:26:43.273+0000 7f899dffb640 10 monclient: dump: 2024-03-25T18:26:43.277 INFO:teuthology.orchestra.run.smithi115.stderr:epoch 1 2024-03-25T18:26:43.277 INFO:teuthology.orchestra.run.smithi115.stderr:fsid c8ea72cf-9e99-4383-9b52-6c23d365fdfc 2024-03-25T18:26:43.277 INFO:teuthology.orchestra.run.smithi115.stderr:last_changed 2024-03-25T18:25:50.510519+0000 2024-03-25T18:26:43.277 INFO:teuthology.orchestra.run.smithi115.stderr:created 2024-03-25T18:25:50.510519+0000 2024-03-25T18:26:43.277 INFO:teuthology.orchestra.run.smithi115.stderr:min_mon_release 19 (squid) 2024-03-25T18:26:43.277 INFO:teuthology.orchestra.run.smithi115.stderr:election_strategy: 3 2024-03-25T18:26:43.277 INFO:teuthology.orchestra.run.smithi115.stderr:0: v1:172.21.15.1:6789/0 mon.a 2024-03-25T18:26:43.277 INFO:teuthology.orchestra.run.smithi115.stderr:1: v1:172.21.15.115:6789/0 mon.b 2024-03-25T18:26:43.278 INFO:teuthology.orchestra.run.smithi115.stderr:2: v1:172.21.15.1:6790/0 mon.c 2024-03-25T18:26:43.278 INFO:teuthology.orchestra.run.smithi115.stderr: 2024-03-25T18:26:43.278 INFO:teuthology.orchestra.run.smithi115.stderr:2024-03-25T18:26:43.273+0000 7f899dffb640 1 -- 172.21.15.115:0/2946476143 <== mon.1 v1:172.21.15.115:6789/0 7 ==== mgrmap(e 1) v1 ==== 807+0+0 (unknown 3189685207 0 0) 0x7f89a0005520 con 0x7f89a80b4b50 2024-03-25T18:26:43.278 INFO:teuthology.orchestra.run.smithi115.stderr:2024-03-25T18:26:43.273+0000 7f899dffb640 1 -- 172.21.15.115:0/2946476143 <== mon.1 v1:172.21.15.115:6789/0 8 ==== osd_map(7..7 src has 1..7) v4 ==== 1300+0+0 (unknown 2977368184 0 0) 0x7f89a0006b70 con 0x7f89a80b4b50 2024-03-25T18:26:43.278 INFO:teuthology.orchestra.run.smithi115.stderr:2024-03-25T18:26:43.273+0000 7f899dffb640 10 client.4141.objecter ms_dispatch 0x7f89a8000b90 osd_map(7..7 src has 1..7) v4 2024-03-25T18:26:43.278 INFO:teuthology.orchestra.run.smithi115.stderr:2024-03-25T18:26:43.273+0000 7f899dffb640 3 client.4141.objecter handle_osd_map got epochs [7,7] > 0 2024-03-25T18:26:43.278 INFO:teuthology.orchestra.run.smithi115.stderr:2024-03-25T18:26:43.273+0000 7f899dffb640 3 client.4141.objecter handle_osd_map decoding full epoch 7 2024-03-25T18:26:43.278 INFO:teuthology.orchestra.run.smithi115.stderr:2024-03-25T18:26:43.273+0000 7f899f7fe640 4 set_mon_vals no callback set 2024-03-25T18:26:43.278 INFO:teuthology.orchestra.run.smithi115.stderr:2024-03-25T18:26:43.273+0000 7f899dffb640 20 client.4141.objecter dump_active .. 0 homeless 2024-03-25T18:26:43.366 INFO:teuthology.orchestra.run.smithi115.stderr:2024-03-25T18:26:43.365+0000 7f899dffb640 1 -- 172.21.15.115:0/2946476143 <== mon.1 v1:172.21.15.115:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+73141 (unknown 1092875540 0 2031569661) 0x7f89a00057d0 con 0x7f89a80b4b50 2024-03-25T18:26:43.367 INFO:teuthology.orchestra.run.smithi115.stderr:2024-03-25T18:26:43.365+0000 7f899dffb640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:26:43.367 INFO:teuthology.orchestra.run.smithi115.stderr:2024-03-25T18:26:43.365+0000 7f899dffb640 10 monclient: _finish_command 1 = system:0 2024-03-25T18:26:43.435 INFO:teuthology.orchestra.run.smithi115.stderr:2024-03-25T18:26:43.434+0000 7f89af4ba640 10 start_mon_command cmd=[{"prefix": "osd new", "uuid": "88cd903d-3b55-4840-9396-6e3cfe005282", "id": 5}] 2024-03-25T18:26:43.435 INFO:teuthology.orchestra.run.smithi115.stderr:2024-03-25T18:26:43.434+0000 7f89af4ba640 10 monclient: _send_command 2 [{"prefix": "osd new", "uuid": "88cd903d-3b55-4840-9396-6e3cfe005282", "id": 5}] 2024-03-25T18:26:43.436 INFO:teuthology.orchestra.run.smithi115.stderr:2024-03-25T18:26:43.434+0000 7f89af4ba640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.115:6789/0 2024-03-25T18:26:43.436 INFO:teuthology.orchestra.run.smithi115.stderr:2024-03-25T18:26:43.434+0000 7f89af4ba640 1 -- 172.21.15.115:0/2946476143 --> v1:172.21.15.115:6789/0 -- mon_command({"prefix": "osd new", "uuid": "88cd903d-3b55-4840-9396-6e3cfe005282", "id": 5} v 0) v1 -- 0x7f89a80681c0 con 0x7f89a80b4b50 2024-03-25T18:26:43.511 INFO:teuthology.orchestra.run.smithi115.stderr:2024-03-25T18:26:43.510+0000 7f899dffb640 1 -- 172.21.15.115:0/2946476143 <== mon.1 v1:172.21.15.115:6789/0 10 ==== mon_command_ack([{"prefix": "osd new", "uuid": "88cd903d-3b55-4840-9396-6e3cfe005282", "id": 5}]=0 v8) v1 ==== 112+0+2 (unknown 4144389883 0 1988412176) 0x7f89a00066a0 con 0x7f89a80b4b50 2024-03-25T18:26:43.511 INFO:teuthology.orchestra.run.smithi115.stderr:2024-03-25T18:26:43.510+0000 7f899dffb640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd new", "uuid": "88cd903d-3b55-4840-9396-6e3cfe005282", "id": 5}] 2024-03-25T18:26:43.511 INFO:teuthology.orchestra.run.smithi115.stderr:2024-03-25T18:26:43.510+0000 7f899dffb640 10 monclient: _finish_command 2 = system:0 2024-03-25T18:26:43.512 INFO:teuthology.orchestra.run.smithi115.stdout:5 2024-03-25T18:26:43.513 INFO:teuthology.orchestra.run.smithi115.stderr:2024-03-25T18:26:43.511+0000 7f89af4ba640 10 librados: watch_flush enter 2024-03-25T18:26:43.513 INFO:teuthology.orchestra.run.smithi115.stderr:2024-03-25T18:26:43.511+0000 7f89af4ba640 10 librados: watch_flush exit 2024-03-25T18:26:43.513 INFO:teuthology.orchestra.run.smithi115.stderr:2024-03-25T18:26:43.511+0000 7f89af4ba640 20 client.4141.objecter shutdown clearing up homeless session... 2024-03-25T18:26:43.513 INFO:teuthology.orchestra.run.smithi115.stderr:2024-03-25T18:26:43.511+0000 7f89af4ba640 10 client.4141.objecter successfully canceled tick 2024-03-25T18:26:43.513 INFO:teuthology.orchestra.run.smithi115.stderr:2024-03-25T18:26:43.511+0000 7f89af4ba640 10 monclient: shutdown 2024-03-25T18:26:43.513 INFO:teuthology.orchestra.run.smithi115.stderr:2024-03-25T18:26:43.511+0000 7f89af4ba640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:26:43.513 INFO:teuthology.orchestra.run.smithi115.stderr:2024-03-25T18:26:43.511+0000 7f89af4ba640 1 -- 172.21.15.115:0/2946476143 >> v1:172.21.15.115:6789/0 conn(0x7f89a80b4b50 legacy=0x7f89a813a7b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:26:43.513 INFO:teuthology.orchestra.run.smithi115.stderr:2024-03-25T18:26:43.511+0000 7f89af4ba640 1 -- 172.21.15.115:0/2946476143 shutdown_connections 2024-03-25T18:26:43.513 INFO:teuthology.orchestra.run.smithi115.stderr:2024-03-25T18:26:43.511+0000 7f89af4ba640 1 -- 172.21.15.115:0/2946476143 >> 172.21.15.115:0/2946476143 conn(0x7f89a8066920 msgr2=0x7f89a8066d00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:26:43.513 INFO:teuthology.orchestra.run.smithi115.stderr:2024-03-25T18:26:43.511+0000 7f89af4ba640 1 -- 172.21.15.115:0/2946476143 shutdown_connections 2024-03-25T18:26:43.513 INFO:teuthology.orchestra.run.smithi115.stderr:2024-03-25T18:26:43.511+0000 7f89af4ba640 1 -- 172.21.15.115:0/2946476143 wait complete. 2024-03-25T18:26:43.514 INFO:teuthology.orchestra.run.smithi115.stderr:2024-03-25T18:26:43.511+0000 7f89af4ba640 1 librados: shutdown 2024-03-25T18:26:43.522 DEBUG:teuthology.orchestra.run.smithi115:> sudo ceph --cluster ceph osd new 89db7355-bc9d-46f4-8ab0-e8d9e744e026 6 2024-03-25T18:26:43.623 INFO:teuthology.orchestra.run.smithi115.stderr:2024-03-25T18:26:43.623+0000 7ffa0cc0f640 10 monclient: get_monmap_and_config 2024-03-25T18:26:43.624 INFO:teuthology.orchestra.run.smithi115.stderr:2024-03-25T18:26:43.623+0000 7ffa0cc0f640 10 monclient: build_initial_monmap 2024-03-25T18:26:43.624 INFO:teuthology.orchestra.run.smithi115.stderr:2024-03-25T18:26:43.623+0000 7ffa0cc0f640 10 monclient: monmap: 2024-03-25T18:26:43.624 INFO:teuthology.orchestra.run.smithi115.stderr:epoch 0 2024-03-25T18:26:43.624 INFO:teuthology.orchestra.run.smithi115.stderr:fsid c8ea72cf-9e99-4383-9b52-6c23d365fdfc 2024-03-25T18:26:43.624 INFO:teuthology.orchestra.run.smithi115.stderr:last_changed 2024-03-25T18:26:43.624038+0000 2024-03-25T18:26:43.624 INFO:teuthology.orchestra.run.smithi115.stderr:created 2024-03-25T18:26:43.624038+0000 2024-03-25T18:26:43.624 INFO:teuthology.orchestra.run.smithi115.stderr:min_mon_release 0 (unknown) 2024-03-25T18:26:43.624 INFO:teuthology.orchestra.run.smithi115.stderr:election_strategy: 3 2024-03-25T18:26:43.624 INFO:teuthology.orchestra.run.smithi115.stderr:0: v1:172.21.15.1:6789/0 mon.noname-a 2024-03-25T18:26:43.624 INFO:teuthology.orchestra.run.smithi115.stderr:1: v1:172.21.15.115:6789/0 mon.noname-c 2024-03-25T18:26:43.624 INFO:teuthology.orchestra.run.smithi115.stderr:2: v1:172.21.15.1:6790/0 mon.noname-b 2024-03-25T18:26:43.624 INFO:teuthology.orchestra.run.smithi115.stderr: 2024-03-25T18:26:43.626 INFO:teuthology.orchestra.run.smithi115.stderr:2024-03-25T18:26:43.623+0000 7ffa0cc0f640 1 Processor -- start 2024-03-25T18:26:43.626 INFO:teuthology.orchestra.run.smithi115.stderr:2024-03-25T18:26:43.623+0000 7ffa0cc0f640 1 -- start start 2024-03-25T18:26:43.626 INFO:teuthology.orchestra.run.smithi115.stderr:2024-03-25T18:26:43.623+0000 7ffa0cc0f640 10 monclient: init 2024-03-25T18:26:43.626 INFO:teuthology.orchestra.run.smithi115.stderr:2024-03-25T18:26:43.624+0000 7ffa0cc0f640 10 monclient: _reopen_session rank -1 2024-03-25T18:26:43.626 INFO:teuthology.orchestra.run.smithi115.stderr:2024-03-25T18:26:43.624+0000 7ffa0cc0f640 10 monclient: _add_conns ranks=[2,1,0] 2024-03-25T18:26:43.626 INFO:teuthology.orchestra.run.smithi115.stderr:2024-03-25T18:26:43.624+0000 7ffa0cc0f640 10 monclient(hunting): picked mon.noname-b con 0x7ffa080b4b50 addr v1:172.21.15.1:6790/0 2024-03-25T18:26:43.626 INFO:teuthology.orchestra.run.smithi115.stderr:2024-03-25T18:26:43.624+0000 7ffa0cc0f640 10 monclient(hunting): picked mon.noname-c con 0x7ffa0813ff20 addr v1:172.21.15.115:6789/0 2024-03-25T18:26:43.627 INFO:teuthology.orchestra.run.smithi115.stderr:2024-03-25T18:26:43.624+0000 7ffa0cc0f640 10 monclient(hunting): picked mon.noname-a con 0x7ffa08142dd0 addr v1:172.21.15.1:6789/0 2024-03-25T18:26:43.627 INFO:teuthology.orchestra.run.smithi115.stderr:2024-03-25T18:26:43.624+0000 7ffa0cc0f640 1 -- --> v1:172.21.15.1:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ffa080ab440 con 0x7ffa08142dd0 2024-03-25T18:26:43.627 INFO:teuthology.orchestra.run.smithi115.stderr:2024-03-25T18:26:43.624+0000 7ffa0cc0f640 1 -- --> v1:172.21.15.115:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ffa080ab6b0 con 0x7ffa0813ff20 2024-03-25T18:26:43.627 INFO:teuthology.orchestra.run.smithi115.stderr:2024-03-25T18:26:43.624+0000 7ffa0cc0f640 1 -- --> v1:172.21.15.1:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ffa080a9f10 con 0x7ffa080b4b50 2024-03-25T18:26:43.627 INFO:teuthology.orchestra.run.smithi115.stderr:2024-03-25T18:26:43.624+0000 7ffa0cc0f640 10 monclient(hunting): _renew_subs 2024-03-25T18:26:43.627 INFO:teuthology.orchestra.run.smithi115.stderr:2024-03-25T18:26:43.624+0000 7ffa0cc0f640 10 monclient(hunting): authenticate will time out at 7494.780374s 2024-03-25T18:26:43.627 INFO:teuthology.orchestra.run.smithi115.stderr:2024-03-25T18:26:43.624+0000 7ffa05d9b640 1 --1- >> v1:172.21.15.115:6789/0 conn(0x7ffa0813ff20 0x7ffa081426c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.115:6789/0 says I am v1:172.21.15.115:44644/0 (socket says 172.21.15.115:44644) 2024-03-25T18:26:43.627 INFO:teuthology.orchestra.run.smithi115.stderr:2024-03-25T18:26:43.624+0000 7ffa05d9b640 1 -- 172.21.15.115:0/3868113511 learned_addr learned my addr 172.21.15.115:0/3868113511 (peer_addr_for_me v1:172.21.15.115:0/0) 2024-03-25T18:26:43.628 INFO:teuthology.orchestra.run.smithi115.stderr:2024-03-25T18:26:43.627+0000 7ffa0559a640 1 -- 172.21.15.115:0/3868113511 <== mon.1 v1:172.21.15.115:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3890659340 0 0) 0x7ffa080ab6b0 con 0x7ffa0813ff20 2024-03-25T18:26:43.628 INFO:teuthology.orchestra.run.smithi115.stderr:2024-03-25T18:26:43.627+0000 7ffa0559a640 10 monclient(hunting): _negotiate 2024-03-25T18:26:43.628 INFO:teuthology.orchestra.run.smithi115.stderr:2024-03-25T18:26:43.627+0000 7ffa0559a640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:26:43.628 INFO:teuthology.orchestra.run.smithi115.stderr:2024-03-25T18:26:43.627+0000 7ffa0559a640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:26:43.628 INFO:teuthology.orchestra.run.smithi115.stderr:2024-03-25T18:26:43.627+0000 7ffa0559a640 10 monclient(hunting): my global_id is 4144 2024-03-25T18:26:43.628 INFO:teuthology.orchestra.run.smithi115.stderr:2024-03-25T18:26:43.627+0000 7ffa0559a640 1 -- 172.21.15.115:0/3868113511 --> v1:172.21.15.115:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ff9f0003520 con 0x7ffa0813ff20 2024-03-25T18:26:43.628 INFO:teuthology.orchestra.run.smithi115.stderr:2024-03-25T18:26:43.627+0000 7ffa0559a640 1 -- 172.21.15.115:0/3868113511 <== mon.2 v1:172.21.15.1:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3683645878 0 0) 0x7ffa080a9f10 con 0x7ffa080b4b50 2024-03-25T18:26:43.628 INFO:teuthology.orchestra.run.smithi115.stderr:2024-03-25T18:26:43.627+0000 7ffa0559a640 10 monclient(hunting): _negotiate 2024-03-25T18:26:43.629 INFO:teuthology.orchestra.run.smithi115.stderr:2024-03-25T18:26:43.627+0000 7ffa0559a640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:26:43.629 INFO:teuthology.orchestra.run.smithi115.stderr:2024-03-25T18:26:43.627+0000 7ffa0559a640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:26:43.629 INFO:teuthology.orchestra.run.smithi115.stderr:2024-03-25T18:26:43.627+0000 7ffa0559a640 10 monclient(hunting): my global_id is 4148 2024-03-25T18:26:43.629 INFO:teuthology.orchestra.run.smithi115.stderr:2024-03-25T18:26:43.627+0000 7ffa0559a640 1 -- 172.21.15.115:0/3868113511 --> v1:172.21.15.1:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ffa080ab6b0 con 0x7ffa080b4b50 2024-03-25T18:26:43.629 INFO:teuthology.orchestra.run.smithi115.stderr:2024-03-25T18:26:43.628+0000 7ffa0559a640 1 -- 172.21.15.115:0/3868113511 <== mon.0 v1:172.21.15.1:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 937910068 0 0) 0x7ffa080ab440 con 0x7ffa08142dd0 2024-03-25T18:26:43.629 INFO:teuthology.orchestra.run.smithi115.stderr:2024-03-25T18:26:43.628+0000 7ffa0559a640 10 monclient(hunting): _negotiate 2024-03-25T18:26:43.629 INFO:teuthology.orchestra.run.smithi115.stderr:2024-03-25T18:26:43.628+0000 7ffa0559a640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:26:43.629 INFO:teuthology.orchestra.run.smithi115.stderr:2024-03-25T18:26:43.628+0000 7ffa0559a640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:26:43.630 INFO:teuthology.orchestra.run.smithi115.stderr:2024-03-25T18:26:43.628+0000 7ffa0559a640 10 monclient(hunting): my global_id is 4149 2024-03-25T18:26:43.630 INFO:teuthology.orchestra.run.smithi115.stderr:2024-03-25T18:26:43.628+0000 7ffa0559a640 1 -- 172.21.15.115:0/3868113511 --> v1:172.21.15.1:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ffa080a9f10 con 0x7ffa08142dd0 2024-03-25T18:26:43.631 INFO:teuthology.orchestra.run.smithi115.stderr:2024-03-25T18:26:43.630+0000 7ffa0559a640 1 -- 172.21.15.115:0/3868113511 <== mon.1 v1:172.21.15.115:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 605875612 0 0) 0x7ff9f0003520 con 0x7ffa0813ff20 2024-03-25T18:26:43.631 INFO:teuthology.orchestra.run.smithi115.stderr:2024-03-25T18:26:43.630+0000 7ffa0559a640 1 -- 172.21.15.115:0/3868113511 >> v1:172.21.15.1:6790/0 conn(0x7ffa080b4b50 legacy=0x7ffa0813f810 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:26:43.631 INFO:teuthology.orchestra.run.smithi115.stderr:2024-03-25T18:26:43.630+0000 7ffa0559a640 1 -- 172.21.15.115:0/3868113511 >> v1:172.21.15.1:6789/0 conn(0x7ffa08142dd0 legacy=0x7ffa0814df50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:26:43.631 INFO:teuthology.orchestra.run.smithi115.stderr:2024-03-25T18:26:43.630+0000 7ffa0559a640 10 monclient: _finish_hunting 0 2024-03-25T18:26:43.631 INFO:teuthology.orchestra.run.smithi115.stderr:2024-03-25T18:26:43.630+0000 7ffa0559a640 1 monclient: found mon.noname-c 2024-03-25T18:26:43.631 INFO:teuthology.orchestra.run.smithi115.stderr:2024-03-25T18:26:43.630+0000 7ffa0559a640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:26:43.632 INFO:teuthology.orchestra.run.smithi115.stderr:2024-03-25T18:26:43.630+0000 7ffa0559a640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.115:6789/0 2024-03-25T18:26:43.632 INFO:teuthology.orchestra.run.smithi115.stderr:2024-03-25T18:26:43.630+0000 7ffa0559a640 1 -- 172.21.15.115:0/3868113511 --> v1:172.21.15.115:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ffa080aa0a0 con 0x7ffa0813ff20 2024-03-25T18:26:43.632 INFO:teuthology.orchestra.run.smithi115.stderr:2024-03-25T18:26:43.630+0000 7ffa0559a640 10 monclient: _finish_auth 0 2024-03-25T18:26:43.632 INFO:teuthology.orchestra.run.smithi115.stderr:2024-03-25T18:26:43.630+0000 7ffa0559a640 10 monclient: _check_auth_tickets 2024-03-25T18:26:43.632 INFO:teuthology.orchestra.run.smithi115.stderr:2024-03-25T18:26:43.630+0000 7ffa0559a640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:26:43.632 INFO:teuthology.orchestra.run.smithi115.stderr:2024-03-25T18:26:43.630+0000 7ffa0559a640 1 -- 172.21.15.115:0/3868113511 <== mon.1 v1:172.21.15.115:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1586965025 0 0) 0x7ff9fc001690 con 0x7ffa0813ff20 2024-03-25T18:26:43.632 INFO:teuthology.orchestra.run.smithi115.stderr:2024-03-25T18:26:43.630+0000 7ffa0cc0f640 5 monclient: authenticate success, global_id 4144 2024-03-25T18:26:43.632 INFO:teuthology.orchestra.run.smithi115.stderr:2024-03-25T18:26:43.630+0000 7ffa0cc0f640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T18:26:43.632 INFO:teuthology.orchestra.run.smithi115.stderr:2024-03-25T18:26:43.630+0000 7ffa0559a640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:26:43.632 INFO:teuthology.orchestra.run.smithi115.stderr:2024-03-25T18:26:43.630+0000 7ffa0559a640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-03-25T18:26:43.632 INFO:teuthology.orchestra.run.smithi115.stderr:2024-03-25T18:26:43.630+0000 7ffa0559a640 10 monclient: dump: 2024-03-25T18:26:43.632 INFO:teuthology.orchestra.run.smithi115.stderr:epoch 1 2024-03-25T18:26:43.632 INFO:teuthology.orchestra.run.smithi115.stderr:fsid c8ea72cf-9e99-4383-9b52-6c23d365fdfc 2024-03-25T18:26:43.632 INFO:teuthology.orchestra.run.smithi115.stderr:last_changed 2024-03-25T18:25:50.510519+0000 2024-03-25T18:26:43.633 INFO:teuthology.orchestra.run.smithi115.stderr:created 2024-03-25T18:25:50.510519+0000 2024-03-25T18:26:43.633 INFO:teuthology.orchestra.run.smithi115.stderr:min_mon_release 19 (squid) 2024-03-25T18:26:43.633 INFO:teuthology.orchestra.run.smithi115.stderr:election_strategy: 3 2024-03-25T18:26:43.633 INFO:teuthology.orchestra.run.smithi115.stderr:0: v1:172.21.15.1:6789/0 mon.a 2024-03-25T18:26:43.633 INFO:teuthology.orchestra.run.smithi115.stderr:1: v1:172.21.15.115:6789/0 mon.b 2024-03-25T18:26:43.633 INFO:teuthology.orchestra.run.smithi115.stderr:2: v1:172.21.15.1:6790/0 mon.c 2024-03-25T18:26:43.633 INFO:teuthology.orchestra.run.smithi115.stderr: 2024-03-25T18:26:43.633 INFO:teuthology.orchestra.run.smithi115.stderr:2024-03-25T18:26:43.630+0000 7ffa0cc0f640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T18:26:43.636 INFO:teuthology.orchestra.run.smithi115.stderr:2024-03-25T18:26:43.632+0000 7ffa0559a640 1 -- 172.21.15.115:0/3868113511 <== mon.1 v1:172.21.15.115:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7ff9fc0017f0 con 0x7ffa0813ff20 2024-03-25T18:26:43.636 INFO:teuthology.orchestra.run.smithi115.stderr:2024-03-25T18:26:43.632+0000 7ffa0559a640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:26:43.636 INFO:teuthology.orchestra.run.smithi115.stderr:2024-03-25T18:26:43.632+0000 7ffa0559a640 1 -- 172.21.15.115:0/3868113511 <== mon.1 v1:172.21.15.115:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1586965025 0 0) 0x7ff9fc001b80 con 0x7ffa0813ff20 2024-03-25T18:26:43.636 INFO:teuthology.orchestra.run.smithi115.stderr:2024-03-25T18:26:43.632+0000 7ffa0cc0f640 10 monclient: get_monmap_and_config success 2024-03-25T18:26:43.636 INFO:teuthology.orchestra.run.smithi115.stderr:2024-03-25T18:26:43.632+0000 7ffa0cc0f640 4 set_mon_vals no callback set 2024-03-25T18:26:43.637 INFO:teuthology.orchestra.run.smithi115.stderr:2024-03-25T18:26:43.632+0000 7ffa0559a640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:26:43.637 INFO:teuthology.orchestra.run.smithi115.stderr:2024-03-25T18:26:43.632+0000 7ffa0559a640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-25T18:26:43.637 INFO:teuthology.orchestra.run.smithi115.stderr:2024-03-25T18:26:43.632+0000 7ffa0559a640 10 monclient: dump: 2024-03-25T18:26:43.637 INFO:teuthology.orchestra.run.smithi115.stderr:epoch 1 2024-03-25T18:26:43.637 INFO:teuthology.orchestra.run.smithi115.stderr:fsid c8ea72cf-9e99-4383-9b52-6c23d365fdfc 2024-03-25T18:26:43.637 INFO:teuthology.orchestra.run.smithi115.stderr:last_changed 2024-03-25T18:25:50.510519+0000 2024-03-25T18:26:43.637 INFO:teuthology.orchestra.run.smithi115.stderr:created 2024-03-25T18:25:50.510519+0000 2024-03-25T18:26:43.637 INFO:teuthology.orchestra.run.smithi115.stderr:min_mon_release 19 (squid) 2024-03-25T18:26:43.637 INFO:teuthology.orchestra.run.smithi115.stderr:election_strategy: 3 2024-03-25T18:26:43.637 INFO:teuthology.orchestra.run.smithi115.stderr:0: v1:172.21.15.1:6789/0 mon.a 2024-03-25T18:26:43.637 INFO:teuthology.orchestra.run.smithi115.stderr:1: v1:172.21.15.115:6789/0 mon.b 2024-03-25T18:26:43.637 INFO:teuthology.orchestra.run.smithi115.stderr:2: v1:172.21.15.1:6790/0 mon.c 2024-03-25T18:26:43.637 INFO:teuthology.orchestra.run.smithi115.stderr: 2024-03-25T18:26:43.637 INFO:teuthology.orchestra.run.smithi115.stderr:2024-03-25T18:26:43.632+0000 7ffa0cc0f640 10 monclient: shutdown 2024-03-25T18:26:43.638 INFO:teuthology.orchestra.run.smithi115.stderr:2024-03-25T18:26:43.632+0000 7ffa0cc0f640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:26:43.638 INFO:teuthology.orchestra.run.smithi115.stderr:2024-03-25T18:26:43.632+0000 7ffa0cc0f640 1 -- 172.21.15.115:0/3868113511 >> v1:172.21.15.115:6789/0 conn(0x7ffa0813ff20 legacy=0x7ffa081426c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:26:43.638 INFO:teuthology.orchestra.run.smithi115.stderr:2024-03-25T18:26:43.632+0000 7ffa0cc0f640 1 -- 172.21.15.115:0/3868113511 shutdown_connections 2024-03-25T18:26:43.638 INFO:teuthology.orchestra.run.smithi115.stderr:2024-03-25T18:26:43.632+0000 7ffa0cc0f640 1 -- 172.21.15.115:0/3868113511 >> 172.21.15.115:0/3868113511 conn(0x7ffa08066920 msgr2=0x7ffa08066d30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:26:43.638 INFO:teuthology.orchestra.run.smithi115.stderr:2024-03-25T18:26:43.632+0000 7ffa0cc0f640 1 -- 172.21.15.115:0/3868113511 shutdown_connections 2024-03-25T18:26:43.638 INFO:teuthology.orchestra.run.smithi115.stderr:2024-03-25T18:26:43.632+0000 7ffa0cc0f640 1 -- 172.21.15.115:0/3868113511 wait complete. 2024-03-25T18:26:43.638 INFO:teuthology.orchestra.run.smithi115.stderr:2024-03-25T18:26:43.633+0000 7ffa0cc0f640 10 monclient: build_initial_monmap 2024-03-25T18:26:43.638 INFO:teuthology.orchestra.run.smithi115.stderr:2024-03-25T18:26:43.633+0000 7ffa0cc0f640 10 monclient: monmap: 2024-03-25T18:26:43.638 INFO:teuthology.orchestra.run.smithi115.stderr:epoch 0 2024-03-25T18:26:43.638 INFO:teuthology.orchestra.run.smithi115.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T18:26:43.638 INFO:teuthology.orchestra.run.smithi115.stderr:last_changed 0.000000 2024-03-25T18:26:43.638 INFO:teuthology.orchestra.run.smithi115.stderr:created 0.000000 2024-03-25T18:26:43.638 INFO:teuthology.orchestra.run.smithi115.stderr:min_mon_release 0 (unknown) 2024-03-25T18:26:43.639 INFO:teuthology.orchestra.run.smithi115.stderr:election_strategy: 1 2024-03-25T18:26:43.639 INFO:teuthology.orchestra.run.smithi115.stderr:0: v1:172.21.15.1:6789/0 mon.noname-a 2024-03-25T18:26:43.639 INFO:teuthology.orchestra.run.smithi115.stderr:1: v1:172.21.15.115:6789/0 mon.noname-b 2024-03-25T18:26:43.639 INFO:teuthology.orchestra.run.smithi115.stderr:2: v1:172.21.15.1:6790/0 mon.noname-c 2024-03-25T18:26:43.639 INFO:teuthology.orchestra.run.smithi115.stderr: 2024-03-25T18:26:43.639 INFO:teuthology.orchestra.run.smithi115.stderr:2024-03-25T18:26:43.634+0000 7ffa0cc0f640 1 librados: starting msgr at 2024-03-25T18:26:43.639 INFO:teuthology.orchestra.run.smithi115.stderr:2024-03-25T18:26:43.634+0000 7ffa0cc0f640 1 librados: starting objecter 2024-03-25T18:26:43.639 INFO:teuthology.orchestra.run.smithi115.stderr:2024-03-25T18:26:43.634+0000 7ffa0cc0f640 1 Processor -- start 2024-03-25T18:26:43.639 INFO:teuthology.orchestra.run.smithi115.stderr:2024-03-25T18:26:43.634+0000 7ffa0cc0f640 1 -- start start 2024-03-25T18:26:43.639 INFO:teuthology.orchestra.run.smithi115.stderr:2024-03-25T18:26:43.634+0000 7ffa0cc0f640 1 librados: setting wanted keys 2024-03-25T18:26:43.639 INFO:teuthology.orchestra.run.smithi115.stderr:2024-03-25T18:26:43.634+0000 7ffa0cc0f640 1 librados: calling monclient init 2024-03-25T18:26:43.639 INFO:teuthology.orchestra.run.smithi115.stderr:2024-03-25T18:26:43.634+0000 7ffa0cc0f640 10 monclient: init 2024-03-25T18:26:43.639 INFO:teuthology.orchestra.run.smithi115.stderr:2024-03-25T18:26:43.634+0000 7ffa0cc0f640 10 monclient: _reopen_session rank -1 2024-03-25T18:26:43.639 INFO:teuthology.orchestra.run.smithi115.stderr:2024-03-25T18:26:43.634+0000 7ffa0cc0f640 10 monclient: _add_conns ranks=[0,2,1] 2024-03-25T18:26:43.639 INFO:teuthology.orchestra.run.smithi115.stderr:2024-03-25T18:26:43.634+0000 7ffa0cc0f640 10 monclient(hunting): picked mon.noname-a con 0x7ffa080b4b50 addr v1:172.21.15.1:6789/0 2024-03-25T18:26:43.640 INFO:teuthology.orchestra.run.smithi115.stderr:2024-03-25T18:26:43.634+0000 7ffa0cc0f640 10 monclient(hunting): picked mon.noname-c con 0x7ffa0813ff20 addr v1:172.21.15.1:6790/0 2024-03-25T18:26:43.640 INFO:teuthology.orchestra.run.smithi115.stderr:2024-03-25T18:26:43.634+0000 7ffa0cc0f640 10 monclient(hunting): picked mon.noname-b con 0x7ffa08142dd0 addr v1:172.21.15.115:6789/0 2024-03-25T18:26:43.640 INFO:teuthology.orchestra.run.smithi115.stderr:2024-03-25T18:26:43.634+0000 7ffa0cc0f640 1 -- --> v1:172.21.15.1:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ffa0813a190 con 0x7ffa080b4b50 2024-03-25T18:26:43.640 INFO:teuthology.orchestra.run.smithi115.stderr:2024-03-25T18:26:43.634+0000 7ffa0cc0f640 1 -- --> v1:172.21.15.115:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ffa0813ad30 con 0x7ffa08142dd0 2024-03-25T18:26:43.640 INFO:teuthology.orchestra.run.smithi115.stderr:2024-03-25T18:26:43.634+0000 7ffa0cc0f640 1 -- --> v1:172.21.15.1:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ffa081447d0 con 0x7ffa0813ff20 2024-03-25T18:26:43.640 INFO:teuthology.orchestra.run.smithi115.stderr:2024-03-25T18:26:43.634+0000 7ffa0cc0f640 10 monclient(hunting): _renew_subs 2024-03-25T18:26:43.640 INFO:teuthology.orchestra.run.smithi115.stderr:2024-03-25T18:26:43.634+0000 7ffa0cc0f640 10 monclient(hunting): authenticate will time out at 7494.790615s 2024-03-25T18:26:43.640 INFO:teuthology.orchestra.run.smithi115.stderr:2024-03-25T18:26:43.635+0000 7ffa0659c640 1 --1- >> v1:172.21.15.1:6789/0 conn(0x7ffa080b4b50 0x7ffa081397d0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.1:6789/0 says I am v1:172.21.15.115:55648/0 (socket says 172.21.15.115:55648) 2024-03-25T18:26:43.640 INFO:teuthology.orchestra.run.smithi115.stderr:2024-03-25T18:26:43.635+0000 7ffa0659c640 1 -- 172.21.15.115:0/3219722743 learned_addr learned my addr 172.21.15.115:0/3219722743 (peer_addr_for_me v1:172.21.15.115:0/0) 2024-03-25T18:26:43.640 INFO:teuthology.orchestra.run.smithi115.stderr:2024-03-25T18:26:43.635+0000 7ff9f6ffd640 10 client.?.objecter ms_handle_connect 0x7ffa080b4b50 2024-03-25T18:26:43.640 INFO:teuthology.orchestra.run.smithi115.stderr:2024-03-25T18:26:43.635+0000 7ff9f6ffd640 10 client.?.objecter resend_mon_ops 2024-03-25T18:26:43.640 INFO:teuthology.orchestra.run.smithi115.stderr:2024-03-25T18:26:43.635+0000 7ff9f6ffd640 10 client.?.objecter ms_handle_connect 0x7ffa0813ff20 2024-03-25T18:26:43.640 INFO:teuthology.orchestra.run.smithi115.stderr:2024-03-25T18:26:43.635+0000 7ff9f6ffd640 10 client.?.objecter resend_mon_ops 2024-03-25T18:26:43.640 INFO:teuthology.orchestra.run.smithi115.stderr:2024-03-25T18:26:43.635+0000 7ff9f6ffd640 10 client.?.objecter ms_handle_connect 0x7ffa08142dd0 2024-03-25T18:26:43.640 INFO:teuthology.orchestra.run.smithi115.stderr:2024-03-25T18:26:43.635+0000 7ff9f6ffd640 10 client.?.objecter resend_mon_ops 2024-03-25T18:26:43.641 INFO:teuthology.orchestra.run.smithi115.stderr:2024-03-25T18:26:43.637+0000 7ff9f6ffd640 1 -- 172.21.15.115:0/3219722743 <== mon.2 v1:172.21.15.1:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3467747689 0 0) 0x7ffa081447d0 con 0x7ffa0813ff20 2024-03-25T18:26:43.642 INFO:teuthology.orchestra.run.smithi115.stderr:2024-03-25T18:26:43.637+0000 7ff9f6ffd640 10 monclient(hunting): _negotiate 2024-03-25T18:26:43.642 INFO:teuthology.orchestra.run.smithi115.stderr:2024-03-25T18:26:43.637+0000 7ff9f6ffd640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:26:43.642 INFO:teuthology.orchestra.run.smithi115.stderr:2024-03-25T18:26:43.637+0000 7ff9f6ffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:26:43.642 INFO:teuthology.orchestra.run.smithi115.stderr:2024-03-25T18:26:43.637+0000 7ff9f6ffd640 10 monclient(hunting): my global_id is 4151 2024-03-25T18:26:43.642 INFO:teuthology.orchestra.run.smithi115.stderr:2024-03-25T18:26:43.637+0000 7ff9f6ffd640 1 -- 172.21.15.115:0/3219722743 --> v1:172.21.15.1:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ff9e8004eb0 con 0x7ffa0813ff20 2024-03-25T18:26:43.642 INFO:teuthology.orchestra.run.smithi115.stderr:2024-03-25T18:26:43.637+0000 7ff9f6ffd640 1 -- 172.21.15.115:0/3219722743 <== mon.1 v1:172.21.15.115:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1316352762 0 0) 0x7ffa0813ad30 con 0x7ffa08142dd0 2024-03-25T18:26:43.642 INFO:teuthology.orchestra.run.smithi115.stderr:2024-03-25T18:26:43.637+0000 7ff9f6ffd640 10 monclient(hunting): _negotiate 2024-03-25T18:26:43.642 INFO:teuthology.orchestra.run.smithi115.stderr:2024-03-25T18:26:43.637+0000 7ff9f6ffd640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:26:43.642 INFO:teuthology.orchestra.run.smithi115.stderr:2024-03-25T18:26:43.637+0000 7ff9f6ffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:26:43.642 INFO:teuthology.orchestra.run.smithi115.stderr:2024-03-25T18:26:43.637+0000 7ff9f6ffd640 10 monclient(hunting): my global_id is 4147 2024-03-25T18:26:43.642 INFO:teuthology.orchestra.run.smithi115.stderr:2024-03-25T18:26:43.637+0000 7ff9f6ffd640 1 -- 172.21.15.115:0/3219722743 --> v1:172.21.15.115:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ffa081447d0 con 0x7ffa08142dd0 2024-03-25T18:26:43.642 INFO:teuthology.orchestra.run.smithi115.stderr:2024-03-25T18:26:43.638+0000 7ff9f6ffd640 1 -- 172.21.15.115:0/3219722743 <== mon.0 v1:172.21.15.1:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3702757293 0 0) 0x7ffa0813a190 con 0x7ffa080b4b50 2024-03-25T18:26:43.643 INFO:teuthology.orchestra.run.smithi115.stderr:2024-03-25T18:26:43.638+0000 7ff9f6ffd640 10 monclient(hunting): _negotiate 2024-03-25T18:26:43.643 INFO:teuthology.orchestra.run.smithi115.stderr:2024-03-25T18:26:43.638+0000 7ff9f6ffd640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:26:43.643 INFO:teuthology.orchestra.run.smithi115.stderr:2024-03-25T18:26:43.638+0000 7ff9f6ffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:26:43.643 INFO:teuthology.orchestra.run.smithi115.stderr:2024-03-25T18:26:43.638+0000 7ff9f6ffd640 10 monclient(hunting): my global_id is 4152 2024-03-25T18:26:43.643 INFO:teuthology.orchestra.run.smithi115.stderr:2024-03-25T18:26:43.638+0000 7ff9f6ffd640 1 -- 172.21.15.115:0/3219722743 --> v1:172.21.15.1:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ffa0813ad30 con 0x7ffa080b4b50 2024-03-25T18:26:43.644 INFO:teuthology.orchestra.run.smithi115.stderr:2024-03-25T18:26:43.642+0000 7ff9f6ffd640 1 -- 172.21.15.115:0/3219722743 <== mon.2 v1:172.21.15.1:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3968599869 0 0) 0x7ff9e8004eb0 con 0x7ffa0813ff20 2024-03-25T18:26:43.644 INFO:teuthology.orchestra.run.smithi115.stderr:2024-03-25T18:26:43.642+0000 7ff9f6ffd640 1 -- 172.21.15.115:0/3219722743 --> v1:172.21.15.1:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7ffa0813a190 con 0x7ffa0813ff20 2024-03-25T18:26:43.644 INFO:teuthology.orchestra.run.smithi115.stderr:2024-03-25T18:26:43.642+0000 7ff9f6ffd640 1 -- 172.21.15.115:0/3219722743 <== mon.2 v1:172.21.15.1:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1586965025 0 0) 0x7ff9fc001b00 con 0x7ffa0813ff20 2024-03-25T18:26:43.644 INFO:teuthology.orchestra.run.smithi115.stderr:2024-03-25T18:26:43.642+0000 7ff9f6ffd640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:26:43.644 INFO:teuthology.orchestra.run.smithi115.stderr:2024-03-25T18:26:43.642+0000 7ff9f6ffd640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-03-25T18:26:43.644 INFO:teuthology.orchestra.run.smithi115.stderr:2024-03-25T18:26:43.642+0000 7ff9f6ffd640 10 monclient(hunting): dump: 2024-03-25T18:26:43.644 INFO:teuthology.orchestra.run.smithi115.stderr:epoch 1 2024-03-25T18:26:43.644 INFO:teuthology.orchestra.run.smithi115.stderr:fsid c8ea72cf-9e99-4383-9b52-6c23d365fdfc 2024-03-25T18:26:43.644 INFO:teuthology.orchestra.run.smithi115.stderr:last_changed 2024-03-25T18:25:50.510519+0000 2024-03-25T18:26:43.645 INFO:teuthology.orchestra.run.smithi115.stderr:created 2024-03-25T18:25:50.510519+0000 2024-03-25T18:26:43.645 INFO:teuthology.orchestra.run.smithi115.stderr:min_mon_release 19 (squid) 2024-03-25T18:26:43.645 INFO:teuthology.orchestra.run.smithi115.stderr:election_strategy: 3 2024-03-25T18:26:43.645 INFO:teuthology.orchestra.run.smithi115.stderr:0: v1:172.21.15.1:6789/0 mon.a 2024-03-25T18:26:43.645 INFO:teuthology.orchestra.run.smithi115.stderr:1: v1:172.21.15.115:6789/0 mon.b 2024-03-25T18:26:43.645 INFO:teuthology.orchestra.run.smithi115.stderr:2: v1:172.21.15.1:6790/0 mon.c 2024-03-25T18:26:43.645 INFO:teuthology.orchestra.run.smithi115.stderr: 2024-03-25T18:26:43.645 INFO:teuthology.orchestra.run.smithi115.stderr:2024-03-25T18:26:43.642+0000 7ff9f6ffd640 10 monclient(hunting): _finish_auth 0 2024-03-25T18:26:43.645 INFO:teuthology.orchestra.run.smithi115.stderr:2024-03-25T18:26:43.642+0000 7ff9f6ffd640 1 -- 172.21.15.115:0/3219722743 <== mon.1 v1:172.21.15.115:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3754997450 0 0) 0x7ffa081447d0 con 0x7ffa08142dd0 2024-03-25T18:26:43.645 INFO:teuthology.orchestra.run.smithi115.stderr:2024-03-25T18:26:43.642+0000 7ff9f6ffd640 1 -- 172.21.15.115:0/3219722743 --> v1:172.21.15.115:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7ff9e8004eb0 con 0x7ffa08142dd0 2024-03-25T18:26:43.645 INFO:teuthology.orchestra.run.smithi115.stderr:2024-03-25T18:26:43.642+0000 7ff9f6ffd640 1 -- 172.21.15.115:0/3219722743 <== mon.1 v1:172.21.15.115:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1586965025 0 0) 0x7ffa00001b50 con 0x7ffa08142dd0 2024-03-25T18:26:43.645 INFO:teuthology.orchestra.run.smithi115.stderr:2024-03-25T18:26:43.642+0000 7ff9f6ffd640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:26:43.645 INFO:teuthology.orchestra.run.smithi115.stderr:2024-03-25T18:26:43.642+0000 7ff9f6ffd640 10 monclient(hunting): got monmap 1 from mon.b (according to old e1) 2024-03-25T18:26:43.645 INFO:teuthology.orchestra.run.smithi115.stderr:2024-03-25T18:26:43.642+0000 7ff9f6ffd640 10 monclient(hunting): dump: 2024-03-25T18:26:43.646 INFO:teuthology.orchestra.run.smithi115.stderr:epoch 1 2024-03-25T18:26:43.646 INFO:teuthology.orchestra.run.smithi115.stderr:fsid c8ea72cf-9e99-4383-9b52-6c23d365fdfc 2024-03-25T18:26:43.646 INFO:teuthology.orchestra.run.smithi115.stderr:last_changed 2024-03-25T18:25:50.510519+0000 2024-03-25T18:26:43.646 INFO:teuthology.orchestra.run.smithi115.stderr:created 2024-03-25T18:25:50.510519+0000 2024-03-25T18:26:43.646 INFO:teuthology.orchestra.run.smithi115.stderr:min_mon_release 19 (squid) 2024-03-25T18:26:43.646 INFO:teuthology.orchestra.run.smithi115.stderr:election_strategy: 3 2024-03-25T18:26:43.646 INFO:teuthology.orchestra.run.smithi115.stderr:0: v1:172.21.15.1:6789/0 mon.a 2024-03-25T18:26:43.646 INFO:teuthology.orchestra.run.smithi115.stderr:1: v1:172.21.15.115:6789/0 mon.b 2024-03-25T18:26:43.646 INFO:teuthology.orchestra.run.smithi115.stderr:2: v1:172.21.15.1:6790/0 mon.c 2024-03-25T18:26:43.646 INFO:teuthology.orchestra.run.smithi115.stderr: 2024-03-25T18:26:43.648 INFO:teuthology.orchestra.run.smithi115.stderr:2024-03-25T18:26:43.644+0000 7ff9f6ffd640 1 -- 172.21.15.115:0/3219722743 <== mon.0 v1:172.21.15.1:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1127112110 0 0) 0x7ffa0813ad30 con 0x7ffa080b4b50 2024-03-25T18:26:43.648 INFO:teuthology.orchestra.run.smithi115.stderr:2024-03-25T18:26:43.644+0000 7ff9f6ffd640 1 -- 172.21.15.115:0/3219722743 --> v1:172.21.15.1:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7ffa081447d0 con 0x7ffa080b4b50 2024-03-25T18:26:43.648 INFO:teuthology.orchestra.run.smithi115.stderr:2024-03-25T18:26:43.644+0000 7ff9f6ffd640 1 -- 172.21.15.115:0/3219722743 <== mon.0 v1:172.21.15.1:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1586965025 0 0) 0x7ff9f80018b0 con 0x7ffa080b4b50 2024-03-25T18:26:43.648 INFO:teuthology.orchestra.run.smithi115.stderr:2024-03-25T18:26:43.644+0000 7ff9f6ffd640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:26:43.648 INFO:teuthology.orchestra.run.smithi115.stderr:2024-03-25T18:26:43.644+0000 7ff9f6ffd640 10 monclient(hunting): got monmap 1 from mon.a (according to old e1) 2024-03-25T18:26:43.648 INFO:teuthology.orchestra.run.smithi115.stderr:2024-03-25T18:26:43.644+0000 7ff9f6ffd640 10 monclient(hunting): dump: 2024-03-25T18:26:43.648 INFO:teuthology.orchestra.run.smithi115.stderr:epoch 1 2024-03-25T18:26:43.648 INFO:teuthology.orchestra.run.smithi115.stderr:fsid c8ea72cf-9e99-4383-9b52-6c23d365fdfc 2024-03-25T18:26:43.648 INFO:teuthology.orchestra.run.smithi115.stderr:last_changed 2024-03-25T18:25:50.510519+0000 2024-03-25T18:26:43.648 INFO:teuthology.orchestra.run.smithi115.stderr:created 2024-03-25T18:25:50.510519+0000 2024-03-25T18:26:43.649 INFO:teuthology.orchestra.run.smithi115.stderr:min_mon_release 19 (squid) 2024-03-25T18:26:43.649 INFO:teuthology.orchestra.run.smithi115.stderr:election_strategy: 3 2024-03-25T18:26:43.649 INFO:teuthology.orchestra.run.smithi115.stderr:0: v1:172.21.15.1:6789/0 mon.a 2024-03-25T18:26:43.649 INFO:teuthology.orchestra.run.smithi115.stderr:1: v1:172.21.15.115:6789/0 mon.b 2024-03-25T18:26:43.649 INFO:teuthology.orchestra.run.smithi115.stderr:2: v1:172.21.15.1:6790/0 mon.c 2024-03-25T18:26:43.649 INFO:teuthology.orchestra.run.smithi115.stderr: 2024-03-25T18:26:43.649 INFO:teuthology.orchestra.run.smithi115.stderr:2024-03-25T18:26:43.645+0000 7ff9f6ffd640 1 -- 172.21.15.115:0/3219722743 <== mon.2 v1:172.21.15.1:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 3745787407 0 0) 0x7ffa0813a190 con 0x7ffa0813ff20 2024-03-25T18:26:43.649 INFO:teuthology.orchestra.run.smithi115.stderr:2024-03-25T18:26:43.645+0000 7ff9f6ffd640 1 -- 172.21.15.115:0/3219722743 >> v1:172.21.15.115:6789/0 conn(0x7ffa08142dd0 legacy=0x7ffa081e21f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:26:43.649 INFO:teuthology.orchestra.run.smithi115.stderr:2024-03-25T18:26:43.645+0000 7ff9f6ffd640 1 -- 172.21.15.115:0/3219722743 >> v1:172.21.15.1:6789/0 conn(0x7ffa080b4b50 legacy=0x7ffa081397d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:26:43.649 INFO:teuthology.orchestra.run.smithi115.stderr:2024-03-25T18:26:43.645+0000 7ff9f6ffd640 10 monclient: _finish_hunting 0 2024-03-25T18:26:43.649 INFO:teuthology.orchestra.run.smithi115.stderr:2024-03-25T18:26:43.645+0000 7ff9f6ffd640 1 monclient: found mon.c 2024-03-25T18:26:43.649 INFO:teuthology.orchestra.run.smithi115.stderr:2024-03-25T18:26:43.645+0000 7ff9f6ffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:26:43.649 INFO:teuthology.orchestra.run.smithi115.stderr:2024-03-25T18:26:43.645+0000 7ff9f6ffd640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.1:6790/0 2024-03-25T18:26:43.649 INFO:teuthology.orchestra.run.smithi115.stderr:2024-03-25T18:26:43.645+0000 7ff9f6ffd640 1 -- 172.21.15.115:0/3219722743 --> v1:172.21.15.1:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ffa08137f00 con 0x7ffa0813ff20 2024-03-25T18:26:43.649 INFO:teuthology.orchestra.run.smithi115.stderr:2024-03-25T18:26:43.645+0000 7ff9f6ffd640 10 monclient: _finish_auth 0 2024-03-25T18:26:43.650 INFO:teuthology.orchestra.run.smithi115.stderr:2024-03-25T18:26:43.645+0000 7ff9f6ffd640 10 monclient: _check_auth_tickets 2024-03-25T18:26:43.650 INFO:teuthology.orchestra.run.smithi115.stderr:2024-03-25T18:26:43.645+0000 7ff9f6ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:26:43.650 INFO:teuthology.orchestra.run.smithi115.stderr:2024-03-25T18:26:43.645+0000 7ffa0cc0f640 5 monclient: authenticate success, global_id 4151 2024-03-25T18:26:43.650 INFO:teuthology.orchestra.run.smithi115.stderr:2024-03-25T18:26:43.645+0000 7ffa0cc0f640 10 monclient: _renew_subs 2024-03-25T18:26:43.650 INFO:teuthology.orchestra.run.smithi115.stderr:2024-03-25T18:26:43.645+0000 7ffa0cc0f640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.1:6790/0 2024-03-25T18:26:43.650 INFO:teuthology.orchestra.run.smithi115.stderr:2024-03-25T18:26:43.645+0000 7ffa0cc0f640 1 -- 172.21.15.115:0/3219722743 --> v1:172.21.15.1:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ffa08138150 con 0x7ffa0813ff20 2024-03-25T18:26:43.650 INFO:teuthology.orchestra.run.smithi115.stderr:2024-03-25T18:26:43.645+0000 7ffa0cc0f640 10 client.4151.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T18:26:43.650 INFO:teuthology.orchestra.run.smithi115.stderr:2024-03-25T18:26:43.645+0000 7ffa0cc0f640 10 monclient: _renew_subs 2024-03-25T18:26:43.650 INFO:teuthology.orchestra.run.smithi115.stderr:2024-03-25T18:26:43.645+0000 7ffa0cc0f640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.1:6790/0 2024-03-25T18:26:43.650 INFO:teuthology.orchestra.run.smithi115.stderr:2024-03-25T18:26:43.645+0000 7ffa0cc0f640 1 -- 172.21.15.115:0/3219722743 --> v1:172.21.15.1:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7ffa08136670 con 0x7ffa0813ff20 2024-03-25T18:26:43.650 INFO:teuthology.orchestra.run.smithi115.stderr:2024-03-25T18:26:43.645+0000 7ffa0cc0f640 1 librados: init done 2024-03-25T18:26:43.650 INFO:teuthology.orchestra.run.smithi115.stderr:2024-03-25T18:26:43.646+0000 7ffa0cc0f640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T18:26:43.650 INFO:teuthology.orchestra.run.smithi115.stderr:2024-03-25T18:26:43.646+0000 7ffa0cc0f640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:26:43.650 INFO:teuthology.orchestra.run.smithi115.stderr:2024-03-25T18:26:43.646+0000 7ffa0cc0f640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.1:6790/0 2024-03-25T18:26:43.651 INFO:teuthology.orchestra.run.smithi115.stderr:2024-03-25T18:26:43.646+0000 7ffa0cc0f640 1 -- 172.21.15.115:0/3219722743 --> v1:172.21.15.1:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ffa08139ee0 con 0x7ffa0813ff20 2024-03-25T18:26:43.651 INFO:teuthology.orchestra.run.smithi115.stderr:2024-03-25T18:26:43.649+0000 7ff9f6ffd640 1 -- 172.21.15.115:0/3219722743 <== mon.2 v1:172.21.15.1:6790/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7ff9fc004a90 con 0x7ffa0813ff20 2024-03-25T18:26:43.651 INFO:teuthology.orchestra.run.smithi115.stderr:2024-03-25T18:26:43.649+0000 7ff9f6ffd640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:26:43.651 INFO:teuthology.orchestra.run.smithi115.stderr:2024-03-25T18:26:43.649+0000 7ff9f6ffd640 1 -- 172.21.15.115:0/3219722743 <== mon.2 v1:172.21.15.1:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1586965025 0 0) 0x7ff9fc001250 con 0x7ffa0813ff20 2024-03-25T18:26:43.652 INFO:teuthology.orchestra.run.smithi115.stderr:2024-03-25T18:26:43.649+0000 7ff9f6ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:26:43.652 INFO:teuthology.orchestra.run.smithi115.stderr:2024-03-25T18:26:43.649+0000 7ff9f6ffd640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-25T18:26:43.652 INFO:teuthology.orchestra.run.smithi115.stderr:2024-03-25T18:26:43.649+0000 7ff9f6ffd640 10 monclient: dump: 2024-03-25T18:26:43.652 INFO:teuthology.orchestra.run.smithi115.stderr:epoch 1 2024-03-25T18:26:43.652 INFO:teuthology.orchestra.run.smithi115.stderr:fsid c8ea72cf-9e99-4383-9b52-6c23d365fdfc 2024-03-25T18:26:43.652 INFO:teuthology.orchestra.run.smithi115.stderr:last_changed 2024-03-25T18:25:50.510519+0000 2024-03-25T18:26:43.652 INFO:teuthology.orchestra.run.smithi115.stderr:created 2024-03-25T18:25:50.510519+0000 2024-03-25T18:26:43.652 INFO:teuthology.orchestra.run.smithi115.stderr:min_mon_release 19 (squid) 2024-03-25T18:26:43.652 INFO:teuthology.orchestra.run.smithi115.stderr:election_strategy: 3 2024-03-25T18:26:43.652 INFO:teuthology.orchestra.run.smithi115.stderr:0: v1:172.21.15.1:6789/0 mon.a 2024-03-25T18:26:43.652 INFO:teuthology.orchestra.run.smithi115.stderr:1: v1:172.21.15.115:6789/0 mon.b 2024-03-25T18:26:43.652 INFO:teuthology.orchestra.run.smithi115.stderr:2: v1:172.21.15.1:6790/0 mon.c 2024-03-25T18:26:43.652 INFO:teuthology.orchestra.run.smithi115.stderr: 2024-03-25T18:26:43.652 INFO:teuthology.orchestra.run.smithi115.stderr:2024-03-25T18:26:43.649+0000 7ffa04d99640 4 set_mon_vals no callback set 2024-03-25T18:26:43.652 INFO:teuthology.orchestra.run.smithi115.stderr:2024-03-25T18:26:43.649+0000 7ff9f6ffd640 1 -- 172.21.15.115:0/3219722743 <== mon.2 v1:172.21.15.1:6790/0 7 ==== mgrmap(e 1) v1 ==== 807+0+0 (unknown 3189685207 0 0) 0x7ff9fc005170 con 0x7ffa0813ff20 2024-03-25T18:26:43.653 INFO:teuthology.orchestra.run.smithi115.stderr:2024-03-25T18:26:43.649+0000 7ff9f6ffd640 1 -- 172.21.15.115:0/3219722743 <== mon.2 v1:172.21.15.1:6790/0 8 ==== osd_map(8..8 src has 1..8) v4 ==== 1415+0+0 (unknown 1356527986 0 0) 0x7ff9fc006900 con 0x7ffa0813ff20 2024-03-25T18:26:43.653 INFO:teuthology.orchestra.run.smithi115.stderr:2024-03-25T18:26:43.649+0000 7ff9f6ffd640 10 client.4151.objecter ms_dispatch 0x7ffa08000b90 osd_map(8..8 src has 1..8) v4 2024-03-25T18:26:43.653 INFO:teuthology.orchestra.run.smithi115.stderr:2024-03-25T18:26:43.649+0000 7ff9f6ffd640 3 client.4151.objecter handle_osd_map got epochs [8,8] > 0 2024-03-25T18:26:43.653 INFO:teuthology.orchestra.run.smithi115.stderr:2024-03-25T18:26:43.649+0000 7ff9f6ffd640 3 client.4151.objecter handle_osd_map decoding full epoch 8 2024-03-25T18:26:43.653 INFO:teuthology.orchestra.run.smithi115.stderr:2024-03-25T18:26:43.649+0000 7ff9f6ffd640 20 client.4151.objecter dump_active .. 0 homeless 2024-03-25T18:26:43.751 INFO:teuthology.orchestra.run.smithi115.stderr:2024-03-25T18:26:43.750+0000 7ff9f6ffd640 1 -- 172.21.15.115:0/3219722743 <== mon.2 v1:172.21.15.1:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+73141 (unknown 1092875540 0 2031569661) 0x7ff9fc006420 con 0x7ffa0813ff20 2024-03-25T18:26:43.751 INFO:teuthology.orchestra.run.smithi115.stderr:2024-03-25T18:26:43.750+0000 7ff9f6ffd640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:26:43.751 INFO:teuthology.orchestra.run.smithi115.stderr:2024-03-25T18:26:43.750+0000 7ff9f6ffd640 10 monclient: _finish_command 1 = system:0 2024-03-25T18:26:43.819 INFO:tasks.ceph.mgr.x.smithi115.stderr:2024-03-25T18:26:43.819+0000 7f99dd247040 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2024-03-25T18:26:43.835 INFO:teuthology.orchestra.run.smithi115.stderr:2024-03-25T18:26:43.834+0000 7ffa0cc0f640 10 start_mon_command cmd=[{"prefix": "osd new", "uuid": "89db7355-bc9d-46f4-8ab0-e8d9e744e026", "id": 6}] 2024-03-25T18:26:43.835 INFO:teuthology.orchestra.run.smithi115.stderr:2024-03-25T18:26:43.834+0000 7ffa0cc0f640 10 monclient: _send_command 2 [{"prefix": "osd new", "uuid": "89db7355-bc9d-46f4-8ab0-e8d9e744e026", "id": 6}] 2024-03-25T18:26:43.835 INFO:teuthology.orchestra.run.smithi115.stderr:2024-03-25T18:26:43.834+0000 7ffa0cc0f640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.1:6790/0 2024-03-25T18:26:43.835 INFO:teuthology.orchestra.run.smithi115.stderr:2024-03-25T18:26:43.834+0000 7ffa0cc0f640 1 -- 172.21.15.115:0/3219722743 --> v1:172.21.15.1:6790/0 -- mon_command({"prefix": "osd new", "uuid": "89db7355-bc9d-46f4-8ab0-e8d9e744e026", "id": 6} v 0) v1 -- 0x7ffa081445d0 con 0x7ffa0813ff20 2024-03-25T18:26:43.911 INFO:teuthology.orchestra.run.smithi115.stderr:2024-03-25T18:26:43.910+0000 7ff9f6ffd640 1 -- 172.21.15.115:0/3219722743 <== mon.2 v1:172.21.15.1:6790/0 10 ==== mon_command_ack([{"prefix": "osd new", "uuid": "89db7355-bc9d-46f4-8ab0-e8d9e744e026", "id": 6}]=0 v9) v1 ==== 112+0+2 (unknown 781755617 0 1113811849) 0x7ff9fc006730 con 0x7ffa0813ff20 2024-03-25T18:26:43.911 INFO:teuthology.orchestra.run.smithi115.stderr:2024-03-25T18:26:43.910+0000 7ff9f6ffd640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd new", "uuid": "89db7355-bc9d-46f4-8ab0-e8d9e744e026", "id": 6}] 2024-03-25T18:26:43.911 INFO:teuthology.orchestra.run.smithi115.stderr:2024-03-25T18:26:43.910+0000 7ff9f6ffd640 10 monclient: _finish_command 2 = system:0 2024-03-25T18:26:43.911 INFO:teuthology.orchestra.run.smithi115.stdout:6 2024-03-25T18:26:43.912 INFO:teuthology.orchestra.run.smithi115.stderr:2024-03-25T18:26:43.910+0000 7ffa0cc0f640 10 librados: watch_flush enter 2024-03-25T18:26:43.912 INFO:teuthology.orchestra.run.smithi115.stderr:2024-03-25T18:26:43.910+0000 7ffa0cc0f640 10 librados: watch_flush exit 2024-03-25T18:26:43.912 INFO:teuthology.orchestra.run.smithi115.stderr:2024-03-25T18:26:43.910+0000 7ffa0cc0f640 20 client.4151.objecter shutdown clearing up homeless session... 2024-03-25T18:26:43.913 INFO:teuthology.orchestra.run.smithi115.stderr:2024-03-25T18:26:43.910+0000 7ffa0cc0f640 10 client.4151.objecter successfully canceled tick 2024-03-25T18:26:43.913 INFO:teuthology.orchestra.run.smithi115.stderr:2024-03-25T18:26:43.911+0000 7ffa0cc0f640 10 monclient: shutdown 2024-03-25T18:26:43.913 INFO:teuthology.orchestra.run.smithi115.stderr:2024-03-25T18:26:43.911+0000 7ffa0cc0f640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:26:43.913 INFO:teuthology.orchestra.run.smithi115.stderr:2024-03-25T18:26:43.911+0000 7ffa0cc0f640 1 -- 172.21.15.115:0/3219722743 >> v1:172.21.15.1:6790/0 conn(0x7ffa0813ff20 legacy=0x7ffa0813a620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:26:43.913 INFO:teuthology.orchestra.run.smithi115.stderr:2024-03-25T18:26:43.911+0000 7ffa0cc0f640 1 -- 172.21.15.115:0/3219722743 shutdown_connections 2024-03-25T18:26:43.913 INFO:teuthology.orchestra.run.smithi115.stderr:2024-03-25T18:26:43.911+0000 7ffa0cc0f640 1 -- 172.21.15.115:0/3219722743 >> 172.21.15.115:0/3219722743 conn(0x7ffa08066920 msgr2=0x7ffa08066d00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:26:43.913 INFO:teuthology.orchestra.run.smithi115.stderr:2024-03-25T18:26:43.911+0000 7ffa0cc0f640 1 -- 172.21.15.115:0/3219722743 shutdown_connections 2024-03-25T18:26:43.913 INFO:teuthology.orchestra.run.smithi115.stderr:2024-03-25T18:26:43.911+0000 7ffa0cc0f640 1 -- 172.21.15.115:0/3219722743 wait complete. 2024-03-25T18:26:43.913 INFO:teuthology.orchestra.run.smithi115.stderr:2024-03-25T18:26:43.911+0000 7ffa0cc0f640 1 librados: shutdown 2024-03-25T18:26:43.922 DEBUG:teuthology.orchestra.run.smithi115:> sudo ceph --cluster ceph osd new cac47bee-0914-4f08-896f-348838882866 7 2024-03-25T18:26:44.020 INFO:teuthology.orchestra.run.smithi115.stderr:2024-03-25T18:26:44.019+0000 7ff460cdd640 10 monclient: get_monmap_and_config 2024-03-25T18:26:44.020 INFO:teuthology.orchestra.run.smithi115.stderr:2024-03-25T18:26:44.019+0000 7ff460cdd640 10 monclient: build_initial_monmap 2024-03-25T18:26:44.020 INFO:teuthology.orchestra.run.smithi115.stderr:2024-03-25T18:26:44.019+0000 7ff460cdd640 10 monclient: monmap: 2024-03-25T18:26:44.020 INFO:teuthology.orchestra.run.smithi115.stderr:epoch 0 2024-03-25T18:26:44.021 INFO:teuthology.orchestra.run.smithi115.stderr:fsid c8ea72cf-9e99-4383-9b52-6c23d365fdfc 2024-03-25T18:26:44.021 INFO:teuthology.orchestra.run.smithi115.stderr:last_changed 2024-03-25T18:26:44.020639+0000 2024-03-25T18:26:44.021 INFO:teuthology.orchestra.run.smithi115.stderr:created 2024-03-25T18:26:44.020639+0000 2024-03-25T18:26:44.021 INFO:teuthology.orchestra.run.smithi115.stderr:min_mon_release 0 (unknown) 2024-03-25T18:26:44.021 INFO:teuthology.orchestra.run.smithi115.stderr:election_strategy: 3 2024-03-25T18:26:44.021 INFO:teuthology.orchestra.run.smithi115.stderr:0: v1:172.21.15.1:6789/0 mon.noname-a 2024-03-25T18:26:44.021 INFO:teuthology.orchestra.run.smithi115.stderr:1: v1:172.21.15.115:6789/0 mon.noname-c 2024-03-25T18:26:44.021 INFO:teuthology.orchestra.run.smithi115.stderr:2: v1:172.21.15.1:6790/0 mon.noname-b 2024-03-25T18:26:44.021 INFO:teuthology.orchestra.run.smithi115.stderr: 2024-03-25T18:26:44.022 INFO:teuthology.orchestra.run.smithi115.stderr:2024-03-25T18:26:44.020+0000 7ff460cdd640 1 Processor -- start 2024-03-25T18:26:44.023 INFO:teuthology.orchestra.run.smithi115.stderr:2024-03-25T18:26:44.020+0000 7ff460cdd640 1 -- start start 2024-03-25T18:26:44.023 INFO:teuthology.orchestra.run.smithi115.stderr:2024-03-25T18:26:44.020+0000 7ff460cdd640 10 monclient: init 2024-03-25T18:26:44.023 INFO:teuthology.orchestra.run.smithi115.stderr:2024-03-25T18:26:44.021+0000 7ff460cdd640 10 monclient: _reopen_session rank -1 2024-03-25T18:26:44.023 INFO:teuthology.orchestra.run.smithi115.stderr:2024-03-25T18:26:44.021+0000 7ff460cdd640 10 monclient: _add_conns ranks=[1,2,0] 2024-03-25T18:26:44.023 INFO:teuthology.orchestra.run.smithi115.stderr:2024-03-25T18:26:44.021+0000 7ff460cdd640 10 monclient(hunting): picked mon.noname-c con 0x7ff45c0b4f00 addr v1:172.21.15.115:6789/0 2024-03-25T18:26:44.023 INFO:teuthology.orchestra.run.smithi115.stderr:2024-03-25T18:26:44.021+0000 7ff460cdd640 10 monclient(hunting): picked mon.noname-b con 0x7ff45c0b4510 addr v1:172.21.15.1:6790/0 2024-03-25T18:26:44.023 INFO:teuthology.orchestra.run.smithi115.stderr:2024-03-25T18:26:44.021+0000 7ff460cdd640 10 monclient(hunting): picked mon.noname-a con 0x7ff45c142ad0 addr v1:172.21.15.1:6789/0 2024-03-25T18:26:44.023 INFO:teuthology.orchestra.run.smithi115.stderr:2024-03-25T18:26:44.021+0000 7ff460cdd640 1 -- --> v1:172.21.15.1:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff45c0a9d90 con 0x7ff45c142ad0 2024-03-25T18:26:44.023 INFO:teuthology.orchestra.run.smithi115.stderr:2024-03-25T18:26:44.021+0000 7ff460cdd640 1 -- --> v1:172.21.15.115:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff45c0aa2b0 con 0x7ff45c0b4f00 2024-03-25T18:26:44.023 INFO:teuthology.orchestra.run.smithi115.stderr:2024-03-25T18:26:44.021+0000 7ff460cdd640 1 -- --> v1:172.21.15.1:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff45c13fc60 con 0x7ff45c0b4510 2024-03-25T18:26:44.023 INFO:teuthology.orchestra.run.smithi115.stderr:2024-03-25T18:26:44.021+0000 7ff460cdd640 10 monclient(hunting): _renew_subs 2024-03-25T18:26:44.023 INFO:teuthology.orchestra.run.smithi115.stderr:2024-03-25T18:26:44.021+0000 7ff460cdd640 10 monclient(hunting): authenticate will time out at 7495.177018s 2024-03-25T18:26:44.023 INFO:teuthology.orchestra.run.smithi115.stderr:2024-03-25T18:26:44.021+0000 7ff45a59c640 1 --1- >> v1:172.21.15.115:6789/0 conn(0x7ff45c0b4f00 0x7ff45c1423c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.115:6789/0 says I am v1:172.21.15.115:44666/0 (socket says 172.21.15.115:44666) 2024-03-25T18:26:44.024 INFO:teuthology.orchestra.run.smithi115.stderr:2024-03-25T18:26:44.021+0000 7ff45a59c640 1 -- 172.21.15.115:0/273510519 learned_addr learned my addr 172.21.15.115:0/273510519 (peer_addr_for_me v1:172.21.15.115:0/0) 2024-03-25T18:26:44.024 INFO:teuthology.orchestra.run.smithi115.stderr:2024-03-25T18:26:44.021+0000 7ff45ad9d640 1 --1- >> v1:172.21.15.1:6789/0 conn(0x7ff45c142ad0 0x7ff45c14dc80 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.1:6789/0 says I am v1:172.21.15.115:55664/0 (socket says 172.21.15.115:55664) 2024-03-25T18:26:44.024 INFO:teuthology.orchestra.run.smithi115.stderr:2024-03-25T18:26:44.023+0000 7ff459d9b640 1 -- 172.21.15.115:0/273510519 <== mon.1 v1:172.21.15.115:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3456150721 0 0) 0x7ff45c0aa2b0 con 0x7ff45c0b4f00 2024-03-25T18:26:44.024 INFO:teuthology.orchestra.run.smithi115.stderr:2024-03-25T18:26:44.023+0000 7ff459d9b640 10 monclient(hunting): _negotiate 2024-03-25T18:26:44.024 INFO:teuthology.orchestra.run.smithi115.stderr:2024-03-25T18:26:44.023+0000 7ff459d9b640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:26:44.024 INFO:teuthology.orchestra.run.smithi115.stderr:2024-03-25T18:26:44.023+0000 7ff459d9b640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:26:44.025 INFO:teuthology.orchestra.run.smithi115.stderr:2024-03-25T18:26:44.023+0000 7ff459d9b640 10 monclient(hunting): my global_id is 4150 2024-03-25T18:26:44.025 INFO:teuthology.orchestra.run.smithi115.stderr:2024-03-25T18:26:44.023+0000 7ff459d9b640 1 -- 172.21.15.115:0/273510519 --> v1:172.21.15.115:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ff448003520 con 0x7ff45c0b4f00 2024-03-25T18:26:44.025 INFO:teuthology.orchestra.run.smithi115.stderr:2024-03-25T18:26:44.024+0000 7ff459d9b640 1 -- 172.21.15.115:0/273510519 <== mon.2 v1:172.21.15.1:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3505794312 0 0) 0x7ff45c13fc60 con 0x7ff45c0b4510 2024-03-25T18:26:44.026 INFO:teuthology.orchestra.run.smithi115.stderr:2024-03-25T18:26:44.024+0000 7ff459d9b640 10 monclient(hunting): _negotiate 2024-03-25T18:26:44.026 INFO:teuthology.orchestra.run.smithi115.stderr:2024-03-25T18:26:44.024+0000 7ff459d9b640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:26:44.026 INFO:teuthology.orchestra.run.smithi115.stderr:2024-03-25T18:26:44.024+0000 7ff459d9b640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:26:44.026 INFO:teuthology.orchestra.run.smithi115.stderr:2024-03-25T18:26:44.024+0000 7ff459d9b640 10 monclient(hunting): my global_id is 4154 2024-03-25T18:26:44.026 INFO:teuthology.orchestra.run.smithi115.stderr:2024-03-25T18:26:44.024+0000 7ff459d9b640 1 -- 172.21.15.115:0/273510519 --> v1:172.21.15.1:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ff45c0aa2b0 con 0x7ff45c0b4510 2024-03-25T18:26:44.026 INFO:teuthology.orchestra.run.smithi115.stderr:2024-03-25T18:26:44.024+0000 7ff459d9b640 1 -- 172.21.15.115:0/273510519 <== mon.0 v1:172.21.15.1:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4089151301 0 0) 0x7ff45c0a9d90 con 0x7ff45c142ad0 2024-03-25T18:26:44.026 INFO:teuthology.orchestra.run.smithi115.stderr:2024-03-25T18:26:44.024+0000 7ff459d9b640 10 monclient(hunting): _negotiate 2024-03-25T18:26:44.026 INFO:teuthology.orchestra.run.smithi115.stderr:2024-03-25T18:26:44.024+0000 7ff459d9b640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:26:44.026 INFO:teuthology.orchestra.run.smithi115.stderr:2024-03-25T18:26:44.024+0000 7ff459d9b640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:26:44.026 INFO:teuthology.orchestra.run.smithi115.stderr:2024-03-25T18:26:44.024+0000 7ff459d9b640 10 monclient(hunting): my global_id is 4155 2024-03-25T18:26:44.026 INFO:teuthology.orchestra.run.smithi115.stderr:2024-03-25T18:26:44.024+0000 7ff459d9b640 1 -- 172.21.15.115:0/273510519 --> v1:172.21.15.1:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ff45c13fc60 con 0x7ff45c142ad0 2024-03-25T18:26:44.028 INFO:teuthology.orchestra.run.smithi115.stderr:2024-03-25T18:26:44.027+0000 7ff459d9b640 1 -- 172.21.15.115:0/273510519 <== mon.1 v1:172.21.15.115:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3536969726 0 0) 0x7ff448003520 con 0x7ff45c0b4f00 2024-03-25T18:26:44.028 INFO:teuthology.orchestra.run.smithi115.stderr:2024-03-25T18:26:44.027+0000 7ff459d9b640 1 -- 172.21.15.115:0/273510519 >> v1:172.21.15.1:6790/0 conn(0x7ff45c0b4510 legacy=0x7ff45c13f550 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:26:44.028 INFO:teuthology.orchestra.run.smithi115.stderr:2024-03-25T18:26:44.027+0000 7ff459d9b640 1 -- 172.21.15.115:0/273510519 >> v1:172.21.15.1:6789/0 conn(0x7ff45c142ad0 legacy=0x7ff45c14dc80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:26:44.028 INFO:teuthology.orchestra.run.smithi115.stderr:2024-03-25T18:26:44.027+0000 7ff459d9b640 10 monclient: _finish_hunting 0 2024-03-25T18:26:44.028 INFO:teuthology.orchestra.run.smithi115.stderr:2024-03-25T18:26:44.027+0000 7ff459d9b640 1 monclient: found mon.noname-c 2024-03-25T18:26:44.028 INFO:teuthology.orchestra.run.smithi115.stderr:2024-03-25T18:26:44.027+0000 7ff459d9b640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:26:44.028 INFO:teuthology.orchestra.run.smithi115.stderr:2024-03-25T18:26:44.027+0000 7ff459d9b640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.115:6789/0 2024-03-25T18:26:44.028 INFO:teuthology.orchestra.run.smithi115.stderr:2024-03-25T18:26:44.027+0000 7ff459d9b640 1 -- 172.21.15.115:0/273510519 --> v1:172.21.15.115:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff45c13fdf0 con 0x7ff45c0b4f00 2024-03-25T18:26:44.028 INFO:teuthology.orchestra.run.smithi115.stderr:2024-03-25T18:26:44.027+0000 7ff459d9b640 10 monclient: _finish_auth 0 2024-03-25T18:26:44.028 INFO:teuthology.orchestra.run.smithi115.stderr:2024-03-25T18:26:44.027+0000 7ff459d9b640 10 monclient: _check_auth_tickets 2024-03-25T18:26:44.028 INFO:teuthology.orchestra.run.smithi115.stderr:2024-03-25T18:26:44.027+0000 7ff459d9b640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:26:44.029 INFO:teuthology.orchestra.run.smithi115.stderr:2024-03-25T18:26:44.027+0000 7ff459d9b640 1 -- 172.21.15.115:0/273510519 <== mon.1 v1:172.21.15.115:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1586965025 0 0) 0x7ff44c001690 con 0x7ff45c0b4f00 2024-03-25T18:26:44.029 INFO:teuthology.orchestra.run.smithi115.stderr:2024-03-25T18:26:44.027+0000 7ff459d9b640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:26:44.029 INFO:teuthology.orchestra.run.smithi115.stderr:2024-03-25T18:26:44.027+0000 7ff459d9b640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-03-25T18:26:44.029 INFO:teuthology.orchestra.run.smithi115.stderr:2024-03-25T18:26:44.027+0000 7ff459d9b640 10 monclient: dump: 2024-03-25T18:26:44.029 INFO:teuthology.orchestra.run.smithi115.stderr:epoch 1 2024-03-25T18:26:44.029 INFO:teuthology.orchestra.run.smithi115.stderr:fsid c8ea72cf-9e99-4383-9b52-6c23d365fdfc 2024-03-25T18:26:44.029 INFO:teuthology.orchestra.run.smithi115.stderr:last_changed 2024-03-25T18:25:50.510519+0000 2024-03-25T18:26:44.029 INFO:teuthology.orchestra.run.smithi115.stderr:created 2024-03-25T18:25:50.510519+0000 2024-03-25T18:26:44.029 INFO:teuthology.orchestra.run.smithi115.stderr:min_mon_release 19 (squid) 2024-03-25T18:26:44.029 INFO:teuthology.orchestra.run.smithi115.stderr:election_strategy: 3 2024-03-25T18:26:44.029 INFO:teuthology.orchestra.run.smithi115.stderr:0: v1:172.21.15.1:6789/0 mon.a 2024-03-25T18:26:44.029 INFO:teuthology.orchestra.run.smithi115.stderr:1: v1:172.21.15.115:6789/0 mon.b 2024-03-25T18:26:44.029 INFO:teuthology.orchestra.run.smithi115.stderr:2: v1:172.21.15.1:6790/0 mon.c 2024-03-25T18:26:44.029 INFO:teuthology.orchestra.run.smithi115.stderr: 2024-03-25T18:26:44.029 INFO:teuthology.orchestra.run.smithi115.stderr:2024-03-25T18:26:44.027+0000 7ff460cdd640 5 monclient: authenticate success, global_id 4150 2024-03-25T18:26:44.029 INFO:teuthology.orchestra.run.smithi115.stderr:2024-03-25T18:26:44.027+0000 7ff460cdd640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T18:26:44.031 INFO:teuthology.orchestra.run.smithi115.stderr:2024-03-25T18:26:44.028+0000 7ff459d9b640 1 -- 172.21.15.115:0/273510519 <== mon.1 v1:172.21.15.115:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7ff44c0017f0 con 0x7ff45c0b4f00 2024-03-25T18:26:44.031 INFO:teuthology.orchestra.run.smithi115.stderr:2024-03-25T18:26:44.028+0000 7ff459d9b640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:26:44.031 INFO:teuthology.orchestra.run.smithi115.stderr:2024-03-25T18:26:44.028+0000 7ff459d9b640 1 -- 172.21.15.115:0/273510519 <== mon.1 v1:172.21.15.115:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1586965025 0 0) 0x7ff44c001c00 con 0x7ff45c0b4f00 2024-03-25T18:26:44.031 INFO:teuthology.orchestra.run.smithi115.stderr:2024-03-25T18:26:44.028+0000 7ff459d9b640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:26:44.031 INFO:teuthology.orchestra.run.smithi115.stderr:2024-03-25T18:26:44.028+0000 7ff459d9b640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-25T18:26:44.031 INFO:teuthology.orchestra.run.smithi115.stderr:2024-03-25T18:26:44.028+0000 7ff459d9b640 10 monclient: dump: 2024-03-25T18:26:44.032 INFO:teuthology.orchestra.run.smithi115.stderr:epoch 1 2024-03-25T18:26:44.032 INFO:teuthology.orchestra.run.smithi115.stderr:fsid c8ea72cf-9e99-4383-9b52-6c23d365fdfc 2024-03-25T18:26:44.032 INFO:teuthology.orchestra.run.smithi115.stderr:last_changed 2024-03-25T18:25:50.510519+0000 2024-03-25T18:26:44.032 INFO:teuthology.orchestra.run.smithi115.stderr:created 2024-03-25T18:25:50.510519+0000 2024-03-25T18:26:44.032 INFO:teuthology.orchestra.run.smithi115.stderr:min_mon_release 19 (squid) 2024-03-25T18:26:44.032 INFO:teuthology.orchestra.run.smithi115.stderr:election_strategy: 3 2024-03-25T18:26:44.032 INFO:teuthology.orchestra.run.smithi115.stderr:0: v1:172.21.15.1:6789/0 mon.a 2024-03-25T18:26:44.032 INFO:teuthology.orchestra.run.smithi115.stderr:1: v1:172.21.15.115:6789/0 mon.b 2024-03-25T18:26:44.032 INFO:teuthology.orchestra.run.smithi115.stderr:2: v1:172.21.15.1:6790/0 mon.c 2024-03-25T18:26:44.032 INFO:teuthology.orchestra.run.smithi115.stderr: 2024-03-25T18:26:44.032 INFO:teuthology.orchestra.run.smithi115.stderr:2024-03-25T18:26:44.028+0000 7ff460cdd640 10 monclient: get_monmap_and_config success 2024-03-25T18:26:44.032 INFO:teuthology.orchestra.run.smithi115.stderr:2024-03-25T18:26:44.028+0000 7ff460cdd640 4 set_mon_vals no callback set 2024-03-25T18:26:44.032 INFO:teuthology.orchestra.run.smithi115.stderr:2024-03-25T18:26:44.029+0000 7ff460cdd640 10 monclient: shutdown 2024-03-25T18:26:44.032 INFO:teuthology.orchestra.run.smithi115.stderr:2024-03-25T18:26:44.029+0000 7ff460cdd640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:26:44.032 INFO:teuthology.orchestra.run.smithi115.stderr:2024-03-25T18:26:44.029+0000 7ff460cdd640 1 -- 172.21.15.115:0/273510519 >> v1:172.21.15.115:6789/0 conn(0x7ff45c0b4f00 legacy=0x7ff45c1423c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:26:44.032 INFO:teuthology.orchestra.run.smithi115.stderr:2024-03-25T18:26:44.029+0000 7ff460cdd640 1 -- 172.21.15.115:0/273510519 shutdown_connections 2024-03-25T18:26:44.033 INFO:teuthology.orchestra.run.smithi115.stderr:2024-03-25T18:26:44.029+0000 7ff460cdd640 1 -- 172.21.15.115:0/273510519 >> 172.21.15.115:0/273510519 conn(0x7ff45c066b00 msgr2=0x7ff45c066f10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:26:44.033 INFO:teuthology.orchestra.run.smithi115.stderr:2024-03-25T18:26:44.029+0000 7ff460cdd640 1 -- 172.21.15.115:0/273510519 shutdown_connections 2024-03-25T18:26:44.033 INFO:teuthology.orchestra.run.smithi115.stderr:2024-03-25T18:26:44.029+0000 7ff460cdd640 1 -- 172.21.15.115:0/273510519 wait complete. 2024-03-25T18:26:44.033 INFO:teuthology.orchestra.run.smithi115.stderr:2024-03-25T18:26:44.030+0000 7ff460cdd640 10 monclient: build_initial_monmap 2024-03-25T18:26:44.033 INFO:teuthology.orchestra.run.smithi115.stderr:2024-03-25T18:26:44.030+0000 7ff460cdd640 10 monclient: monmap: 2024-03-25T18:26:44.033 INFO:teuthology.orchestra.run.smithi115.stderr:epoch 0 2024-03-25T18:26:44.033 INFO:teuthology.orchestra.run.smithi115.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T18:26:44.033 INFO:teuthology.orchestra.run.smithi115.stderr:last_changed 0.000000 2024-03-25T18:26:44.033 INFO:teuthology.orchestra.run.smithi115.stderr:created 0.000000 2024-03-25T18:26:44.033 INFO:teuthology.orchestra.run.smithi115.stderr:min_mon_release 0 (unknown) 2024-03-25T18:26:44.033 INFO:teuthology.orchestra.run.smithi115.stderr:election_strategy: 1 2024-03-25T18:26:44.033 INFO:teuthology.orchestra.run.smithi115.stderr:0: v1:172.21.15.1:6789/0 mon.noname-a 2024-03-25T18:26:44.033 INFO:teuthology.orchestra.run.smithi115.stderr:1: v1:172.21.15.115:6789/0 mon.noname-b 2024-03-25T18:26:44.033 INFO:teuthology.orchestra.run.smithi115.stderr:2: v1:172.21.15.1:6790/0 mon.noname-c 2024-03-25T18:26:44.033 INFO:teuthology.orchestra.run.smithi115.stderr: 2024-03-25T18:26:44.033 INFO:teuthology.orchestra.run.smithi115.stderr:2024-03-25T18:26:44.030+0000 7ff460cdd640 1 librados: starting msgr at 2024-03-25T18:26:44.034 INFO:teuthology.orchestra.run.smithi115.stderr:2024-03-25T18:26:44.030+0000 7ff460cdd640 1 librados: starting objecter 2024-03-25T18:26:44.034 INFO:teuthology.orchestra.run.smithi115.stderr:2024-03-25T18:26:44.030+0000 7ff460cdd640 1 Processor -- start 2024-03-25T18:26:44.034 INFO:teuthology.orchestra.run.smithi115.stderr:2024-03-25T18:26:44.030+0000 7ff460cdd640 1 -- start start 2024-03-25T18:26:44.034 INFO:teuthology.orchestra.run.smithi115.stderr:2024-03-25T18:26:44.030+0000 7ff460cdd640 1 librados: setting wanted keys 2024-03-25T18:26:44.034 INFO:teuthology.orchestra.run.smithi115.stderr:2024-03-25T18:26:44.030+0000 7ff460cdd640 1 librados: calling monclient init 2024-03-25T18:26:44.034 INFO:teuthology.orchestra.run.smithi115.stderr:2024-03-25T18:26:44.030+0000 7ff460cdd640 10 monclient: init 2024-03-25T18:26:44.036 INFO:teuthology.orchestra.run.smithi115.stderr:2024-03-25T18:26:44.031+0000 7ff460cdd640 10 monclient: _reopen_session rank -1 2024-03-25T18:26:44.036 INFO:teuthology.orchestra.run.smithi115.stderr:2024-03-25T18:26:44.031+0000 7ff460cdd640 10 monclient: _add_conns ranks=[1,0,2] 2024-03-25T18:26:44.037 INFO:teuthology.orchestra.run.smithi115.stderr:2024-03-25T18:26:44.031+0000 7ff460cdd640 10 monclient(hunting): picked mon.noname-b con 0x7ff45c0b4510 addr v1:172.21.15.115:6789/0 2024-03-25T18:26:44.037 INFO:teuthology.orchestra.run.smithi115.stderr:2024-03-25T18:26:44.031+0000 7ff460cdd640 10 monclient(hunting): picked mon.noname-a con 0x7ff45c0b4f00 addr v1:172.21.15.1:6789/0 2024-03-25T18:26:44.037 INFO:teuthology.orchestra.run.smithi115.stderr:2024-03-25T18:26:44.031+0000 7ff460cdd640 10 monclient(hunting): picked mon.noname-c con 0x7ff45c142ad0 addr v1:172.21.15.1:6790/0 2024-03-25T18:26:44.037 INFO:teuthology.orchestra.run.smithi115.stderr:2024-03-25T18:26:44.031+0000 7ff460cdd640 1 -- --> v1:172.21.15.1:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff45c0b5ad0 con 0x7ff45c0b4f00 2024-03-25T18:26:44.037 INFO:teuthology.orchestra.run.smithi115.stderr:2024-03-25T18:26:44.031+0000 7ff460cdd640 1 -- --> v1:172.21.15.115:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff45c0b6d10 con 0x7ff45c0b4510 2024-03-25T18:26:44.037 INFO:teuthology.orchestra.run.smithi115.stderr:2024-03-25T18:26:44.031+0000 7ff460cdd640 1 -- --> v1:172.21.15.1:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff45c0b79e0 con 0x7ff45c142ad0 2024-03-25T18:26:44.037 INFO:teuthology.orchestra.run.smithi115.stderr:2024-03-25T18:26:44.031+0000 7ff460cdd640 10 monclient(hunting): _renew_subs 2024-03-25T18:26:44.037 INFO:teuthology.orchestra.run.smithi115.stderr:2024-03-25T18:26:44.031+0000 7ff460cdd640 10 monclient(hunting): authenticate will time out at 7495.187165s 2024-03-25T18:26:44.037 INFO:teuthology.orchestra.run.smithi115.stderr:2024-03-25T18:26:44.031+0000 7ff45a59c640 1 --1- >> v1:172.21.15.115:6789/0 conn(0x7ff45c0b4510 0x7ff45c0b8730 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.115:6789/0 says I am v1:172.21.15.115:44668/0 (socket says 172.21.15.115:44668) 2024-03-25T18:26:44.037 INFO:teuthology.orchestra.run.smithi115.stderr:2024-03-25T18:26:44.031+0000 7ff453fff640 1 --1- >> v1:172.21.15.1:6789/0 conn(0x7ff45c0b4f00 0x7ff45c0b5f60 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.1:6789/0 says I am v1:172.21.15.115:55678/0 (socket says 172.21.15.115:55678) 2024-03-25T18:26:44.037 INFO:teuthology.orchestra.run.smithi115.stderr:2024-03-25T18:26:44.031+0000 7ff45a59c640 1 -- 172.21.15.115:0/594149751 learned_addr learned my addr 172.21.15.115:0/594149751 (peer_addr_for_me v1:172.21.15.115:0/0) 2024-03-25T18:26:44.037 INFO:teuthology.orchestra.run.smithi115.stderr:2024-03-25T18:26:44.032+0000 7ff452ffd640 10 client.?.objecter ms_handle_connect 0x7ff45c0b4510 2024-03-25T18:26:44.037 INFO:teuthology.orchestra.run.smithi115.stderr:2024-03-25T18:26:44.032+0000 7ff452ffd640 10 client.?.objecter resend_mon_ops 2024-03-25T18:26:44.037 INFO:teuthology.orchestra.run.smithi115.stderr:2024-03-25T18:26:44.032+0000 7ff452ffd640 10 client.?.objecter ms_handle_connect 0x7ff45c0b4f00 2024-03-25T18:26:44.038 INFO:teuthology.orchestra.run.smithi115.stderr:2024-03-25T18:26:44.032+0000 7ff452ffd640 10 client.?.objecter resend_mon_ops 2024-03-25T18:26:44.038 INFO:teuthology.orchestra.run.smithi115.stderr:2024-03-25T18:26:44.032+0000 7ff452ffd640 10 client.?.objecter ms_handle_connect 0x7ff45c142ad0 2024-03-25T18:26:44.038 INFO:teuthology.orchestra.run.smithi115.stderr:2024-03-25T18:26:44.032+0000 7ff452ffd640 10 client.?.objecter resend_mon_ops 2024-03-25T18:26:44.038 INFO:teuthology.orchestra.run.smithi115.stderr:2024-03-25T18:26:44.033+0000 7ff452ffd640 1 -- 172.21.15.115:0/594149751 <== mon.0 v1:172.21.15.1:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2484455067 0 0) 0x7ff45c0b5ad0 con 0x7ff45c0b4f00 2024-03-25T18:26:44.038 INFO:teuthology.orchestra.run.smithi115.stderr:2024-03-25T18:26:44.033+0000 7ff452ffd640 10 monclient(hunting): _negotiate 2024-03-25T18:26:44.038 INFO:teuthology.orchestra.run.smithi115.stderr:2024-03-25T18:26:44.033+0000 7ff452ffd640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:26:44.038 INFO:teuthology.orchestra.run.smithi115.stderr:2024-03-25T18:26:44.033+0000 7ff452ffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:26:44.038 INFO:teuthology.orchestra.run.smithi115.stderr:2024-03-25T18:26:44.033+0000 7ff452ffd640 10 monclient(hunting): my global_id is 4158 2024-03-25T18:26:44.038 INFO:teuthology.orchestra.run.smithi115.stderr:2024-03-25T18:26:44.033+0000 7ff452ffd640 1 -- 172.21.15.115:0/594149751 --> v1:172.21.15.1:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ff43c004eb0 con 0x7ff45c0b4f00 2024-03-25T18:26:44.038 INFO:teuthology.orchestra.run.smithi115.stderr:2024-03-25T18:26:44.033+0000 7ff452ffd640 1 -- 172.21.15.115:0/594149751 <== mon.1 v1:172.21.15.115:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2767712533 0 0) 0x7ff45c0b6d10 con 0x7ff45c0b4510 2024-03-25T18:26:44.038 INFO:teuthology.orchestra.run.smithi115.stderr:2024-03-25T18:26:44.033+0000 7ff452ffd640 10 monclient(hunting): _negotiate 2024-03-25T18:26:44.038 INFO:teuthology.orchestra.run.smithi115.stderr:2024-03-25T18:26:44.033+0000 7ff452ffd640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:26:44.038 INFO:teuthology.orchestra.run.smithi115.stderr:2024-03-25T18:26:44.033+0000 7ff452ffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:26:44.038 INFO:teuthology.orchestra.run.smithi115.stderr:2024-03-25T18:26:44.033+0000 7ff452ffd640 10 monclient(hunting): my global_id is 4153 2024-03-25T18:26:44.039 INFO:teuthology.orchestra.run.smithi115.stderr:2024-03-25T18:26:44.033+0000 7ff452ffd640 1 -- 172.21.15.115:0/594149751 --> v1:172.21.15.115:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ff45c0b5ad0 con 0x7ff45c0b4510 2024-03-25T18:26:44.039 INFO:teuthology.orchestra.run.smithi115.stderr:2024-03-25T18:26:44.034+0000 7ff452ffd640 1 -- 172.21.15.115:0/594149751 <== mon.2 v1:172.21.15.1:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3640505420 0 0) 0x7ff45c0b79e0 con 0x7ff45c142ad0 2024-03-25T18:26:44.039 INFO:teuthology.orchestra.run.smithi115.stderr:2024-03-25T18:26:44.034+0000 7ff452ffd640 10 monclient(hunting): _negotiate 2024-03-25T18:26:44.039 INFO:teuthology.orchestra.run.smithi115.stderr:2024-03-25T18:26:44.034+0000 7ff452ffd640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:26:44.039 INFO:teuthology.orchestra.run.smithi115.stderr:2024-03-25T18:26:44.034+0000 7ff452ffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:26:44.039 INFO:teuthology.orchestra.run.smithi115.stderr:2024-03-25T18:26:44.034+0000 7ff452ffd640 10 monclient(hunting): my global_id is 4157 2024-03-25T18:26:44.039 INFO:teuthology.orchestra.run.smithi115.stderr:2024-03-25T18:26:44.034+0000 7ff452ffd640 1 -- 172.21.15.115:0/594149751 --> v1:172.21.15.1:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ff45c0b6d10 con 0x7ff45c142ad0 2024-03-25T18:26:44.040 INFO:teuthology.orchestra.run.smithi115.stderr:2024-03-25T18:26:44.038+0000 7ff452ffd640 1 -- 172.21.15.115:0/594149751 <== mon.1 v1:172.21.15.115:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1830132477 0 0) 0x7ff45c0b5ad0 con 0x7ff45c0b4510 2024-03-25T18:26:44.040 INFO:teuthology.orchestra.run.smithi115.stderr:2024-03-25T18:26:44.038+0000 7ff452ffd640 1 -- 172.21.15.115:0/594149751 --> v1:172.21.15.115:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7ff45c0b79e0 con 0x7ff45c0b4510 2024-03-25T18:26:44.040 INFO:teuthology.orchestra.run.smithi115.stderr:2024-03-25T18:26:44.038+0000 7ff452ffd640 1 -- 172.21.15.115:0/594149751 <== mon.1 v1:172.21.15.115:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1586965025 0 0) 0x7ff44c001b40 con 0x7ff45c0b4510 2024-03-25T18:26:44.040 INFO:teuthology.orchestra.run.smithi115.stderr:2024-03-25T18:26:44.038+0000 7ff452ffd640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:26:44.040 INFO:teuthology.orchestra.run.smithi115.stderr:2024-03-25T18:26:44.038+0000 7ff452ffd640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-03-25T18:26:44.040 INFO:teuthology.orchestra.run.smithi115.stderr:2024-03-25T18:26:44.038+0000 7ff452ffd640 10 monclient(hunting): dump: 2024-03-25T18:26:44.040 INFO:teuthology.orchestra.run.smithi115.stderr:epoch 1 2024-03-25T18:26:44.040 INFO:teuthology.orchestra.run.smithi115.stderr:fsid c8ea72cf-9e99-4383-9b52-6c23d365fdfc 2024-03-25T18:26:44.041 INFO:teuthology.orchestra.run.smithi115.stderr:last_changed 2024-03-25T18:25:50.510519+0000 2024-03-25T18:26:44.041 INFO:teuthology.orchestra.run.smithi115.stderr:created 2024-03-25T18:25:50.510519+0000 2024-03-25T18:26:44.041 INFO:teuthology.orchestra.run.smithi115.stderr:min_mon_release 19 (squid) 2024-03-25T18:26:44.041 INFO:teuthology.orchestra.run.smithi115.stderr:election_strategy: 3 2024-03-25T18:26:44.041 INFO:teuthology.orchestra.run.smithi115.stderr:0: v1:172.21.15.1:6789/0 mon.a 2024-03-25T18:26:44.041 INFO:teuthology.orchestra.run.smithi115.stderr:1: v1:172.21.15.115:6789/0 mon.b 2024-03-25T18:26:44.041 INFO:teuthology.orchestra.run.smithi115.stderr:2: v1:172.21.15.1:6790/0 mon.c 2024-03-25T18:26:44.041 INFO:teuthology.orchestra.run.smithi115.stderr: 2024-03-25T18:26:44.041 INFO:teuthology.orchestra.run.smithi115.stderr:2024-03-25T18:26:44.038+0000 7ff452ffd640 10 monclient(hunting): _finish_auth 0 2024-03-25T18:26:44.041 INFO:teuthology.orchestra.run.smithi115.stderr:2024-03-25T18:26:44.038+0000 7ff452ffd640 1 -- 172.21.15.115:0/594149751 <== mon.2 v1:172.21.15.1:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 671778566 0 0) 0x7ff45c0b6d10 con 0x7ff45c142ad0 2024-03-25T18:26:44.041 INFO:teuthology.orchestra.run.smithi115.stderr:2024-03-25T18:26:44.038+0000 7ff452ffd640 1 -- 172.21.15.115:0/594149751 --> v1:172.21.15.1:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7ff45c0b5ad0 con 0x7ff45c142ad0 2024-03-25T18:26:44.041 INFO:teuthology.orchestra.run.smithi115.stderr:2024-03-25T18:26:44.038+0000 7ff452ffd640 1 -- 172.21.15.115:0/594149751 <== mon.2 v1:172.21.15.1:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1586965025 0 0) 0x7ff454001bd0 con 0x7ff45c142ad0 2024-03-25T18:26:44.041 INFO:teuthology.orchestra.run.smithi115.stderr:2024-03-25T18:26:44.038+0000 7ff452ffd640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:26:44.041 INFO:teuthology.orchestra.run.smithi115.stderr:2024-03-25T18:26:44.038+0000 7ff452ffd640 10 monclient(hunting): got monmap 1 from mon.c (according to old e1) 2024-03-25T18:26:44.041 INFO:teuthology.orchestra.run.smithi115.stderr:2024-03-25T18:26:44.038+0000 7ff452ffd640 10 monclient(hunting): dump: 2024-03-25T18:26:44.041 INFO:teuthology.orchestra.run.smithi115.stderr:epoch 1 2024-03-25T18:26:44.042 INFO:teuthology.orchestra.run.smithi115.stderr:fsid c8ea72cf-9e99-4383-9b52-6c23d365fdfc 2024-03-25T18:26:44.042 INFO:teuthology.orchestra.run.smithi115.stderr:last_changed 2024-03-25T18:25:50.510519+0000 2024-03-25T18:26:44.042 INFO:teuthology.orchestra.run.smithi115.stderr:created 2024-03-25T18:25:50.510519+0000 2024-03-25T18:26:44.042 INFO:teuthology.orchestra.run.smithi115.stderr:min_mon_release 19 (squid) 2024-03-25T18:26:44.042 INFO:teuthology.orchestra.run.smithi115.stderr:election_strategy: 3 2024-03-25T18:26:44.042 INFO:teuthology.orchestra.run.smithi115.stderr:0: v1:172.21.15.1:6789/0 mon.a 2024-03-25T18:26:44.042 INFO:teuthology.orchestra.run.smithi115.stderr:1: v1:172.21.15.115:6789/0 mon.b 2024-03-25T18:26:44.042 INFO:teuthology.orchestra.run.smithi115.stderr:2: v1:172.21.15.1:6790/0 mon.c 2024-03-25T18:26:44.042 INFO:teuthology.orchestra.run.smithi115.stderr: 2024-03-25T18:26:44.044 INFO:teuthology.orchestra.run.smithi115.stderr:2024-03-25T18:26:44.040+0000 7ff452ffd640 1 -- 172.21.15.115:0/594149751 <== mon.0 v1:172.21.15.1:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1520456689 0 0) 0x7ff43c004eb0 con 0x7ff45c0b4f00 2024-03-25T18:26:44.044 INFO:teuthology.orchestra.run.smithi115.stderr:2024-03-25T18:26:44.040+0000 7ff452ffd640 1 -- 172.21.15.115:0/594149751 --> v1:172.21.15.1:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7ff45c0b6d10 con 0x7ff45c0b4f00 2024-03-25T18:26:44.044 INFO:teuthology.orchestra.run.smithi115.stderr:2024-03-25T18:26:44.040+0000 7ff452ffd640 1 -- 172.21.15.115:0/594149751 <== mon.0 v1:172.21.15.1:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1586965025 0 0) 0x7ff4440019f0 con 0x7ff45c0b4f00 2024-03-25T18:26:44.044 INFO:teuthology.orchestra.run.smithi115.stderr:2024-03-25T18:26:44.040+0000 7ff452ffd640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:26:44.044 INFO:teuthology.orchestra.run.smithi115.stderr:2024-03-25T18:26:44.040+0000 7ff452ffd640 10 monclient(hunting): got monmap 1 from mon.a (according to old e1) 2024-03-25T18:26:44.044 INFO:teuthology.orchestra.run.smithi115.stderr:2024-03-25T18:26:44.040+0000 7ff452ffd640 10 monclient(hunting): dump: 2024-03-25T18:26:44.044 INFO:teuthology.orchestra.run.smithi115.stderr:epoch 1 2024-03-25T18:26:44.044 INFO:teuthology.orchestra.run.smithi115.stderr:fsid c8ea72cf-9e99-4383-9b52-6c23d365fdfc 2024-03-25T18:26:44.044 INFO:teuthology.orchestra.run.smithi115.stderr:last_changed 2024-03-25T18:25:50.510519+0000 2024-03-25T18:26:44.044 INFO:teuthology.orchestra.run.smithi115.stderr:created 2024-03-25T18:25:50.510519+0000 2024-03-25T18:26:44.044 INFO:teuthology.orchestra.run.smithi115.stderr:min_mon_release 19 (squid) 2024-03-25T18:26:44.045 INFO:teuthology.orchestra.run.smithi115.stderr:election_strategy: 3 2024-03-25T18:26:44.045 INFO:teuthology.orchestra.run.smithi115.stderr:0: v1:172.21.15.1:6789/0 mon.a 2024-03-25T18:26:44.045 INFO:teuthology.orchestra.run.smithi115.stderr:1: v1:172.21.15.115:6789/0 mon.b 2024-03-25T18:26:44.045 INFO:teuthology.orchestra.run.smithi115.stderr:2: v1:172.21.15.1:6790/0 mon.c 2024-03-25T18:26:44.045 INFO:teuthology.orchestra.run.smithi115.stderr: 2024-03-25T18:26:44.045 INFO:teuthology.orchestra.run.smithi115.stderr:2024-03-25T18:26:44.041+0000 7ff452ffd640 1 -- 172.21.15.115:0/594149751 <== mon.1 v1:172.21.15.115:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 3489150608 0 0) 0x7ff45c0b79e0 con 0x7ff45c0b4510 2024-03-25T18:26:44.045 INFO:teuthology.orchestra.run.smithi115.stderr:2024-03-25T18:26:44.041+0000 7ff452ffd640 1 -- 172.21.15.115:0/594149751 >> v1:172.21.15.1:6790/0 conn(0x7ff45c142ad0 legacy=0x7ff45c1e1f20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:26:44.045 INFO:teuthology.orchestra.run.smithi115.stderr:2024-03-25T18:26:44.041+0000 7ff452ffd640 1 -- 172.21.15.115:0/594149751 >> v1:172.21.15.1:6789/0 conn(0x7ff45c0b4f00 legacy=0x7ff45c0b5f60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:26:44.045 INFO:teuthology.orchestra.run.smithi115.stderr:2024-03-25T18:26:44.041+0000 7ff452ffd640 10 monclient: _finish_hunting 0 2024-03-25T18:26:44.045 INFO:teuthology.orchestra.run.smithi115.stderr:2024-03-25T18:26:44.041+0000 7ff452ffd640 1 monclient: found mon.b 2024-03-25T18:26:44.045 INFO:teuthology.orchestra.run.smithi115.stderr:2024-03-25T18:26:44.041+0000 7ff452ffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:26:44.045 INFO:teuthology.orchestra.run.smithi115.stderr:2024-03-25T18:26:44.041+0000 7ff452ffd640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.115:6789/0 2024-03-25T18:26:44.045 INFO:teuthology.orchestra.run.smithi115.stderr:2024-03-25T18:26:44.041+0000 7ff452ffd640 1 -- 172.21.15.115:0/594149751 --> v1:172.21.15.115:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff45c0b7cc0 con 0x7ff45c0b4510 2024-03-25T18:26:44.045 INFO:teuthology.orchestra.run.smithi115.stderr:2024-03-25T18:26:44.041+0000 7ff452ffd640 10 monclient: _finish_auth 0 2024-03-25T18:26:44.046 INFO:teuthology.orchestra.run.smithi115.stderr:2024-03-25T18:26:44.041+0000 7ff452ffd640 10 monclient: _check_auth_tickets 2024-03-25T18:26:44.046 INFO:teuthology.orchestra.run.smithi115.stderr:2024-03-25T18:26:44.041+0000 7ff452ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:26:44.046 INFO:teuthology.orchestra.run.smithi115.stderr:2024-03-25T18:26:44.041+0000 7ff460cdd640 5 monclient: authenticate success, global_id 4153 2024-03-25T18:26:44.046 INFO:teuthology.orchestra.run.smithi115.stderr:2024-03-25T18:26:44.041+0000 7ff460cdd640 10 monclient: _renew_subs 2024-03-25T18:26:44.046 INFO:teuthology.orchestra.run.smithi115.stderr:2024-03-25T18:26:44.041+0000 7ff460cdd640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.115:6789/0 2024-03-25T18:26:44.046 INFO:teuthology.orchestra.run.smithi115.stderr:2024-03-25T18:26:44.041+0000 7ff460cdd640 1 -- 172.21.15.115:0/594149751 --> v1:172.21.15.115:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff45c0b7fa0 con 0x7ff45c0b4510 2024-03-25T18:26:44.046 INFO:teuthology.orchestra.run.smithi115.stderr:2024-03-25T18:26:44.041+0000 7ff460cdd640 10 client.4153.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T18:26:44.046 INFO:teuthology.orchestra.run.smithi115.stderr:2024-03-25T18:26:44.041+0000 7ff460cdd640 10 monclient: _renew_subs 2024-03-25T18:26:44.046 INFO:teuthology.orchestra.run.smithi115.stderr:2024-03-25T18:26:44.041+0000 7ff460cdd640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.115:6789/0 2024-03-25T18:26:44.046 INFO:teuthology.orchestra.run.smithi115.stderr:2024-03-25T18:26:44.041+0000 7ff460cdd640 1 -- 172.21.15.115:0/594149751 --> v1:172.21.15.115:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7ff45c0b6850 con 0x7ff45c0b4510 2024-03-25T18:26:44.046 INFO:teuthology.orchestra.run.smithi115.stderr:2024-03-25T18:26:44.041+0000 7ff460cdd640 1 librados: init done 2024-03-25T18:26:44.046 INFO:teuthology.orchestra.run.smithi115.stderr:2024-03-25T18:26:44.042+0000 7ff460cdd640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T18:26:44.046 INFO:teuthology.orchestra.run.smithi115.stderr:2024-03-25T18:26:44.042+0000 7ff460cdd640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:26:44.046 INFO:teuthology.orchestra.run.smithi115.stderr:2024-03-25T18:26:44.042+0000 7ff460cdd640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.115:6789/0 2024-03-25T18:26:44.047 INFO:teuthology.orchestra.run.smithi115.stderr:2024-03-25T18:26:44.042+0000 7ff460cdd640 1 -- 172.21.15.115:0/594149751 --> v1:172.21.15.115:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff45c0b6b00 con 0x7ff45c0b4510 2024-03-25T18:26:44.088 INFO:tasks.ceph.mgr.y.smithi001.stderr:2024-03-25T18:26:44.086+0000 7faca7441040 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2024-03-25T18:26:44.103 INFO:tasks.ceph.mgr.x.smithi115.stderr:2024-03-25T18:26:44.102+0000 7f99dd247040 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2024-03-25T18:26:44.142 INFO:teuthology.orchestra.run.smithi115.stderr:2024-03-25T18:26:44.141+0000 7ff452ffd640 1 -- 172.21.15.115:0/594149751 <== mon.1 v1:172.21.15.115:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7ff44c0011f0 con 0x7ff45c0b4510 2024-03-25T18:26:44.142 INFO:teuthology.orchestra.run.smithi115.stderr:2024-03-25T18:26:44.141+0000 7ff452ffd640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:26:44.142 INFO:teuthology.orchestra.run.smithi115.stderr:2024-03-25T18:26:44.141+0000 7ff452ffd640 1 -- 172.21.15.115:0/594149751 <== mon.1 v1:172.21.15.115:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1586965025 0 0) 0x7ff44c004c30 con 0x7ff45c0b4510 2024-03-25T18:26:44.142 INFO:teuthology.orchestra.run.smithi115.stderr:2024-03-25T18:26:44.141+0000 7ff452ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:26:44.142 INFO:teuthology.orchestra.run.smithi115.stderr:2024-03-25T18:26:44.141+0000 7ff452ffd640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-25T18:26:44.142 INFO:teuthology.orchestra.run.smithi115.stderr:2024-03-25T18:26:44.141+0000 7ff452ffd640 10 monclient: dump: 2024-03-25T18:26:44.143 INFO:teuthology.orchestra.run.smithi115.stderr:epoch 1 2024-03-25T18:26:44.143 INFO:teuthology.orchestra.run.smithi115.stderr:fsid c8ea72cf-9e99-4383-9b52-6c23d365fdfc 2024-03-25T18:26:44.143 INFO:teuthology.orchestra.run.smithi115.stderr:last_changed 2024-03-25T18:25:50.510519+0000 2024-03-25T18:26:44.143 INFO:teuthology.orchestra.run.smithi115.stderr:created 2024-03-25T18:25:50.510519+0000 2024-03-25T18:26:44.143 INFO:teuthology.orchestra.run.smithi115.stderr:min_mon_release 19 (squid) 2024-03-25T18:26:44.143 INFO:teuthology.orchestra.run.smithi115.stderr:election_strategy: 3 2024-03-25T18:26:44.143 INFO:teuthology.orchestra.run.smithi115.stderr:0: v1:172.21.15.1:6789/0 mon.a 2024-03-25T18:26:44.143 INFO:teuthology.orchestra.run.smithi115.stderr:1: v1:172.21.15.115:6789/0 mon.b 2024-03-25T18:26:44.143 INFO:teuthology.orchestra.run.smithi115.stderr:2: v1:172.21.15.1:6790/0 mon.c 2024-03-25T18:26:44.143 INFO:teuthology.orchestra.run.smithi115.stderr: 2024-03-25T18:26:44.143 INFO:teuthology.orchestra.run.smithi115.stderr:2024-03-25T18:26:44.141+0000 7ff452ffd640 1 -- 172.21.15.115:0/594149751 <== mon.1 v1:172.21.15.115:6789/0 7 ==== mgrmap(e 1) v1 ==== 807+0+0 (unknown 3189685207 0 0) 0x7ff44c005120 con 0x7ff45c0b4510 2024-03-25T18:26:44.143 INFO:teuthology.orchestra.run.smithi115.stderr:2024-03-25T18:26:44.141+0000 7ff45959a640 4 set_mon_vals no callback set 2024-03-25T18:26:44.143 INFO:teuthology.orchestra.run.smithi115.stderr:2024-03-25T18:26:44.141+0000 7ff452ffd640 1 -- 172.21.15.115:0/594149751 <== mon.1 v1:172.21.15.115:6789/0 8 ==== osd_map(9..9 src has 1..9) v4 ==== 1530+0+0 (unknown 2919229232 0 0) 0x7ff44c0069a0 con 0x7ff45c0b4510 2024-03-25T18:26:44.143 INFO:teuthology.orchestra.run.smithi115.stderr:2024-03-25T18:26:44.141+0000 7ff452ffd640 10 client.4153.objecter ms_dispatch 0x7ff45c000b90 osd_map(9..9 src has 1..9) v4 2024-03-25T18:26:44.143 INFO:teuthology.orchestra.run.smithi115.stderr:2024-03-25T18:26:44.141+0000 7ff452ffd640 3 client.4153.objecter handle_osd_map got epochs [9,9] > 0 2024-03-25T18:26:44.144 INFO:teuthology.orchestra.run.smithi115.stderr:2024-03-25T18:26:44.141+0000 7ff452ffd640 3 client.4153.objecter handle_osd_map decoding full epoch 9 2024-03-25T18:26:44.144 INFO:teuthology.orchestra.run.smithi115.stderr:2024-03-25T18:26:44.141+0000 7ff452ffd640 20 client.4153.objecter dump_active .. 0 homeless 2024-03-25T18:26:44.144 INFO:teuthology.orchestra.run.smithi115.stderr:2024-03-25T18:26:44.141+0000 7ff452ffd640 1 -- 172.21.15.115:0/594149751 <== mon.1 v1:172.21.15.115:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+73141 (unknown 1092875540 0 2031569661) 0x7ff44c018dc0 con 0x7ff45c0b4510 2024-03-25T18:26:44.144 INFO:teuthology.orchestra.run.smithi115.stderr:2024-03-25T18:26:44.141+0000 7ff452ffd640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:26:44.144 INFO:teuthology.orchestra.run.smithi115.stderr:2024-03-25T18:26:44.141+0000 7ff452ffd640 10 monclient: _finish_command 1 = system:0 2024-03-25T18:26:44.156 INFO:tasks.ceph.mgr.x.smithi115.stderr:2024-03-25T18:26:44.156+0000 7f99dd247040 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2024-03-25T18:26:44.212 INFO:teuthology.orchestra.run.smithi115.stderr:2024-03-25T18:26:44.211+0000 7ff460cdd640 10 start_mon_command cmd=[{"prefix": "osd new", "uuid": "cac47bee-0914-4f08-896f-348838882866", "id": 7}] 2024-03-25T18:26:44.212 INFO:teuthology.orchestra.run.smithi115.stderr:2024-03-25T18:26:44.211+0000 7ff460cdd640 10 monclient: _send_command 2 [{"prefix": "osd new", "uuid": "cac47bee-0914-4f08-896f-348838882866", "id": 7}] 2024-03-25T18:26:44.212 INFO:teuthology.orchestra.run.smithi115.stderr:2024-03-25T18:26:44.211+0000 7ff460cdd640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.115:6789/0 2024-03-25T18:26:44.212 INFO:teuthology.orchestra.run.smithi115.stderr:2024-03-25T18:26:44.211+0000 7ff460cdd640 1 -- 172.21.15.115:0/594149751 --> v1:172.21.15.115:6789/0 -- mon_command({"prefix": "osd new", "uuid": "cac47bee-0914-4f08-896f-348838882866", "id": 7} v 0) v1 -- 0x7ff45c04f730 con 0x7ff45c0b4510 2024-03-25T18:26:44.274 INFO:tasks.ceph.mgr.x.smithi115.stderr:2024-03-25T18:26:44.273+0000 7f99dd247040 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2024-03-25T18:26:44.292 INFO:teuthology.orchestra.run.smithi115.stderr:2024-03-25T18:26:44.292+0000 7ff452ffd640 1 -- 172.21.15.115:0/594149751 <== mon.1 v1:172.21.15.115:6789/0 10 ==== mon_command_ack([{"prefix": "osd new", "uuid": "cac47bee-0914-4f08-896f-348838882866", "id": 7}]=0 v10) v1 ==== 112+0+2 (unknown 709001820 0 1371665406) 0x7ff44c006410 con 0x7ff45c0b4510 2024-03-25T18:26:44.292 INFO:teuthology.orchestra.run.smithi115.stderr:2024-03-25T18:26:44.292+0000 7ff452ffd640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd new", "uuid": "cac47bee-0914-4f08-896f-348838882866", "id": 7}] 2024-03-25T18:26:44.293 INFO:teuthology.orchestra.run.smithi115.stderr:2024-03-25T18:26:44.292+0000 7ff452ffd640 10 monclient: _finish_command 2 = system:0 2024-03-25T18:26:44.293 INFO:teuthology.orchestra.run.smithi115.stdout:7 2024-03-25T18:26:44.294 INFO:teuthology.orchestra.run.smithi115.stderr:2024-03-25T18:26:44.292+0000 7ff460cdd640 10 librados: watch_flush enter 2024-03-25T18:26:44.294 INFO:teuthology.orchestra.run.smithi115.stderr:2024-03-25T18:26:44.292+0000 7ff460cdd640 10 librados: watch_flush exit 2024-03-25T18:26:44.294 INFO:teuthology.orchestra.run.smithi115.stderr:2024-03-25T18:26:44.292+0000 7ff460cdd640 20 client.4153.objecter shutdown clearing up homeless session... 2024-03-25T18:26:44.294 INFO:teuthology.orchestra.run.smithi115.stderr:2024-03-25T18:26:44.292+0000 7ff460cdd640 10 client.4153.objecter successfully canceled tick 2024-03-25T18:26:44.294 INFO:teuthology.orchestra.run.smithi115.stderr:2024-03-25T18:26:44.292+0000 7ff460cdd640 10 monclient: shutdown 2024-03-25T18:26:44.295 INFO:teuthology.orchestra.run.smithi115.stderr:2024-03-25T18:26:44.292+0000 7ff460cdd640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:26:44.295 INFO:teuthology.orchestra.run.smithi115.stderr:2024-03-25T18:26:44.292+0000 7ff460cdd640 1 -- 172.21.15.115:0/594149751 >> v1:172.21.15.115:6789/0 conn(0x7ff45c0b4510 legacy=0x7ff45c0b8730 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:26:44.295 INFO:teuthology.orchestra.run.smithi115.stderr:2024-03-25T18:26:44.292+0000 7ff460cdd640 1 -- 172.21.15.115:0/594149751 shutdown_connections 2024-03-25T18:26:44.295 INFO:teuthology.orchestra.run.smithi115.stderr:2024-03-25T18:26:44.292+0000 7ff460cdd640 1 -- 172.21.15.115:0/594149751 >> 172.21.15.115:0/594149751 conn(0x7ff45c066b00 msgr2=0x7ff45c1402c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:26:44.295 INFO:teuthology.orchestra.run.smithi115.stderr:2024-03-25T18:26:44.293+0000 7ff460cdd640 1 -- 172.21.15.115:0/594149751 shutdown_connections 2024-03-25T18:26:44.295 INFO:teuthology.orchestra.run.smithi115.stderr:2024-03-25T18:26:44.293+0000 7ff460cdd640 1 -- 172.21.15.115:0/594149751 wait complete. 2024-03-25T18:26:44.295 INFO:teuthology.orchestra.run.smithi115.stderr:2024-03-25T18:26:44.293+0000 7ff460cdd640 1 librados: shutdown 2024-03-25T18:26:44.303 DEBUG:tasks.ceph_manager:running osd.0 under valgrind with args ['cd', '/home/ubuntu/cephtest', Raw('&&'), 'sudo', 'adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'daemon-helper', 'term', 'env', 'OPENSSL_ia32cap=~0x1000000000000000', 'valgrind', '--trace-children=no', '--child-silent-after-fork=yes', '--soname-synonyms=somalloc=*tcmalloc*', '--num-callers=50', '--suppressions=/home/ubuntu/cephtest/valgrind.supp', '--xml=yes', '--xml-file=/var/log/ceph/valgrind/osd.0.log', '--time-stamp=yes', '--vgdb=yes', '--exit-on-first-error=yes', '--error-exitcode=42', '--tool=memcheck'] 2024-03-25T18:26:44.304 INFO:tasks.ceph.osd.0:Restarting daemon 2024-03-25T18:26:44.304 DEBUG:teuthology.orchestra.run.smithi001:> cd /home/ubuntu/cephtest && sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term env 'OPENSSL_ia32cap=~0x1000000000000000' valgrind --trace-children=no --child-silent-after-fork=yes '--soname-synonyms=somalloc=*tcmalloc*' --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.0.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 0 2024-03-25T18:26:44.306 INFO:tasks.ceph.osd.0:Started 2024-03-25T18:26:44.306 DEBUG:tasks.ceph_manager:running osd.1 under valgrind with args ['cd', '/home/ubuntu/cephtest', Raw('&&'), 'sudo', 'adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'daemon-helper', 'term', 'env', 'OPENSSL_ia32cap=~0x1000000000000000', 'valgrind', '--trace-children=no', '--child-silent-after-fork=yes', '--soname-synonyms=somalloc=*tcmalloc*', '--num-callers=50', '--suppressions=/home/ubuntu/cephtest/valgrind.supp', '--xml=yes', '--xml-file=/var/log/ceph/valgrind/osd.1.log', '--time-stamp=yes', '--vgdb=yes', '--exit-on-first-error=yes', '--error-exitcode=42', '--tool=memcheck'] 2024-03-25T18:26:44.306 INFO:tasks.ceph.osd.1:Restarting daemon 2024-03-25T18:26:44.307 DEBUG:teuthology.orchestra.run.smithi001:> cd /home/ubuntu/cephtest && sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term env 'OPENSSL_ia32cap=~0x1000000000000000' valgrind --trace-children=no --child-silent-after-fork=yes '--soname-synonyms=somalloc=*tcmalloc*' --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.1.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 1 2024-03-25T18:26:44.309 INFO:tasks.ceph.osd.1:Started 2024-03-25T18:26:44.309 DEBUG:tasks.ceph_manager:running osd.2 under valgrind with args ['cd', '/home/ubuntu/cephtest', Raw('&&'), 'sudo', 'adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'daemon-helper', 'term', 'env', 'OPENSSL_ia32cap=~0x1000000000000000', 'valgrind', '--trace-children=no', '--child-silent-after-fork=yes', '--soname-synonyms=somalloc=*tcmalloc*', '--num-callers=50', '--suppressions=/home/ubuntu/cephtest/valgrind.supp', '--xml=yes', '--xml-file=/var/log/ceph/valgrind/osd.2.log', '--time-stamp=yes', '--vgdb=yes', '--exit-on-first-error=yes', '--error-exitcode=42', '--tool=memcheck'] 2024-03-25T18:26:44.309 INFO:tasks.ceph.osd.2:Restarting daemon 2024-03-25T18:26:44.309 DEBUG:teuthology.orchestra.run.smithi001:> cd /home/ubuntu/cephtest && sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term env 'OPENSSL_ia32cap=~0x1000000000000000' valgrind --trace-children=no --child-silent-after-fork=yes '--soname-synonyms=somalloc=*tcmalloc*' --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.2.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 2 2024-03-25T18:26:44.311 INFO:tasks.ceph.osd.2:Started 2024-03-25T18:26:44.311 DEBUG:tasks.ceph_manager:running osd.3 under valgrind with args ['cd', '/home/ubuntu/cephtest', Raw('&&'), 'sudo', 'adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'daemon-helper', 'term', 'env', 'OPENSSL_ia32cap=~0x1000000000000000', 'valgrind', '--trace-children=no', '--child-silent-after-fork=yes', '--soname-synonyms=somalloc=*tcmalloc*', '--num-callers=50', '--suppressions=/home/ubuntu/cephtest/valgrind.supp', '--xml=yes', '--xml-file=/var/log/ceph/valgrind/osd.3.log', '--time-stamp=yes', '--vgdb=yes', '--exit-on-first-error=yes', '--error-exitcode=42', '--tool=memcheck'] 2024-03-25T18:26:44.312 INFO:tasks.ceph.osd.3:Restarting daemon 2024-03-25T18:26:44.312 DEBUG:teuthology.orchestra.run.smithi001:> cd /home/ubuntu/cephtest && sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term env 'OPENSSL_ia32cap=~0x1000000000000000' valgrind --trace-children=no --child-silent-after-fork=yes '--soname-synonyms=somalloc=*tcmalloc*' --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.3.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 3 2024-03-25T18:26:44.314 INFO:tasks.ceph.osd.3:Started 2024-03-25T18:26:44.315 DEBUG:tasks.ceph_manager:running osd.4 under valgrind with args ['cd', '/home/ubuntu/cephtest', Raw('&&'), 'sudo', 'adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'daemon-helper', 'term', 'env', 'OPENSSL_ia32cap=~0x1000000000000000', 'valgrind', '--trace-children=no', '--child-silent-after-fork=yes', '--soname-synonyms=somalloc=*tcmalloc*', '--num-callers=50', '--suppressions=/home/ubuntu/cephtest/valgrind.supp', '--xml=yes', '--xml-file=/var/log/ceph/valgrind/osd.4.log', '--time-stamp=yes', '--vgdb=yes', '--exit-on-first-error=yes', '--error-exitcode=42', '--tool=memcheck'] 2024-03-25T18:26:44.315 INFO:tasks.ceph.osd.4:Restarting daemon 2024-03-25T18:26:44.315 DEBUG:teuthology.orchestra.run.smithi115:> cd /home/ubuntu/cephtest && sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term env 'OPENSSL_ia32cap=~0x1000000000000000' valgrind --trace-children=no --child-silent-after-fork=yes '--soname-synonyms=somalloc=*tcmalloc*' --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.4.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 4 2024-03-25T18:26:44.317 INFO:tasks.ceph.osd.4:Started 2024-03-25T18:26:44.317 DEBUG:tasks.ceph_manager:running osd.5 under valgrind with args ['cd', '/home/ubuntu/cephtest', Raw('&&'), 'sudo', 'adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'daemon-helper', 'term', 'env', 'OPENSSL_ia32cap=~0x1000000000000000', 'valgrind', '--trace-children=no', '--child-silent-after-fork=yes', '--soname-synonyms=somalloc=*tcmalloc*', '--num-callers=50', '--suppressions=/home/ubuntu/cephtest/valgrind.supp', '--xml=yes', '--xml-file=/var/log/ceph/valgrind/osd.5.log', '--time-stamp=yes', '--vgdb=yes', '--exit-on-first-error=yes', '--error-exitcode=42', '--tool=memcheck'] 2024-03-25T18:26:44.317 INFO:tasks.ceph.osd.5:Restarting daemon 2024-03-25T18:26:44.317 DEBUG:teuthology.orchestra.run.smithi115:> cd /home/ubuntu/cephtest && sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term env 'OPENSSL_ia32cap=~0x1000000000000000' valgrind --trace-children=no --child-silent-after-fork=yes '--soname-synonyms=somalloc=*tcmalloc*' --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.5.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 5 2024-03-25T18:26:44.319 INFO:tasks.ceph.osd.5:Started 2024-03-25T18:26:44.319 DEBUG:tasks.ceph_manager:running osd.6 under valgrind with args ['cd', '/home/ubuntu/cephtest', Raw('&&'), 'sudo', 'adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'daemon-helper', 'term', 'env', 'OPENSSL_ia32cap=~0x1000000000000000', 'valgrind', '--trace-children=no', '--child-silent-after-fork=yes', '--soname-synonyms=somalloc=*tcmalloc*', '--num-callers=50', '--suppressions=/home/ubuntu/cephtest/valgrind.supp', '--xml=yes', '--xml-file=/var/log/ceph/valgrind/osd.6.log', '--time-stamp=yes', '--vgdb=yes', '--exit-on-first-error=yes', '--error-exitcode=42', '--tool=memcheck'] 2024-03-25T18:26:44.320 INFO:tasks.ceph.osd.6:Restarting daemon 2024-03-25T18:26:44.320 DEBUG:teuthology.orchestra.run.smithi115:> cd /home/ubuntu/cephtest && sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term env 'OPENSSL_ia32cap=~0x1000000000000000' valgrind --trace-children=no --child-silent-after-fork=yes '--soname-synonyms=somalloc=*tcmalloc*' --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.6.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 6 2024-03-25T18:26:44.321 INFO:tasks.ceph.osd.6:Started 2024-03-25T18:26:44.322 DEBUG:tasks.ceph_manager:running osd.7 under valgrind with args ['cd', '/home/ubuntu/cephtest', Raw('&&'), 'sudo', 'adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'daemon-helper', 'term', 'env', 'OPENSSL_ia32cap=~0x1000000000000000', 'valgrind', '--trace-children=no', '--child-silent-after-fork=yes', '--soname-synonyms=somalloc=*tcmalloc*', '--num-callers=50', '--suppressions=/home/ubuntu/cephtest/valgrind.supp', '--xml=yes', '--xml-file=/var/log/ceph/valgrind/osd.7.log', '--time-stamp=yes', '--vgdb=yes', '--exit-on-first-error=yes', '--error-exitcode=42', '--tool=memcheck'] 2024-03-25T18:26:44.322 INFO:tasks.ceph.osd.7:Restarting daemon 2024-03-25T18:26:44.322 DEBUG:teuthology.orchestra.run.smithi115:> cd /home/ubuntu/cephtest && sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term env 'OPENSSL_ia32cap=~0x1000000000000000' valgrind --trace-children=no --child-silent-after-fork=yes '--soname-synonyms=somalloc=*tcmalloc*' --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.7.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 7 2024-03-25T18:26:44.324 INFO:tasks.ceph.osd.7:Started 2024-03-25T18:26:44.324 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-25T18:26:44.425 INFO:tasks.ceph.mgr.y.smithi001.stderr:2024-03-25T18:26:44.424+0000 7faca7441040 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2024-03-25T18:26:44.491 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-25T18:26:44.488+0000 7f0b627f6640 10 monclient: get_monmap_and_config 2024-03-25T18:26:44.491 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-25T18:26:44.488+0000 7f0b627f6640 10 monclient: build_initial_monmap 2024-03-25T18:26:44.492 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-25T18:26:44.488+0000 7f0b627f6640 10 monclient: monmap: 2024-03-25T18:26:44.492 INFO:teuthology.orchestra.run.smithi001.stderr:epoch 0 2024-03-25T18:26:44.492 INFO:teuthology.orchestra.run.smithi001.stderr:fsid c8ea72cf-9e99-4383-9b52-6c23d365fdfc 2024-03-25T18:26:44.492 INFO:teuthology.orchestra.run.smithi001.stderr:last_changed 2024-03-25T18:26:44.490485+0000 2024-03-25T18:26:44.492 INFO:teuthology.orchestra.run.smithi001.stderr:created 2024-03-25T18:26:44.490485+0000 2024-03-25T18:26:44.492 INFO:teuthology.orchestra.run.smithi001.stderr:min_mon_release 0 (unknown) 2024-03-25T18:26:44.492 INFO:teuthology.orchestra.run.smithi001.stderr:election_strategy: 3 2024-03-25T18:26:44.492 INFO:teuthology.orchestra.run.smithi001.stderr:0: v1:172.21.15.1:6789/0 mon.noname-a 2024-03-25T18:26:44.492 INFO:teuthology.orchestra.run.smithi001.stderr:1: v1:172.21.15.115:6789/0 mon.noname-c 2024-03-25T18:26:44.492 INFO:teuthology.orchestra.run.smithi001.stderr:2: v1:172.21.15.1:6790/0 mon.noname-b 2024-03-25T18:26:44.492 INFO:teuthology.orchestra.run.smithi001.stderr: 2024-03-25T18:26:44.493 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-25T18:26:44.491+0000 7f0b627f6640 1 Processor -- start 2024-03-25T18:26:44.493 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-25T18:26:44.491+0000 7f0b627f6640 1 -- start start 2024-03-25T18:26:44.493 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-25T18:26:44.491+0000 7f0b627f6640 10 monclient: init 2024-03-25T18:26:44.495 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-25T18:26:44.492+0000 7f0b627f6640 10 monclient: _reopen_session rank -1 2024-03-25T18:26:44.495 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-25T18:26:44.492+0000 7f0b627f6640 10 monclient: _add_conns ranks=[0,2,1] 2024-03-25T18:26:44.495 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-25T18:26:44.492+0000 7f0b627f6640 10 monclient(hunting): picked mon.noname-a con 0x7f0b5c0b1140 addr v1:172.21.15.1:6789/0 2024-03-25T18:26:44.495 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-25T18:26:44.492+0000 7f0b627f6640 10 monclient(hunting): picked mon.noname-b con 0x7f0b5c0a9db0 addr v1:172.21.15.1:6790/0 2024-03-25T18:26:44.495 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-25T18:26:44.492+0000 7f0b627f6640 10 monclient(hunting): picked mon.noname-c con 0x7f0b5c0accc0 addr v1:172.21.15.115:6789/0 2024-03-25T18:26:44.495 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-25T18:26:44.492+0000 7f0b627f6640 1 -- --> v1:172.21.15.1:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0b5c0ac070 con 0x7f0b5c0b1140 2024-03-25T18:26:44.495 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-25T18:26:44.492+0000 7f0b627f6640 1 -- --> v1:172.21.15.115:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0b5c0ac2e0 con 0x7f0b5c0accc0 2024-03-25T18:26:44.495 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-25T18:26:44.492+0000 7f0b627f6640 1 -- --> v1:172.21.15.1:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0b5c0ae120 con 0x7f0b5c0a9db0 2024-03-25T18:26:44.495 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-25T18:26:44.492+0000 7f0b627f6640 10 monclient(hunting): _renew_subs 2024-03-25T18:26:44.495 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-25T18:26:44.492+0000 7f0b627f6640 10 monclient(hunting): authenticate will time out at 7484.963037s 2024-03-25T18:26:44.500 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-25T18:26:44.498+0000 7f0b5bfff640 1 --1- >> v1:172.21.15.1:6789/0 conn(0x7f0b5c0b1140 0x7f0b5c13b390 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.1:6789/0 says I am v1:172.21.15.1:56662/0 (socket says 172.21.15.1:56662) 2024-03-25T18:26:44.500 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-25T18:26:44.498+0000 7f0b5bfff640 1 -- 172.21.15.1:0/3370866479 learned_addr learned my addr 172.21.15.1:0/3370866479 (peer_addr_for_me v1:172.21.15.1:0/0) 2024-03-25T18:26:44.511 INFO:tasks.ceph.mgr.y.smithi001.stderr:2024-03-25T18:26:44.510+0000 7faca7441040 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2024-03-25T18:26:44.603 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-25T18:26:44.601+0000 7f0b5affd640 1 -- 172.21.15.1:0/3370866479 <== mon.1 v1:172.21.15.115:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 478720307 0 0) 0x7f0b5c0ac2e0 con 0x7f0b5c0accc0 2024-03-25T18:26:44.603 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-25T18:26:44.601+0000 7f0b5affd640 10 monclient(hunting): _negotiate 2024-03-25T18:26:44.603 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-25T18:26:44.601+0000 7f0b5affd640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:26:44.603 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-25T18:26:44.601+0000 7f0b5affd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:26:44.604 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-25T18:26:44.601+0000 7f0b5affd640 10 monclient(hunting): my global_id is 4156 2024-03-25T18:26:44.604 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-25T18:26:44.601+0000 7f0b5affd640 1 -- 172.21.15.1:0/3370866479 --> v1:172.21.15.115:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f0b48003520 con 0x7f0b5c0accc0 2024-03-25T18:26:44.630 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-25T18:26:44.629+0000 7f0b5affd640 1 -- 172.21.15.1:0/3370866479 <== mon.1 v1:172.21.15.115:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 337809557 0 0) 0x7f0b48003520 con 0x7f0b5c0accc0 2024-03-25T18:26:44.631 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-25T18:26:44.629+0000 7f0b5affd640 1 -- 172.21.15.1:0/3370866479 >> v1:172.21.15.1:6790/0 conn(0x7f0b5c0a9db0 legacy=0x7f0b5c141b80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:26:44.631 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-25T18:26:44.629+0000 7f0b5affd640 1 -- 172.21.15.1:0/3370866479 >> v1:172.21.15.1:6789/0 conn(0x7f0b5c0b1140 legacy=0x7f0b5c13b390 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:26:44.631 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-25T18:26:44.629+0000 7f0b5affd640 10 monclient: _finish_hunting 0 2024-03-25T18:26:44.631 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-25T18:26:44.629+0000 7f0b5affd640 1 monclient: found mon.noname-c 2024-03-25T18:26:44.631 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-25T18:26:44.629+0000 7f0b5affd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:26:44.631 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-25T18:26:44.629+0000 7f0b5affd640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.115:6789/0 2024-03-25T18:26:44.631 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-25T18:26:44.629+0000 7f0b5affd640 1 -- 172.21.15.1:0/3370866479 --> v1:172.21.15.115:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0b5c13baa0 con 0x7f0b5c0accc0 2024-03-25T18:26:44.631 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-25T18:26:44.629+0000 7f0b5affd640 10 monclient: _finish_auth 0 2024-03-25T18:26:44.631 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-25T18:26:44.629+0000 7f0b5affd640 10 monclient: _check_auth_tickets 2024-03-25T18:26:44.631 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-25T18:26:44.629+0000 7f0b5affd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:26:44.631 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-25T18:26:44.629+0000 7f0b5affd640 1 -- 172.21.15.1:0/3370866479 <== mon.1 v1:172.21.15.115:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1586965025 0 0) 0x7f0b44001a10 con 0x7f0b5c0accc0 2024-03-25T18:26:44.631 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-25T18:26:44.629+0000 7f0b5affd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:26:44.631 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-25T18:26:44.629+0000 7f0b5affd640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-03-25T18:26:44.632 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-25T18:26:44.629+0000 7f0b5affd640 10 monclient: dump: 2024-03-25T18:26:44.632 INFO:teuthology.orchestra.run.smithi001.stderr:epoch 1 2024-03-25T18:26:44.632 INFO:teuthology.orchestra.run.smithi001.stderr:fsid c8ea72cf-9e99-4383-9b52-6c23d365fdfc 2024-03-25T18:26:44.632 INFO:teuthology.orchestra.run.smithi001.stderr:last_changed 2024-03-25T18:25:50.510519+0000 2024-03-25T18:26:44.632 INFO:teuthology.orchestra.run.smithi001.stderr:created 2024-03-25T18:25:50.510519+0000 2024-03-25T18:26:44.632 INFO:teuthology.orchestra.run.smithi001.stderr:min_mon_release 19 (squid) 2024-03-25T18:26:44.632 INFO:teuthology.orchestra.run.smithi001.stderr:election_strategy: 3 2024-03-25T18:26:44.632 INFO:teuthology.orchestra.run.smithi001.stderr:0: v1:172.21.15.1:6789/0 mon.a 2024-03-25T18:26:44.632 INFO:teuthology.orchestra.run.smithi001.stderr:1: v1:172.21.15.115:6789/0 mon.b 2024-03-25T18:26:44.632 INFO:teuthology.orchestra.run.smithi001.stderr:2: v1:172.21.15.1:6790/0 mon.c 2024-03-25T18:26:44.632 INFO:teuthology.orchestra.run.smithi001.stderr: 2024-03-25T18:26:44.632 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-25T18:26:44.629+0000 7f0b627f6640 5 monclient: authenticate success, global_id 4156 2024-03-25T18:26:44.632 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-25T18:26:44.629+0000 7f0b627f6640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T18:26:44.683 INFO:tasks.ceph.mgr.y.smithi001.stderr:2024-03-25T18:26:44.682+0000 7faca7441040 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2024-03-25T18:26:44.935 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-25T18:26:44.932+0000 7f0b5affd640 1 -- 172.21.15.1:0/3370866479 <== mon.1 v1:172.21.15.115:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f0b44001b70 con 0x7f0b5c0accc0 2024-03-25T18:26:44.935 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-25T18:26:44.932+0000 7f0b5affd640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:26:44.935 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-25T18:26:44.932+0000 7f0b5affd640 1 -- 172.21.15.1:0/3370866479 <== mon.1 v1:172.21.15.115:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1586965025 0 0) 0x7f0b44001fc0 con 0x7f0b5c0accc0 2024-03-25T18:26:44.935 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-25T18:26:44.932+0000 7f0b5affd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:26:44.935 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-25T18:26:44.932+0000 7f0b5affd640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-25T18:26:44.935 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-25T18:26:44.932+0000 7f0b5affd640 10 monclient: dump: 2024-03-25T18:26:44.935 INFO:teuthology.orchestra.run.smithi001.stderr:epoch 1 2024-03-25T18:26:44.935 INFO:teuthology.orchestra.run.smithi001.stderr:fsid c8ea72cf-9e99-4383-9b52-6c23d365fdfc 2024-03-25T18:26:44.935 INFO:teuthology.orchestra.run.smithi001.stderr:last_changed 2024-03-25T18:25:50.510519+0000 2024-03-25T18:26:44.935 INFO:teuthology.orchestra.run.smithi001.stderr:created 2024-03-25T18:25:50.510519+0000 2024-03-25T18:26:44.936 INFO:teuthology.orchestra.run.smithi001.stderr:min_mon_release 19 (squid) 2024-03-25T18:26:44.936 INFO:teuthology.orchestra.run.smithi001.stderr:election_strategy: 3 2024-03-25T18:26:44.936 INFO:teuthology.orchestra.run.smithi001.stderr:0: v1:172.21.15.1:6789/0 mon.a 2024-03-25T18:26:44.936 INFO:teuthology.orchestra.run.smithi001.stderr:1: v1:172.21.15.115:6789/0 mon.b 2024-03-25T18:26:44.936 INFO:teuthology.orchestra.run.smithi001.stderr:2: v1:172.21.15.1:6790/0 mon.c 2024-03-25T18:26:44.936 INFO:teuthology.orchestra.run.smithi001.stderr: 2024-03-25T18:26:44.936 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-25T18:26:44.932+0000 7f0b627f6640 10 monclient: get_monmap_and_config success 2024-03-25T18:26:44.936 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-25T18:26:44.932+0000 7f0b627f6640 4 set_mon_vals no callback set 2024-03-25T18:26:44.936 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-25T18:26:44.933+0000 7f0b627f6640 10 monclient: shutdown 2024-03-25T18:26:44.936 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-25T18:26:44.933+0000 7f0b627f6640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:26:44.936 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-25T18:26:44.933+0000 7f0b627f6640 1 -- 172.21.15.1:0/3370866479 >> v1:172.21.15.115:6789/0 conn(0x7f0b5c0accc0 legacy=0x7f0b5c14be90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:26:44.936 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-25T18:26:44.933+0000 7f0b627f6640 1 -- 172.21.15.1:0/3370866479 shutdown_connections 2024-03-25T18:26:44.936 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-25T18:26:44.933+0000 7f0b627f6640 1 -- 172.21.15.1:0/3370866479 >> 172.21.15.1:0/3370866479 conn(0x7f0b5c0b0620 msgr2=0x7f0b5c0b0a30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:26:44.936 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-25T18:26:44.933+0000 7f0b627f6640 1 -- 172.21.15.1:0/3370866479 shutdown_connections 2024-03-25T18:26:44.937 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-25T18:26:44.933+0000 7f0b627f6640 1 -- 172.21.15.1:0/3370866479 wait complete. 2024-03-25T18:26:44.938 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-25T18:26:44.934+0000 7f0b627f6640 10 monclient: build_initial_monmap 2024-03-25T18:26:44.939 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-25T18:26:44.934+0000 7f0b627f6640 10 monclient: monmap: 2024-03-25T18:26:44.939 INFO:teuthology.orchestra.run.smithi001.stderr:epoch 0 2024-03-25T18:26:44.939 INFO:teuthology.orchestra.run.smithi001.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T18:26:44.939 INFO:teuthology.orchestra.run.smithi001.stderr:last_changed 0.000000 2024-03-25T18:26:44.939 INFO:teuthology.orchestra.run.smithi001.stderr:created 0.000000 2024-03-25T18:26:44.939 INFO:teuthology.orchestra.run.smithi001.stderr:min_mon_release 0 (unknown) 2024-03-25T18:26:44.939 INFO:teuthology.orchestra.run.smithi001.stderr:election_strategy: 1 2024-03-25T18:26:44.939 INFO:teuthology.orchestra.run.smithi001.stderr:0: v1:172.21.15.1:6789/0 mon.noname-a 2024-03-25T18:26:44.939 INFO:teuthology.orchestra.run.smithi001.stderr:1: v1:172.21.15.115:6789/0 mon.noname-b 2024-03-25T18:26:44.939 INFO:teuthology.orchestra.run.smithi001.stderr:2: v1:172.21.15.1:6790/0 mon.noname-c 2024-03-25T18:26:44.939 INFO:teuthology.orchestra.run.smithi001.stderr: 2024-03-25T18:26:44.939 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-25T18:26:44.934+0000 7f0b627f6640 1 librados: starting msgr at 2024-03-25T18:26:44.939 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-25T18:26:44.934+0000 7f0b627f6640 1 librados: starting objecter 2024-03-25T18:26:44.939 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-25T18:26:44.935+0000 7f0b627f6640 1 Processor -- start 2024-03-25T18:26:44.940 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-25T18:26:44.935+0000 7f0b627f6640 1 -- start start 2024-03-25T18:26:44.940 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-25T18:26:44.935+0000 7f0b627f6640 1 librados: setting wanted keys 2024-03-25T18:26:44.940 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-25T18:26:44.935+0000 7f0b627f6640 1 librados: calling monclient init 2024-03-25T18:26:44.940 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-25T18:26:44.935+0000 7f0b627f6640 10 monclient: init 2024-03-25T18:26:44.940 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-25T18:26:44.936+0000 7f0b627f6640 10 monclient: _reopen_session rank -1 2024-03-25T18:26:44.940 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-25T18:26:44.936+0000 7f0b627f6640 10 monclient: _add_conns ranks=[0,1,2] 2024-03-25T18:26:44.940 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-25T18:26:44.936+0000 7f0b627f6640 10 monclient(hunting): picked mon.noname-a con 0x7f0b5c0a9db0 addr v1:172.21.15.1:6789/0 2024-03-25T18:26:44.940 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-25T18:26:44.936+0000 7f0b627f6640 10 monclient(hunting): picked mon.noname-b con 0x7f0b5c0accc0 addr v1:172.21.15.115:6789/0 2024-03-25T18:26:44.940 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-25T18:26:44.936+0000 7f0b627f6640 10 monclient(hunting): picked mon.noname-c con 0x7f0b5c0b1140 addr v1:172.21.15.1:6790/0 2024-03-25T18:26:44.940 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-25T18:26:44.936+0000 7f0b627f6640 1 -- --> v1:172.21.15.1:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0b5c06e570 con 0x7f0b5c0a9db0 2024-03-25T18:26:44.940 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-25T18:26:44.936+0000 7f0b627f6640 1 -- --> v1:172.21.15.115:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0b5c13e640 con 0x7f0b5c0accc0 2024-03-25T18:26:44.940 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-25T18:26:44.937+0000 7f0b627f6640 1 -- --> v1:172.21.15.1:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0b5c13dc30 con 0x7f0b5c0b1140 2024-03-25T18:26:44.940 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-25T18:26:44.937+0000 7f0b627f6640 10 monclient(hunting): _renew_subs 2024-03-25T18:26:44.940 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-25T18:26:44.937+0000 7f0b627f6640 10 monclient(hunting): authenticate will time out at 7485.407730s 2024-03-25T18:26:44.941 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-25T18:26:44.938+0000 7f0b5b7fe640 1 --1- >> v1:172.21.15.115:6789/0 conn(0x7f0b5c0accc0 0x7f0b5c1312f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.115:6789/0 says I am v1:172.21.15.1:44506/0 (socket says 172.21.15.1:44506) 2024-03-25T18:26:44.941 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-25T18:26:44.938+0000 7f0b5b7fe640 1 -- 172.21.15.1:0/3547632143 learned_addr learned my addr 172.21.15.1:0/3547632143 (peer_addr_for_me v1:172.21.15.1:0/0) 2024-03-25T18:26:44.941 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-25T18:26:44.939+0000 7f0b58ff9640 10 client.?.objecter ms_handle_connect 0x7f0b5c0accc0 2024-03-25T18:26:44.941 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-25T18:26:44.939+0000 7f0b58ff9640 10 client.?.objecter resend_mon_ops 2024-03-25T18:26:44.977 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-25T18:26:44.976+0000 7f0b58ff9640 10 client.?.objecter ms_handle_connect 0x7f0b5c0b1140 2024-03-25T18:26:44.978 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-25T18:26:44.976+0000 7f0b58ff9640 10 client.?.objecter resend_mon_ops 2024-03-25T18:26:44.990 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-25T18:26:44.989+0000 7f0b58ff9640 10 client.?.objecter ms_handle_connect 0x7f0b5c0a9db0 2024-03-25T18:26:44.990 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-25T18:26:44.989+0000 7f0b58ff9640 10 client.?.objecter resend_mon_ops 2024-03-25T18:26:44.996 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-25T18:26:44.994+0000 7f0b58ff9640 1 -- 172.21.15.1:0/3547632143 <== mon.2 v1:172.21.15.1:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1957135858 0 0) 0x7f0b5c13dc30 con 0x7f0b5c0b1140 2024-03-25T18:26:44.996 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-25T18:26:44.994+0000 7f0b58ff9640 10 monclient(hunting): _negotiate 2024-03-25T18:26:44.996 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-25T18:26:44.994+0000 7f0b58ff9640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:26:44.996 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-25T18:26:44.994+0000 7f0b58ff9640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:26:44.996 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-25T18:26:44.994+0000 7f0b58ff9640 10 monclient(hunting): my global_id is 4166 2024-03-25T18:26:44.996 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-25T18:26:44.994+0000 7f0b58ff9640 1 -- 172.21.15.1:0/3547632143 --> v1:172.21.15.1:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f0b3c004eb0 con 0x7f0b5c0b1140 2024-03-25T18:26:44.997 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-25T18:26:44.996+0000 7f0b58ff9640 1 -- 172.21.15.1:0/3547632143 <== mon.0 v1:172.21.15.1:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 370464055 0 0) 0x7f0b5c06e570 con 0x7f0b5c0a9db0 2024-03-25T18:26:44.997 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-25T18:26:44.996+0000 7f0b58ff9640 10 monclient(hunting): _negotiate 2024-03-25T18:26:44.997 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-25T18:26:44.996+0000 7f0b58ff9640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:26:44.997 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-25T18:26:44.996+0000 7f0b58ff9640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:26:44.997 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-25T18:26:44.996+0000 7f0b58ff9640 10 monclient(hunting): my global_id is 4167 2024-03-25T18:26:44.997 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-25T18:26:44.996+0000 7f0b58ff9640 1 -- 172.21.15.1:0/3547632143 --> v1:172.21.15.1:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f0b5c13dc30 con 0x7f0b5c0a9db0 2024-03-25T18:26:45.007 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-25T18:26:45.005+0000 7f0b58ff9640 1 -- 172.21.15.1:0/3547632143 <== mon.2 v1:172.21.15.1:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3355572172 0 0) 0x7f0b3c004eb0 con 0x7f0b5c0b1140 2024-03-25T18:26:45.007 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-25T18:26:45.005+0000 7f0b58ff9640 1 -- 172.21.15.1:0/3547632143 --> v1:172.21.15.1:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f0b5c06e570 con 0x7f0b5c0b1140 2024-03-25T18:26:45.007 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-25T18:26:45.005+0000 7f0b58ff9640 1 -- 172.21.15.1:0/3547632143 <== mon.2 v1:172.21.15.1:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1586965025 0 0) 0x7f0b44001e80 con 0x7f0b5c0b1140 2024-03-25T18:26:45.007 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-25T18:26:45.005+0000 7f0b58ff9640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:26:45.007 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-25T18:26:45.005+0000 7f0b58ff9640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-03-25T18:26:45.007 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-25T18:26:45.005+0000 7f0b58ff9640 10 monclient(hunting): dump: 2024-03-25T18:26:45.007 INFO:teuthology.orchestra.run.smithi001.stderr:epoch 1 2024-03-25T18:26:45.007 INFO:teuthology.orchestra.run.smithi001.stderr:fsid c8ea72cf-9e99-4383-9b52-6c23d365fdfc 2024-03-25T18:26:45.008 INFO:teuthology.orchestra.run.smithi001.stderr:last_changed 2024-03-25T18:25:50.510519+0000 2024-03-25T18:26:45.008 INFO:teuthology.orchestra.run.smithi001.stderr:created 2024-03-25T18:25:50.510519+0000 2024-03-25T18:26:45.008 INFO:teuthology.orchestra.run.smithi001.stderr:min_mon_release 19 (squid) 2024-03-25T18:26:45.008 INFO:teuthology.orchestra.run.smithi001.stderr:election_strategy: 3 2024-03-25T18:26:45.008 INFO:teuthology.orchestra.run.smithi001.stderr:0: v1:172.21.15.1:6789/0 mon.a 2024-03-25T18:26:45.008 INFO:teuthology.orchestra.run.smithi001.stderr:1: v1:172.21.15.115:6789/0 mon.b 2024-03-25T18:26:45.008 INFO:teuthology.orchestra.run.smithi001.stderr:2: v1:172.21.15.1:6790/0 mon.c 2024-03-25T18:26:45.008 INFO:teuthology.orchestra.run.smithi001.stderr: 2024-03-25T18:26:45.008 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-25T18:26:45.006+0000 7f0b58ff9640 10 monclient(hunting): _finish_auth 0 2024-03-25T18:26:45.010 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-25T18:26:45.009+0000 7f0b58ff9640 1 -- 172.21.15.1:0/3547632143 <== mon.0 v1:172.21.15.1:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2085301149 0 0) 0x7f0b5c13dc30 con 0x7f0b5c0a9db0 2024-03-25T18:26:45.011 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-25T18:26:45.009+0000 7f0b58ff9640 1 -- 172.21.15.1:0/3547632143 --> v1:172.21.15.1:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f0b3c004eb0 con 0x7f0b5c0a9db0 2024-03-25T18:26:45.011 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-25T18:26:45.009+0000 7f0b58ff9640 1 -- 172.21.15.1:0/3547632143 <== mon.0 v1:172.21.15.1:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1586965025 0 0) 0x7f0b540018b0 con 0x7f0b5c0a9db0 2024-03-25T18:26:45.011 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-25T18:26:45.009+0000 7f0b58ff9640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:26:45.011 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-25T18:26:45.009+0000 7f0b58ff9640 10 monclient(hunting): got monmap 1 from mon.a (according to old e1) 2024-03-25T18:26:45.011 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-25T18:26:45.009+0000 7f0b58ff9640 10 monclient(hunting): dump: 2024-03-25T18:26:45.011 INFO:teuthology.orchestra.run.smithi001.stderr:epoch 1 2024-03-25T18:26:45.011 INFO:teuthology.orchestra.run.smithi001.stderr:fsid c8ea72cf-9e99-4383-9b52-6c23d365fdfc 2024-03-25T18:26:45.011 INFO:teuthology.orchestra.run.smithi001.stderr:last_changed 2024-03-25T18:25:50.510519+0000 2024-03-25T18:26:45.011 INFO:teuthology.orchestra.run.smithi001.stderr:created 2024-03-25T18:25:50.510519+0000 2024-03-25T18:26:45.011 INFO:teuthology.orchestra.run.smithi001.stderr:min_mon_release 19 (squid) 2024-03-25T18:26:45.011 INFO:teuthology.orchestra.run.smithi001.stderr:election_strategy: 3 2024-03-25T18:26:45.011 INFO:teuthology.orchestra.run.smithi001.stderr:0: v1:172.21.15.1:6789/0 mon.a 2024-03-25T18:26:45.011 INFO:teuthology.orchestra.run.smithi001.stderr:1: v1:172.21.15.115:6789/0 mon.b 2024-03-25T18:26:45.011 INFO:teuthology.orchestra.run.smithi001.stderr:2: v1:172.21.15.1:6790/0 mon.c 2024-03-25T18:26:45.012 INFO:teuthology.orchestra.run.smithi001.stderr: 2024-03-25T18:26:45.018 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-25T18:26:45.016+0000 7f0b58ff9640 1 -- 172.21.15.1:0/3547632143 <== mon.0 v1:172.21.15.1:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 2433825043 0 0) 0x7f0b3c004eb0 con 0x7f0b5c0a9db0 2024-03-25T18:26:45.018 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-25T18:26:45.017+0000 7f0b58ff9640 1 -- 172.21.15.1:0/3547632143 >> v1:172.21.15.1:6790/0 conn(0x7f0b5c0b1140 legacy=0x7f0b5c1e2bc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:26:45.018 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-25T18:26:45.017+0000 7f0b58ff9640 1 -- 172.21.15.1:0/3547632143 >> v1:172.21.15.115:6789/0 conn(0x7f0b5c0accc0 legacy=0x7f0b5c1312f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:26:45.018 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-25T18:26:45.017+0000 7f0b58ff9640 10 monclient: _finish_hunting 0 2024-03-25T18:26:45.019 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-25T18:26:45.017+0000 7f0b58ff9640 1 monclient: found mon.a 2024-03-25T18:26:45.019 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-25T18:26:45.017+0000 7f0b58ff9640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:26:45.019 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-25T18:26:45.017+0000 7f0b58ff9640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.1:6789/0 2024-03-25T18:26:45.019 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-25T18:26:45.017+0000 7f0b58ff9640 1 -- 172.21.15.1:0/3547632143 --> v1:172.21.15.1:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0b5c13eb60 con 0x7f0b5c0a9db0 2024-03-25T18:26:45.019 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-25T18:26:45.017+0000 7f0b58ff9640 10 monclient: _finish_auth 0 2024-03-25T18:26:45.019 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-25T18:26:45.017+0000 7f0b58ff9640 10 monclient: _check_auth_tickets 2024-03-25T18:26:45.019 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-25T18:26:45.017+0000 7f0b58ff9640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:26:45.019 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-25T18:26:45.017+0000 7f0b627f6640 5 monclient: authenticate success, global_id 4167 2024-03-25T18:26:45.019 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-25T18:26:45.017+0000 7f0b627f6640 10 monclient: _renew_subs 2024-03-25T18:26:45.019 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-25T18:26:45.017+0000 7f0b627f6640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.1:6789/0 2024-03-25T18:26:45.019 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-25T18:26:45.017+0000 7f0b627f6640 1 -- 172.21.15.1:0/3547632143 --> v1:172.21.15.1:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0b5c134b80 con 0x7f0b5c0a9db0 2024-03-25T18:26:45.020 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-25T18:26:45.017+0000 7f0b627f6640 10 client.4167.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T18:26:45.020 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-25T18:26:45.017+0000 7f0b627f6640 10 monclient: _renew_subs 2024-03-25T18:26:45.020 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-25T18:26:45.017+0000 7f0b627f6640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.1:6789/0 2024-03-25T18:26:45.020 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-25T18:26:45.017+0000 7f0b627f6640 1 -- 172.21.15.1:0/3547632143 --> v1:172.21.15.1:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f0b5c134e80 con 0x7f0b5c0a9db0 2024-03-25T18:26:45.020 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-25T18:26:45.017+0000 7f0b627f6640 1 librados: init done 2024-03-25T18:26:45.020 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-25T18:26:45.018+0000 7f0b627f6640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T18:26:45.020 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-25T18:26:45.018+0000 7f0b627f6640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:26:45.020 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-25T18:26:45.018+0000 7f0b627f6640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.1:6789/0 2024-03-25T18:26:45.020 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-25T18:26:45.018+0000 7f0b627f6640 1 -- 172.21.15.1:0/3547632143 --> v1:172.21.15.1:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0b5c137bc0 con 0x7f0b5c0a9db0 2024-03-25T18:26:45.101 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-25T18:26:45.100+0000 7f0b58ff9640 1 -- 172.21.15.1:0/3547632143 <== mon.0 v1:172.21.15.1:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f0b54001d40 con 0x7f0b5c0a9db0 2024-03-25T18:26:45.101 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-25T18:26:45.100+0000 7f0b58ff9640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:26:45.102 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-25T18:26:45.100+0000 7f0b58ff9640 1 -- 172.21.15.1:0/3547632143 <== mon.0 v1:172.21.15.1:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1586965025 0 0) 0x7f0b540018b0 con 0x7f0b5c0a9db0 2024-03-25T18:26:45.102 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-25T18:26:45.100+0000 7f0b58ff9640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:26:45.102 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-25T18:26:45.100+0000 7f0b58ff9640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-25T18:26:45.102 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-25T18:26:45.100+0000 7f0b58ff9640 10 monclient: dump: 2024-03-25T18:26:45.102 INFO:teuthology.orchestra.run.smithi001.stderr:epoch 1 2024-03-25T18:26:45.102 INFO:teuthology.orchestra.run.smithi001.stderr:fsid c8ea72cf-9e99-4383-9b52-6c23d365fdfc 2024-03-25T18:26:45.102 INFO:teuthology.orchestra.run.smithi001.stderr:last_changed 2024-03-25T18:25:50.510519+0000 2024-03-25T18:26:45.102 INFO:teuthology.orchestra.run.smithi001.stderr:created 2024-03-25T18:25:50.510519+0000 2024-03-25T18:26:45.102 INFO:teuthology.orchestra.run.smithi001.stderr:min_mon_release 19 (squid) 2024-03-25T18:26:45.102 INFO:teuthology.orchestra.run.smithi001.stderr:election_strategy: 3 2024-03-25T18:26:45.102 INFO:teuthology.orchestra.run.smithi001.stderr:0: v1:172.21.15.1:6789/0 mon.a 2024-03-25T18:26:45.102 INFO:teuthology.orchestra.run.smithi001.stderr:1: v1:172.21.15.115:6789/0 mon.b 2024-03-25T18:26:45.102 INFO:teuthology.orchestra.run.smithi001.stderr:2: v1:172.21.15.1:6790/0 mon.c 2024-03-25T18:26:45.103 INFO:teuthology.orchestra.run.smithi001.stderr: 2024-03-25T18:26:45.103 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-25T18:26:45.100+0000 7f0b5a7fc640 4 set_mon_vals no callback set 2024-03-25T18:26:45.103 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-25T18:26:45.100+0000 7f0b58ff9640 1 -- 172.21.15.1:0/3547632143 <== mon.0 v1:172.21.15.1:6789/0 7 ==== mgrmap(e 2) v1 ==== 44919+0+0 (unknown 668617283 0 0) 0x7f0b540048e0 con 0x7f0b5c0a9db0 2024-03-25T18:26:45.133 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-25T18:26:45.131+0000 7f0b58ff9640 1 -- 172.21.15.1:0/3547632143 <== mon.0 v1:172.21.15.1:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 996254230 0 0) 0x7f0b540153a0 con 0x7f0b5c0a9db0 2024-03-25T18:26:45.133 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-25T18:26:45.131+0000 7f0b58ff9640 10 client.4167.objecter ms_dispatch 0x7f0b5c000b90 osd_map(10..10 src has 1..10) v4 2024-03-25T18:26:45.133 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-25T18:26:45.131+0000 7f0b58ff9640 3 client.4167.objecter handle_osd_map got epochs [10,10] > 0 2024-03-25T18:26:45.133 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-25T18:26:45.131+0000 7f0b58ff9640 3 client.4167.objecter handle_osd_map decoding full epoch 10 2024-03-25T18:26:45.133 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-25T18:26:45.131+0000 7f0b58ff9640 20 client.4167.objecter dump_active .. 0 homeless 2024-03-25T18:26:45.218 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-25T18:26:45.216+0000 7f0b58ff9640 1 -- 172.21.15.1:0/3547632143 <== mon.0 v1:172.21.15.1:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+73141 (unknown 1092875540 0 2031569661) 0x7f0b54014840 con 0x7f0b5c0a9db0 2024-03-25T18:26:45.218 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-25T18:26:45.216+0000 7f0b58ff9640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:26:45.218 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-25T18:26:45.216+0000 7f0b58ff9640 10 monclient: _finish_command 1 = system:0 2024-03-25T18:26:45.346 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-25T18:26:45.344+0000 7f0b627f6640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:26:45.346 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-25T18:26:45.344+0000 7f0b627f6640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:26:45.347 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-25T18:26:45.344+0000 7f0b627f6640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.1:6789/0 2024-03-25T18:26:45.347 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-25T18:26:45.344+0000 7f0b627f6640 1 -- 172.21.15.1:0/3547632143 --> v1:172.21.15.1:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f0b5c04f400 con 0x7f0b5c0a9db0 2024-03-25T18:26:45.423 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-25T18:26:45.421+0000 7f0b58ff9640 1 -- 172.21.15.1:0/3547632143 <== mon.0 v1:172.21.15.1:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 4092213348) 0x7f0b540054e0 con 0x7f0b5c0a9db0 2024-03-25T18:26:45.423 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-25T18:26:45.421+0000 7f0b58ff9640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:26:45.424 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-25T18:26:45.421+0000 7f0b58ff9640 10 monclient: _finish_command 2 = system:0 2024-03-25T18:26:45.424 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-25T18:26:45.424 INFO:teuthology.orchestra.run.smithi001.stdout:{"epoch":10,"fsid":"c8ea72cf-9e99-4383-9b52-6c23d365fdfc","created":"2024-03-25T18:26:38.856638+0000","modified":"2024-03-25T18:26:44.224208+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T18:26:44.224208+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"b9f34922-3a84-49be-8426-807c1eceaefc","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"42bf306f-55e3-4634-9874-8b4b73ac6644","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"292cb23a-18a9-4207-a7cc-4d5727feaec0","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"45e0e9ab-898d-4673-9590-b4117bc5e84a","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"831a5a09-73ab-4a61-9872-088e76bdbc66","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"88cd903d-3b55-4840-9396-6e3cfe005282","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"89db7355-bc9d-46f4-8ab0-e8d9e744e026","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"cac47bee-0914-4f08-896f-348838882866","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T18:26:45.426 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-25T18:26:45.423+0000 7f0b627f6640 10 librados: watch_flush enter 2024-03-25T18:26:45.426 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-25T18:26:45.423+0000 7f0b627f6640 10 librados: watch_flush exit 2024-03-25T18:26:45.426 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-25T18:26:45.423+0000 7f0b627f6640 20 client.4167.objecter shutdown clearing up homeless session... 2024-03-25T18:26:45.426 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-25T18:26:45.423+0000 7f0b627f6640 10 client.4167.objecter successfully canceled tick 2024-03-25T18:26:45.427 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-25T18:26:45.423+0000 7f0b627f6640 10 monclient: shutdown 2024-03-25T18:26:45.427 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-25T18:26:45.423+0000 7f0b627f6640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:26:45.427 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-25T18:26:45.423+0000 7f0b627f6640 1 -- 172.21.15.1:0/3547632143 >> v1:172.21.15.1:6789/0 conn(0x7f0b5c0a9db0 legacy=0x7f0b5c134470 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:26:45.427 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-25T18:26:45.423+0000 7f0b627f6640 1 -- 172.21.15.1:0/3547632143 shutdown_connections 2024-03-25T18:26:45.427 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-25T18:26:45.423+0000 7f0b627f6640 1 -- 172.21.15.1:0/3547632143 >> 172.21.15.1:0/3547632143 conn(0x7f0b5c0b0620 msgr2=0x7f0b5c0ab160 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:26:45.427 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-25T18:26:45.423+0000 7f0b627f6640 1 -- 172.21.15.1:0/3547632143 shutdown_connections 2024-03-25T18:26:45.427 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-25T18:26:45.423+0000 7f0b627f6640 1 -- 172.21.15.1:0/3547632143 wait complete. 2024-03-25T18:26:45.427 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-25T18:26:45.423+0000 7f0b627f6640 1 librados: shutdown 2024-03-25T18:26:45.447 INFO:tasks.ceph.ceph_manager.ceph:[] 2024-03-25T18:26:45.447 INFO:tasks.ceph:Waiting for OSDs to come up 2024-03-25T18:26:46.254 DEBUG:teuthology.orchestra.run.smithi001:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T18:26:46.399 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:26:46.397+0000 7f02af5f4640 10 monclient: get_monmap_and_config 2024-03-25T18:26:46.399 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:26:46.397+0000 7f02af5f4640 10 monclient: build_initial_monmap 2024-03-25T18:26:46.400 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:26:46.397+0000 7f02af5f4640 10 monclient: monmap: 2024-03-25T18:26:46.400 INFO:teuthology.misc.health.smithi001.stderr:epoch 0 2024-03-25T18:26:46.400 INFO:teuthology.misc.health.smithi001.stderr:fsid c8ea72cf-9e99-4383-9b52-6c23d365fdfc 2024-03-25T18:26:46.400 INFO:teuthology.misc.health.smithi001.stderr:last_changed 2024-03-25T18:26:46.398626+0000 2024-03-25T18:26:46.400 INFO:teuthology.misc.health.smithi001.stderr:created 2024-03-25T18:26:46.398626+0000 2024-03-25T18:26:46.400 INFO:teuthology.misc.health.smithi001.stderr:min_mon_release 0 (unknown) 2024-03-25T18:26:46.400 INFO:teuthology.misc.health.smithi001.stderr:election_strategy: 3 2024-03-25T18:26:46.400 INFO:teuthology.misc.health.smithi001.stderr:0: v1:172.21.15.1:6789/0 mon.noname-a 2024-03-25T18:26:46.400 INFO:teuthology.misc.health.smithi001.stderr:1: v1:172.21.15.115:6789/0 mon.noname-c 2024-03-25T18:26:46.400 INFO:teuthology.misc.health.smithi001.stderr:2: v1:172.21.15.1:6790/0 mon.noname-b 2024-03-25T18:26:46.400 INFO:teuthology.misc.health.smithi001.stderr: 2024-03-25T18:26:46.400 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:26:46.397+0000 7f02af5f4640 1 Processor -- start 2024-03-25T18:26:46.400 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:26:46.398+0000 7f02af5f4640 1 -- start start 2024-03-25T18:26:46.400 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:26:46.398+0000 7f02af5f4640 10 monclient: init 2024-03-25T18:26:46.402 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:26:46.399+0000 7f02af5f4640 10 monclient: _reopen_session rank -1 2024-03-25T18:26:46.402 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:26:46.399+0000 7f02af5f4640 10 monclient: _add_conns ranks=[2,1,0] 2024-03-25T18:26:46.403 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:26:46.399+0000 7f02af5f4640 10 monclient(hunting): picked mon.noname-b con 0x7f02a80b4ff0 addr v1:172.21.15.1:6790/0 2024-03-25T18:26:46.403 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:26:46.399+0000 7f02af5f4640 10 monclient(hunting): picked mon.noname-c con 0x7f02a80b4670 addr v1:172.21.15.115:6789/0 2024-03-25T18:26:46.403 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:26:46.399+0000 7f02af5f4640 10 monclient(hunting): picked mon.noname-a con 0x7f02a813c480 addr v1:172.21.15.1:6789/0 2024-03-25T18:26:46.403 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:26:46.399+0000 7f02af5f4640 1 -- --> v1:172.21.15.1:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f02a80ab0e0 con 0x7f02a813c480 2024-03-25T18:26:46.403 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:26:46.399+0000 7f02af5f4640 1 -- --> v1:172.21.15.115:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f02a80ab3c0 con 0x7f02a80b4670 2024-03-25T18:26:46.403 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:26:46.399+0000 7f02af5f4640 1 -- --> v1:172.21.15.1:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f02a80a9f30 con 0x7f02a80b4ff0 2024-03-25T18:26:46.403 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:26:46.399+0000 7f02af5f4640 10 monclient(hunting): _renew_subs 2024-03-25T18:26:46.403 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:26:46.399+0000 7f02af5f4640 10 monclient(hunting): authenticate will time out at 7486.869925s 2024-03-25T18:26:46.403 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:26:46.400+0000 7f02ad390640 1 --1- >> v1:172.21.15.1:6790/0 conn(0x7f02a80b4ff0 0x7f02a813bd70 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.1:6790/0 says I am v1:172.21.15.1:54770/0 (socket says 172.21.15.1:54770) 2024-03-25T18:26:46.403 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:26:46.400+0000 7f02ad390640 1 -- 172.21.15.1:0/3546623052 learned_addr learned my addr 172.21.15.1:0/3546623052 (peer_addr_for_me v1:172.21.15.1:0/0) 2024-03-25T18:26:46.447 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:26:46.444+0000 7f029ffff640 1 -- 172.21.15.1:0/3546623052 <== mon.0 v1:172.21.15.1:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1905273125 0 0) 0x7f02a80ab0e0 con 0x7f02a813c480 2024-03-25T18:26:46.447 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:26:46.444+0000 7f029ffff640 10 monclient(hunting): _negotiate 2024-03-25T18:26:46.447 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:26:46.444+0000 7f029ffff640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:26:46.447 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:26:46.444+0000 7f029ffff640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:26:46.447 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:26:46.444+0000 7f029ffff640 10 monclient(hunting): my global_id is 4170 2024-03-25T18:26:46.447 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:26:46.444+0000 7f029ffff640 1 -- 172.21.15.1:0/3546623052 --> v1:172.21.15.1:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f0294003520 con 0x7f02a813c480 2024-03-25T18:26:46.451 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:26:46.450+0000 7f029ffff640 1 -- 172.21.15.1:0/3546623052 <== mon.0 v1:172.21.15.1:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1176126056 0 0) 0x7f0294003520 con 0x7f02a813c480 2024-03-25T18:26:46.451 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:26:46.450+0000 7f029ffff640 1 -- 172.21.15.1:0/3546623052 >> v1:172.21.15.1:6790/0 conn(0x7f02a80b4ff0 legacy=0x7f02a813bd70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:26:46.452 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:26:46.450+0000 7f029ffff640 1 -- 172.21.15.1:0/3546623052 >> v1:172.21.15.115:6789/0 conn(0x7f02a80b4670 legacy=0x7f02a81425c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:26:46.452 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:26:46.450+0000 7f029ffff640 10 monclient: _finish_hunting 0 2024-03-25T18:26:46.452 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:26:46.450+0000 7f029ffff640 1 monclient: found mon.noname-a 2024-03-25T18:26:46.452 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:26:46.450+0000 7f029ffff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:26:46.452 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:26:46.450+0000 7f029ffff640 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.1:6789/0 2024-03-25T18:26:46.452 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:26:46.450+0000 7f029ffff640 1 -- 172.21.15.1:0/3546623052 --> v1:172.21.15.1:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f02a80a9b70 con 0x7f02a813c480 2024-03-25T18:26:46.452 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:26:46.450+0000 7f029ffff640 10 monclient: _finish_auth 0 2024-03-25T18:26:46.452 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:26:46.450+0000 7f029ffff640 10 monclient: _check_auth_tickets 2024-03-25T18:26:46.452 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:26:46.450+0000 7f029ffff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:26:46.452 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:26:46.450+0000 7f029ffff640 1 -- 172.21.15.1:0/3546623052 <== mon.0 v1:172.21.15.1:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1586965025 0 0) 0x7f02a0001a10 con 0x7f02a813c480 2024-03-25T18:26:46.452 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:26:46.450+0000 7f029ffff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:26:46.452 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:26:46.450+0000 7f029ffff640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-03-25T18:26:46.452 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:26:46.450+0000 7f029ffff640 10 monclient: dump: 2024-03-25T18:26:46.452 INFO:teuthology.misc.health.smithi001.stderr:epoch 1 2024-03-25T18:26:46.453 INFO:teuthology.misc.health.smithi001.stderr:fsid c8ea72cf-9e99-4383-9b52-6c23d365fdfc 2024-03-25T18:26:46.453 INFO:teuthology.misc.health.smithi001.stderr:last_changed 2024-03-25T18:25:50.510519+0000 2024-03-25T18:26:46.453 INFO:teuthology.misc.health.smithi001.stderr:created 2024-03-25T18:25:50.510519+0000 2024-03-25T18:26:46.453 INFO:teuthology.misc.health.smithi001.stderr:min_mon_release 19 (squid) 2024-03-25T18:26:46.453 INFO:teuthology.misc.health.smithi001.stderr:election_strategy: 3 2024-03-25T18:26:46.453 INFO:teuthology.misc.health.smithi001.stderr:0: v1:172.21.15.1:6789/0 mon.a 2024-03-25T18:26:46.453 INFO:teuthology.misc.health.smithi001.stderr:1: v1:172.21.15.115:6789/0 mon.b 2024-03-25T18:26:46.453 INFO:teuthology.misc.health.smithi001.stderr:2: v1:172.21.15.1:6790/0 mon.c 2024-03-25T18:26:46.453 INFO:teuthology.misc.health.smithi001.stderr: 2024-03-25T18:26:46.453 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:26:46.450+0000 7f02af5f4640 5 monclient: authenticate success, global_id 4170 2024-03-25T18:26:46.453 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:26:46.450+0000 7f02af5f4640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T18:26:46.455 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:26:46.454+0000 7f029ffff640 1 -- 172.21.15.1:0/3546623052 <== mon.0 v1:172.21.15.1:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f02a0001b70 con 0x7f02a813c480 2024-03-25T18:26:46.455 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:26:46.454+0000 7f029ffff640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:26:46.455 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:26:46.454+0000 7f02af5f4640 10 monclient: get_monmap_and_config success 2024-03-25T18:26:46.455 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:26:46.454+0000 7f02af5f4640 4 set_mon_vals no callback set 2024-03-25T18:26:46.456 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:26:46.455+0000 7f02af5f4640 10 monclient: shutdown 2024-03-25T18:26:46.456 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:26:46.455+0000 7f02af5f4640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:26:46.457 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:26:46.455+0000 7f02af5f4640 1 -- 172.21.15.1:0/3546623052 >> v1:172.21.15.1:6789/0 conn(0x7f02a813c480 legacy=0x7f02a814dc00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:26:46.457 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:26:46.455+0000 7f02af5f4640 1 -- 172.21.15.1:0/3546623052 shutdown_connections 2024-03-25T18:26:46.457 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:26:46.455+0000 7f02af5f4640 1 -- 172.21.15.1:0/3546623052 >> 172.21.15.1:0/3546623052 conn(0x7f02a80617a0 msgr2=0x7f02a8061bb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:26:46.457 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:26:46.455+0000 7f02af5f4640 1 -- 172.21.15.1:0/3546623052 shutdown_connections 2024-03-25T18:26:46.457 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:26:46.455+0000 7f02af5f4640 1 -- 172.21.15.1:0/3546623052 wait complete. 2024-03-25T18:26:46.460 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:26:46.456+0000 7f02af5f4640 10 monclient: build_initial_monmap 2024-03-25T18:26:46.460 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:26:46.456+0000 7f02af5f4640 10 monclient: monmap: 2024-03-25T18:26:46.460 INFO:teuthology.misc.health.smithi001.stderr:epoch 0 2024-03-25T18:26:46.460 INFO:teuthology.misc.health.smithi001.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T18:26:46.460 INFO:teuthology.misc.health.smithi001.stderr:last_changed 0.000000 2024-03-25T18:26:46.460 INFO:teuthology.misc.health.smithi001.stderr:created 0.000000 2024-03-25T18:26:46.460 INFO:teuthology.misc.health.smithi001.stderr:min_mon_release 0 (unknown) 2024-03-25T18:26:46.460 INFO:teuthology.misc.health.smithi001.stderr:election_strategy: 1 2024-03-25T18:26:46.461 INFO:teuthology.misc.health.smithi001.stderr:0: v1:172.21.15.1:6789/0 mon.noname-a 2024-03-25T18:26:46.461 INFO:teuthology.misc.health.smithi001.stderr:1: v1:172.21.15.115:6789/0 mon.noname-b 2024-03-25T18:26:46.461 INFO:teuthology.misc.health.smithi001.stderr:2: v1:172.21.15.1:6790/0 mon.noname-c 2024-03-25T18:26:46.461 INFO:teuthology.misc.health.smithi001.stderr: 2024-03-25T18:26:46.461 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:26:46.456+0000 7f02af5f4640 1 librados: starting msgr at 2024-03-25T18:26:46.461 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:26:46.456+0000 7f02af5f4640 1 librados: starting objecter 2024-03-25T18:26:46.461 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:26:46.457+0000 7f02af5f4640 1 Processor -- start 2024-03-25T18:26:46.461 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:26:46.457+0000 7f02af5f4640 1 -- start start 2024-03-25T18:26:46.461 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:26:46.457+0000 7f02af5f4640 1 librados: setting wanted keys 2024-03-25T18:26:46.461 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:26:46.457+0000 7f02af5f4640 1 librados: calling monclient init 2024-03-25T18:26:46.461 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:26:46.457+0000 7f02af5f4640 10 monclient: init 2024-03-25T18:26:46.462 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:26:46.457+0000 7f02af5f4640 10 monclient: _reopen_session rank -1 2024-03-25T18:26:46.462 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:26:46.457+0000 7f02af5f4640 10 monclient: _add_conns ranks=[2,0,1] 2024-03-25T18:26:46.462 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:26:46.458+0000 7f02af5f4640 10 monclient(hunting): picked mon.noname-c con 0x7f02a80b4670 addr v1:172.21.15.1:6790/0 2024-03-25T18:26:46.462 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:26:46.458+0000 7f02af5f4640 10 monclient(hunting): picked mon.noname-a con 0x7f02a80b4ff0 addr v1:172.21.15.1:6789/0 2024-03-25T18:26:46.462 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:26:46.458+0000 7f02af5f4640 10 monclient(hunting): picked mon.noname-b con 0x7f02a813c480 addr v1:172.21.15.115:6789/0 2024-03-25T18:26:46.462 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:26:46.458+0000 7f02af5f4640 1 -- --> v1:172.21.15.1:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f02a8145de0 con 0x7f02a80b4ff0 2024-03-25T18:26:46.463 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:26:46.458+0000 7f02af5f4640 1 -- --> v1:172.21.15.115:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f02a813f6c0 con 0x7f02a813c480 2024-03-25T18:26:46.463 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:26:46.458+0000 7f02af5f4640 1 -- --> v1:172.21.15.1:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f02a81448c0 con 0x7f02a80b4670 2024-03-25T18:26:46.463 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:26:46.458+0000 7f02af5f4640 10 monclient(hunting): _renew_subs 2024-03-25T18:26:46.463 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:26:46.458+0000 7f02af5f4640 10 monclient(hunting): authenticate will time out at 7486.929137s 2024-03-25T18:26:46.463 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:26:46.458+0000 7f02acb8f640 1 --1- >> v1:172.21.15.1:6789/0 conn(0x7f02a80b4ff0 0x7f02a81456d0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.1:6789/0 says I am v1:172.21.15.1:56686/0 (socket says 172.21.15.1:56686) 2024-03-25T18:26:46.463 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:26:46.458+0000 7f02acb8f640 1 -- 172.21.15.1:0/809945256 learned_addr learned my addr 172.21.15.1:0/809945256 (peer_addr_for_me v1:172.21.15.1:0/0) 2024-03-25T18:26:46.465 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:26:46.459+0000 7f029dffb640 10 client.?.objecter ms_handle_connect 0x7f02a80b4ff0 2024-03-25T18:26:46.465 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:26:46.459+0000 7f029dffb640 10 client.?.objecter resend_mon_ops 2024-03-25T18:26:46.465 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:26:46.459+0000 7f029dffb640 10 client.?.objecter ms_handle_connect 0x7f02a813c480 2024-03-25T18:26:46.465 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:26:46.459+0000 7f029dffb640 10 client.?.objecter resend_mon_ops 2024-03-25T18:26:46.465 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:26:46.462+0000 7f029dffb640 1 -- 172.21.15.1:0/809945256 <== mon.0 v1:172.21.15.1:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1402247495 0 0) 0x7f02a8145de0 con 0x7f02a80b4ff0 2024-03-25T18:26:46.465 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:26:46.462+0000 7f029dffb640 10 monclient(hunting): _negotiate 2024-03-25T18:26:46.465 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:26:46.462+0000 7f029dffb640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:26:46.465 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:26:46.462+0000 7f029dffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:26:46.465 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:26:46.462+0000 7f029dffb640 10 monclient(hunting): my global_id is 4173 2024-03-25T18:26:46.466 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:26:46.462+0000 7f029dffb640 1 -- 172.21.15.1:0/809945256 --> v1:172.21.15.1:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f028c004eb0 con 0x7f02a80b4ff0 2024-03-25T18:26:46.466 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:26:46.463+0000 7f029dffb640 1 -- 172.21.15.1:0/809945256 <== mon.1 v1:172.21.15.115:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1440545416 0 0) 0x7f02a813f6c0 con 0x7f02a813c480 2024-03-25T18:26:46.466 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:26:46.463+0000 7f029dffb640 10 monclient(hunting): _negotiate 2024-03-25T18:26:46.466 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:26:46.463+0000 7f029dffb640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:26:46.466 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:26:46.463+0000 7f029dffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:26:46.466 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:26:46.463+0000 7f029dffb640 10 monclient(hunting): my global_id is 4171 2024-03-25T18:26:46.466 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:26:46.463+0000 7f029dffb640 1 -- 172.21.15.1:0/809945256 --> v1:172.21.15.115:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f02a8145de0 con 0x7f02a813c480 2024-03-25T18:26:46.472 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:26:46.470+0000 7f029dffb640 1 -- 172.21.15.1:0/809945256 <== mon.0 v1:172.21.15.1:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 210949450 0 0) 0x7f028c004eb0 con 0x7f02a80b4ff0 2024-03-25T18:26:46.472 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:26:46.470+0000 7f029dffb640 1 -- 172.21.15.1:0/809945256 --> v1:172.21.15.1:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f02a813f6c0 con 0x7f02a80b4ff0 2024-03-25T18:26:46.472 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:26:46.470+0000 7f029dffb640 1 -- 172.21.15.1:0/809945256 <== mon.0 v1:172.21.15.1:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1586965025 0 0) 0x7f0298001a30 con 0x7f02a80b4ff0 2024-03-25T18:26:46.473 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:26:46.470+0000 7f029dffb640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:26:46.473 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:26:46.470+0000 7f029dffb640 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2024-03-25T18:26:46.473 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:26:46.470+0000 7f029dffb640 10 monclient(hunting): dump: 2024-03-25T18:26:46.473 INFO:teuthology.misc.health.smithi001.stderr:epoch 1 2024-03-25T18:26:46.473 INFO:teuthology.misc.health.smithi001.stderr:fsid c8ea72cf-9e99-4383-9b52-6c23d365fdfc 2024-03-25T18:26:46.473 INFO:teuthology.misc.health.smithi001.stderr:last_changed 2024-03-25T18:25:50.510519+0000 2024-03-25T18:26:46.473 INFO:teuthology.misc.health.smithi001.stderr:created 2024-03-25T18:25:50.510519+0000 2024-03-25T18:26:46.473 INFO:teuthology.misc.health.smithi001.stderr:min_mon_release 19 (squid) 2024-03-25T18:26:46.473 INFO:teuthology.misc.health.smithi001.stderr:election_strategy: 3 2024-03-25T18:26:46.473 INFO:teuthology.misc.health.smithi001.stderr:0: v1:172.21.15.1:6789/0 mon.a 2024-03-25T18:26:46.473 INFO:teuthology.misc.health.smithi001.stderr:1: v1:172.21.15.115:6789/0 mon.b 2024-03-25T18:26:46.473 INFO:teuthology.misc.health.smithi001.stderr:2: v1:172.21.15.1:6790/0 mon.c 2024-03-25T18:26:46.473 INFO:teuthology.misc.health.smithi001.stderr: 2024-03-25T18:26:46.473 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:26:46.470+0000 7f029dffb640 10 monclient(hunting): _finish_auth 0 2024-03-25T18:26:46.474 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:26:46.471+0000 7f029dffb640 1 -- 172.21.15.1:0/809945256 <== mon.1 v1:172.21.15.115:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3190964269 0 0) 0x7f02a8145de0 con 0x7f02a813c480 2024-03-25T18:26:46.474 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:26:46.471+0000 7f029dffb640 1 -- 172.21.15.1:0/809945256 --> v1:172.21.15.115:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f028c004eb0 con 0x7f02a813c480 2024-03-25T18:26:46.474 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:26:46.471+0000 7f029dffb640 1 -- 172.21.15.1:0/809945256 <== mon.1 v1:172.21.15.115:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1586965025 0 0) 0x7f02a0001f40 con 0x7f02a813c480 2024-03-25T18:26:46.474 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:26:46.471+0000 7f029dffb640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:26:46.475 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:26:46.471+0000 7f029dffb640 10 monclient(hunting): got monmap 1 from mon.b (according to old e1) 2024-03-25T18:26:46.475 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:26:46.471+0000 7f029dffb640 10 monclient(hunting): dump: 2024-03-25T18:26:46.475 INFO:teuthology.misc.health.smithi001.stderr:epoch 1 2024-03-25T18:26:46.475 INFO:teuthology.misc.health.smithi001.stderr:fsid c8ea72cf-9e99-4383-9b52-6c23d365fdfc 2024-03-25T18:26:46.475 INFO:teuthology.misc.health.smithi001.stderr:last_changed 2024-03-25T18:25:50.510519+0000 2024-03-25T18:26:46.475 INFO:teuthology.misc.health.smithi001.stderr:created 2024-03-25T18:25:50.510519+0000 2024-03-25T18:26:46.475 INFO:teuthology.misc.health.smithi001.stderr:min_mon_release 19 (squid) 2024-03-25T18:26:46.475 INFO:teuthology.misc.health.smithi001.stderr:election_strategy: 3 2024-03-25T18:26:46.475 INFO:teuthology.misc.health.smithi001.stderr:0: v1:172.21.15.1:6789/0 mon.a 2024-03-25T18:26:46.475 INFO:teuthology.misc.health.smithi001.stderr:1: v1:172.21.15.115:6789/0 mon.b 2024-03-25T18:26:46.475 INFO:teuthology.misc.health.smithi001.stderr:2: v1:172.21.15.1:6790/0 mon.c 2024-03-25T18:26:46.475 INFO:teuthology.misc.health.smithi001.stderr: 2024-03-25T18:26:46.479 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:26:46.476+0000 7f029dffb640 1 -- 172.21.15.1:0/809945256 <== mon.0 v1:172.21.15.1:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 3599311550 0 0) 0x7f02a813f6c0 con 0x7f02a80b4ff0 2024-03-25T18:26:46.482 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:26:46.476+0000 7f029dffb640 1 -- 172.21.15.1:0/809945256 >> v1:172.21.15.1:6790/0 conn(0x7f02a80b4670 legacy=0x7f02a8144fc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:26:46.482 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:26:46.476+0000 7f029dffb640 1 -- 172.21.15.1:0/809945256 >> v1:172.21.15.115:6789/0 conn(0x7f02a813c480 legacy=0x7f02a81e2170 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:26:46.482 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:26:46.476+0000 7f029dffb640 10 monclient: _finish_hunting 0 2024-03-25T18:26:46.482 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:26:46.476+0000 7f029dffb640 1 monclient: found mon.a 2024-03-25T18:26:46.482 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:26:46.476+0000 7f029dffb640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:26:46.482 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:26:46.476+0000 7f029dffb640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.1:6789/0 2024-03-25T18:26:46.482 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:26:46.476+0000 7f029dffb640 1 -- 172.21.15.1:0/809945256 --> v1:172.21.15.1:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f02a8144ba0 con 0x7f02a80b4ff0 2024-03-25T18:26:46.482 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:26:46.476+0000 7f029dffb640 10 monclient: _finish_auth 0 2024-03-25T18:26:46.482 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:26:46.476+0000 7f029dffb640 10 monclient: _check_auth_tickets 2024-03-25T18:26:46.482 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:26:46.476+0000 7f029dffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:26:46.483 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:26:46.476+0000 7f02af5f4640 5 monclient: authenticate success, global_id 4173 2024-03-25T18:26:46.483 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:26:46.476+0000 7f02af5f4640 10 monclient: _renew_subs 2024-03-25T18:26:46.483 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:26:46.476+0000 7f02af5f4640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.1:6789/0 2024-03-25T18:26:46.483 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:26:46.476+0000 7f02af5f4640 1 -- 172.21.15.1:0/809945256 --> v1:172.21.15.1:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f02a8136040 con 0x7f02a80b4ff0 2024-03-25T18:26:46.483 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:26:46.477+0000 7f02af5f4640 10 client.4173.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T18:26:46.483 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:26:46.477+0000 7f02af5f4640 10 monclient: _renew_subs 2024-03-25T18:26:46.483 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:26:46.477+0000 7f02af5f4640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.1:6789/0 2024-03-25T18:26:46.483 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:26:46.477+0000 7f02af5f4640 1 -- 172.21.15.1:0/809945256 --> v1:172.21.15.1:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f02a8143660 con 0x7f02a80b4ff0 2024-03-25T18:26:46.483 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:26:46.477+0000 7f02af5f4640 1 librados: init done 2024-03-25T18:26:46.484 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:26:46.478+0000 7f02af5f4640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T18:26:46.484 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:26:46.478+0000 7f02af5f4640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:26:46.484 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:26:46.478+0000 7f02af5f4640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.1:6789/0 2024-03-25T18:26:46.484 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:26:46.478+0000 7f02af5f4640 1 -- 172.21.15.1:0/809945256 --> v1:172.21.15.1:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f02a813f490 con 0x7f02a80b4ff0 2024-03-25T18:26:46.507 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:26:46.505+0000 7f029dffb640 1 -- 172.21.15.1:0/809945256 <== mon.0 v1:172.21.15.1:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f0298001c70 con 0x7f02a80b4ff0 2024-03-25T18:26:46.507 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:26:46.505+0000 7f029dffb640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:26:46.508 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:26:46.505+0000 7f029dffb640 1 -- 172.21.15.1:0/809945256 <== mon.0 v1:172.21.15.1:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1586965025 0 0) 0x7f02980038c0 con 0x7f02a80b4ff0 2024-03-25T18:26:46.508 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:26:46.505+0000 7f029dffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:26:46.508 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:26:46.505+0000 7f029dffb640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-25T18:26:46.508 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:26:46.505+0000 7f029dffb640 10 monclient: dump: 2024-03-25T18:26:46.508 INFO:teuthology.misc.health.smithi001.stderr:epoch 1 2024-03-25T18:26:46.508 INFO:teuthology.misc.health.smithi001.stderr:fsid c8ea72cf-9e99-4383-9b52-6c23d365fdfc 2024-03-25T18:26:46.509 INFO:teuthology.misc.health.smithi001.stderr:last_changed 2024-03-25T18:25:50.510519+0000 2024-03-25T18:26:46.509 INFO:teuthology.misc.health.smithi001.stderr:created 2024-03-25T18:25:50.510519+0000 2024-03-25T18:26:46.509 INFO:teuthology.misc.health.smithi001.stderr:min_mon_release 19 (squid) 2024-03-25T18:26:46.509 INFO:teuthology.misc.health.smithi001.stderr:election_strategy: 3 2024-03-25T18:26:46.509 INFO:teuthology.misc.health.smithi001.stderr:0: v1:172.21.15.1:6789/0 mon.a 2024-03-25T18:26:46.509 INFO:teuthology.misc.health.smithi001.stderr:1: v1:172.21.15.115:6789/0 mon.b 2024-03-25T18:26:46.509 INFO:teuthology.misc.health.smithi001.stderr:2: v1:172.21.15.1:6790/0 mon.c 2024-03-25T18:26:46.510 INFO:teuthology.misc.health.smithi001.stderr: 2024-03-25T18:26:46.510 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:26:46.505+0000 7f029f7fe640 4 set_mon_vals no callback set 2024-03-25T18:26:46.510 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:26:46.506+0000 7f029dffb640 1 -- 172.21.15.1:0/809945256 <== mon.0 v1:172.21.15.1:6789/0 7 ==== mgrmap(e 3) v1 ==== 89499+0+0 (unknown 588105038 0 0) 0x7f0298003aa0 con 0x7f02a80b4ff0 2024-03-25T18:26:46.686 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:26:46.684+0000 7f029dffb640 1 -- 172.21.15.1:0/809945256 <== mon.0 v1:172.21.15.1:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 996254230 0 0) 0x7f0298054600 con 0x7f02a80b4ff0 2024-03-25T18:26:46.686 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:26:46.684+0000 7f029dffb640 10 client.4173.objecter ms_dispatch 0x7f02a8000b90 osd_map(10..10 src has 1..10) v4 2024-03-25T18:26:46.686 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:26:46.684+0000 7f029dffb640 3 client.4173.objecter handle_osd_map got epochs [10,10] > 0 2024-03-25T18:26:46.686 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:26:46.684+0000 7f029dffb640 3 client.4173.objecter handle_osd_map decoding full epoch 10 2024-03-25T18:26:46.686 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:26:46.684+0000 7f029dffb640 20 client.4173.objecter dump_active .. 0 homeless 2024-03-25T18:26:46.687 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:26:46.684+0000 7f029dffb640 1 -- 172.21.15.1:0/809945256 <== mon.0 v1:172.21.15.1:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+73141 (unknown 1092875540 0 2031569661) 0x7f0298050110 con 0x7f02a80b4ff0 2024-03-25T18:26:46.687 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:26:46.684+0000 7f029dffb640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:26:46.687 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:26:46.684+0000 7f029dffb640 10 monclient: _finish_command 1 = system:0 2024-03-25T18:26:46.809 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:26:46.807+0000 7f02af5f4640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:26:46.809 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:26:46.807+0000 7f02af5f4640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:26:46.809 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:26:46.807+0000 7f02af5f4640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.1:6789/0 2024-03-25T18:26:46.809 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:26:46.807+0000 7f02af5f4640 1 -- 172.21.15.1:0/809945256 --> v1:172.21.15.1:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f02a80688c0 con 0x7f02a80b4ff0 2024-03-25T18:26:46.839 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:26:46.837+0000 7f029dffb640 1 -- 172.21.15.1:0/809945256 <== mon.0 v1:172.21.15.1:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 4092213348) 0x7f0298052040 con 0x7f02a80b4ff0 2024-03-25T18:26:46.839 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:26:46.838+0000 7f029dffb640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:26:46.840 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:26:46.838+0000 7f029dffb640 10 monclient: _finish_command 2 = system:0 2024-03-25T18:26:46.840 INFO:teuthology.misc.health.smithi001.stdout: 2024-03-25T18:26:46.840 INFO:teuthology.misc.health.smithi001.stdout:{"epoch":10,"fsid":"c8ea72cf-9e99-4383-9b52-6c23d365fdfc","created":"2024-03-25T18:26:38.856638+0000","modified":"2024-03-25T18:26:44.224208+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T18:26:44.224208+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"b9f34922-3a84-49be-8426-807c1eceaefc","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"42bf306f-55e3-4634-9874-8b4b73ac6644","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"292cb23a-18a9-4207-a7cc-4d5727feaec0","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"45e0e9ab-898d-4673-9590-b4117bc5e84a","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"831a5a09-73ab-4a61-9872-088e76bdbc66","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"88cd903d-3b55-4840-9396-6e3cfe005282","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"89db7355-bc9d-46f4-8ab0-e8d9e744e026","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"cac47bee-0914-4f08-896f-348838882866","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T18:26:46.842 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:26:46.840+0000 7f02af5f4640 10 librados: watch_flush enter 2024-03-25T18:26:46.842 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:26:46.840+0000 7f02af5f4640 10 librados: watch_flush exit 2024-03-25T18:26:46.842 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:26:46.840+0000 7f02af5f4640 20 client.4173.objecter shutdown clearing up homeless session... 2024-03-25T18:26:46.842 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:26:46.840+0000 7f02af5f4640 10 client.4173.objecter successfully canceled tick 2024-03-25T18:26:46.842 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:26:46.840+0000 7f02af5f4640 10 monclient: shutdown 2024-03-25T18:26:46.842 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:26:46.840+0000 7f02af5f4640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:26:46.842 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:26:46.840+0000 7f02af5f4640 1 -- 172.21.15.1:0/809945256 >> v1:172.21.15.1:6789/0 conn(0x7f02a80b4ff0 legacy=0x7f02a81456d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:26:46.842 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:26:46.840+0000 7f02af5f4640 1 -- 172.21.15.1:0/809945256 shutdown_connections 2024-03-25T18:26:46.843 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:26:46.840+0000 7f02af5f4640 1 -- 172.21.15.1:0/809945256 >> 172.21.15.1:0/809945256 conn(0x7f02a80617a0 msgr2=0x7f02a8140700 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:26:46.843 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:26:46.840+0000 7f02af5f4640 1 -- 172.21.15.1:0/809945256 shutdown_connections 2024-03-25T18:26:46.843 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:26:46.840+0000 7f02af5f4640 1 -- 172.21.15.1:0/809945256 wait complete. 2024-03-25T18:26:46.843 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:26:46.840+0000 7f02af5f4640 1 librados: shutdown 2024-03-25T18:26:46.860 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T18:26:47.968 INFO:tasks.ceph.mgr.x.smithi115.stderr:2024-03-25T18:26:47.968+0000 7f997de37640 -1 mgr.server handle_report got status from non-daemon mon.b 2024-03-25T18:26:48.049 INFO:tasks.ceph.mgr.x.smithi115.stderr:2024-03-25T18:26:48.048+0000 7f997de37640 -1 mgr.server handle_report got status from non-daemon mon.c 2024-03-25T18:26:51.173 INFO:tasks.ceph.osd.4.smithi115.stderr:==00:00:00:06.790 35380== Warning: unimplemented fcntl command: 1038 2024-03-25T18:26:51.174 INFO:tasks.ceph.osd.5.smithi115.stderr:==00:00:00:06.790 35381== Warning: unimplemented fcntl command: 1038 2024-03-25T18:26:51.233 INFO:tasks.ceph.osd.4.smithi115.stderr:==00:00:00:06.850 35380== Warning: unimplemented fcntl command: 1038 2024-03-25T18:26:51.233 INFO:tasks.ceph.osd.5.smithi115.stderr:==00:00:00:06.850 35381== Warning: unimplemented fcntl command: 1038 2024-03-25T18:26:51.301 INFO:tasks.ceph.osd.6.smithi115.stderr:==00:00:00:06.918 35382== Warning: unimplemented fcntl command: 1038 2024-03-25T18:26:51.361 INFO:tasks.ceph.osd.6.smithi115.stderr:==00:00:00:06.977 35382== Warning: unimplemented fcntl command: 1038 2024-03-25T18:26:51.506 INFO:tasks.ceph.osd.7.smithi115.stderr:==00:00:00:07.118 35383== Warning: unimplemented fcntl command: 1038 2024-03-25T18:26:51.567 INFO:tasks.ceph.osd.7.smithi115.stderr:==00:00:00:07.179 35383== Warning: unimplemented fcntl command: 1038 2024-03-25T18:26:51.782 INFO:tasks.ceph.osd.4.smithi115.stderr:2024-03-25T18:26:51.775+0000 5dc3480 -1 Falling back to public interface 2024-03-25T18:26:51.787 INFO:tasks.ceph.osd.5.smithi115.stderr:2024-03-25T18:26:51.781+0000 5dc3480 -1 Falling back to public interface 2024-03-25T18:26:51.864 INFO:tasks.ceph.osd.3.smithi001.stderr:==00:00:00:07.466 35191== Warning: unimplemented fcntl command: 1038 2024-03-25T18:26:51.890 INFO:tasks.ceph.osd.1.smithi001.stderr:==00:00:00:07.507 35189== Warning: unimplemented fcntl command: 1038 2024-03-25T18:26:51.912 INFO:tasks.ceph.osd.2.smithi001.stderr:==00:00:00:07.504 35192== Warning: unimplemented fcntl command: 1038 2024-03-25T18:26:51.926 INFO:tasks.ceph.osd.3.smithi001.stderr:==00:00:00:07.528 35191== Warning: unimplemented fcntl command: 1038 2024-03-25T18:26:51.934 INFO:tasks.ceph.osd.6.smithi115.stderr:2024-03-25T18:26:51.928+0000 5dc3480 -1 Falling back to public interface 2024-03-25T18:26:51.952 INFO:tasks.ceph.osd.1.smithi001.stderr:==00:00:00:07.569 35189== Warning: unimplemented fcntl command: 1038 2024-03-25T18:26:51.973 INFO:tasks.ceph.osd.2.smithi001.stderr:==00:00:00:07.565 35192== Warning: unimplemented fcntl command: 1038 2024-03-25T18:26:52.075 INFO:tasks.ceph.osd.0.smithi001.stderr:==00:00:00:07.682 35190== Warning: unimplemented fcntl command: 1038 2024-03-25T18:26:52.122 INFO:tasks.ceph.osd.7.smithi115.stderr:2024-03-25T18:26:52.114+0000 5dc3480 -1 Falling back to public interface 2024-03-25T18:26:52.135 INFO:tasks.ceph.osd.0.smithi001.stderr:==00:00:00:07.742 35190== Warning: unimplemented fcntl command: 1038 2024-03-25T18:26:52.234 INFO:tasks.ceph.osd.4.smithi115.stderr:==00:00:00:07.851 35380== Warning: unimplemented fcntl command: 1038 2024-03-25T18:26:52.256 INFO:tasks.ceph.osd.5.smithi115.stderr:==00:00:00:07.873 35381== Warning: unimplemented fcntl command: 1038 2024-03-25T18:26:52.399 INFO:tasks.ceph.osd.6.smithi115.stderr:==00:00:00:08.015 35382== Warning: unimplemented fcntl command: 1038 2024-03-25T18:26:52.478 INFO:tasks.ceph.osd.3.smithi001.stderr:2024-03-25T18:26:52.464+0000 5dc3480 -1 Falling back to public interface 2024-03-25T18:26:52.505 INFO:tasks.ceph.osd.1.smithi001.stderr:2024-03-25T18:26:52.497+0000 5dc3480 -1 Falling back to public interface 2024-03-25T18:26:52.530 INFO:tasks.ceph.osd.2.smithi001.stderr:2024-03-25T18:26:52.523+0000 5dc3480 -1 Falling back to public interface 2024-03-25T18:26:52.583 INFO:tasks.ceph.osd.7.smithi115.stderr:==00:00:00:08.195 35383== Warning: unimplemented fcntl command: 1038 2024-03-25T18:26:52.601 INFO:tasks.ceph.osd.4.smithi115.stderr:==00:00:00:08.219 35380== Warning: unimplemented fcntl command: 1038 2024-03-25T18:26:52.633 INFO:tasks.ceph.osd.5.smithi115.stderr:==00:00:00:08.250 35381== Warning: unimplemented fcntl command: 1038 2024-03-25T18:26:52.696 INFO:tasks.ceph.osd.0.smithi001.stderr:2024-03-25T18:26:52.688+0000 5dc3480 -1 Falling back to public interface 2024-03-25T18:26:52.761 INFO:tasks.ceph.osd.6.smithi115.stderr:==00:00:00:08.377 35382== Warning: unimplemented fcntl command: 1038 2024-03-25T18:26:52.939 INFO:tasks.ceph.osd.3.smithi001.stderr:==00:00:00:08.541 35191== Warning: unimplemented fcntl command: 1038 2024-03-25T18:26:52.952 INFO:tasks.ceph.osd.7.smithi115.stderr:==00:00:00:08.564 35383== Warning: unimplemented fcntl command: 1038 2024-03-25T18:26:52.964 INFO:tasks.ceph.osd.4.smithi115.stderr:==00:00:00:08.582 35380== Warning: unimplemented fcntl command: 1038 2024-03-25T18:26:52.992 INFO:tasks.ceph.osd.5.smithi115.stderr:==00:00:00:08.609 35381== Warning: unimplemented fcntl command: 1038 2024-03-25T18:26:53.003 INFO:tasks.ceph.osd.1.smithi001.stderr:==00:00:00:08.620 35189== Warning: unimplemented fcntl command: 1038 2024-03-25T18:26:53.071 INFO:tasks.ceph.osd.6.smithi115.stderr:==00:00:00:08.688 35382== Warning: unimplemented fcntl command: 1038 2024-03-25T18:26:53.072 INFO:tasks.ceph.osd.2.smithi001.stderr:==00:00:00:08.663 35192== Warning: unimplemented fcntl command: 1038 2024-03-25T18:26:53.233 INFO:tasks.ceph.osd.0.smithi001.stderr:==00:00:00:08.838 35190== Warning: unimplemented fcntl command: 1038 2024-03-25T18:26:53.253 INFO:tasks.ceph.osd.4.smithi115.stderr:==00:00:00:08.870 35380== Warning: unimplemented fcntl command: 1038 2024-03-25T18:26:53.270 INFO:tasks.ceph.osd.7.smithi115.stderr:==00:00:00:08.882 35383== Warning: unimplemented fcntl command: 1038 2024-03-25T18:26:53.302 INFO:tasks.ceph.osd.5.smithi115.stderr:==00:00:00:08.919 35381== Warning: unimplemented fcntl command: 1038 2024-03-25T18:26:53.304 INFO:tasks.ceph.osd.3.smithi001.stderr:==00:00:00:08.906 35191== Warning: unimplemented fcntl command: 1038 2024-03-25T18:26:53.346 INFO:tasks.ceph.osd.6.smithi115.stderr:==00:00:00:08.962 35382== Warning: unimplemented fcntl command: 1038 2024-03-25T18:26:53.369 INFO:tasks.ceph.osd.1.smithi001.stderr:==00:00:00:08.987 35189== Warning: unimplemented fcntl command: 1038 2024-03-25T18:26:53.442 INFO:tasks.ceph.osd.2.smithi001.stderr:==00:00:00:09.034 35192== Warning: unimplemented fcntl command: 1038 2024-03-25T18:26:53.546 INFO:tasks.ceph.osd.7.smithi115.stderr:==00:00:00:09.158 35383== Warning: unimplemented fcntl command: 1038 2024-03-25T18:26:53.564 INFO:tasks.ceph.osd.4.smithi115.stderr:==00:00:00:09.181 35380== Warning: unimplemented fcntl command: 1038 2024-03-25T18:26:53.606 INFO:tasks.ceph.osd.5.smithi115.stderr:==00:00:00:09.223 35381== Warning: unimplemented fcntl command: 1038 2024-03-25T18:26:53.611 INFO:tasks.ceph.osd.6.smithi115.stderr:==00:00:00:09.228 35382== Warning: unimplemented fcntl command: 1038 2024-03-25T18:26:53.612 INFO:tasks.ceph.osd.0.smithi001.stderr:==00:00:00:09.219 35190== Warning: unimplemented fcntl command: 1038 2024-03-25T18:26:53.621 INFO:tasks.ceph.osd.3.smithi001.stderr:==00:00:00:09.223 35191== Warning: unimplemented fcntl command: 1038 2024-03-25T18:26:53.666 DEBUG:teuthology.orchestra.run.smithi001:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T18:26:53.740 INFO:tasks.ceph.osd.1.smithi001.stderr:==00:00:00:09.357 35189== Warning: unimplemented fcntl command: 1038 2024-03-25T18:26:53.755 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:26:53.753+0000 7ff188e2e640 10 monclient: get_monmap_and_config 2024-03-25T18:26:53.755 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:26:53.753+0000 7ff188e2e640 10 monclient: build_initial_monmap 2024-03-25T18:26:53.756 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:26:53.753+0000 7ff188e2e640 10 monclient: monmap: 2024-03-25T18:26:53.756 INFO:teuthology.misc.health.smithi001.stderr:epoch 0 2024-03-25T18:26:53.756 INFO:teuthology.misc.health.smithi001.stderr:fsid c8ea72cf-9e99-4383-9b52-6c23d365fdfc 2024-03-25T18:26:53.756 INFO:teuthology.misc.health.smithi001.stderr:last_changed 2024-03-25T18:26:53.755343+0000 2024-03-25T18:26:53.756 INFO:teuthology.misc.health.smithi001.stderr:created 2024-03-25T18:26:53.755343+0000 2024-03-25T18:26:53.756 INFO:teuthology.misc.health.smithi001.stderr:min_mon_release 0 (unknown) 2024-03-25T18:26:53.756 INFO:teuthology.misc.health.smithi001.stderr:election_strategy: 3 2024-03-25T18:26:53.756 INFO:teuthology.misc.health.smithi001.stderr:0: v1:172.21.15.1:6789/0 mon.noname-a 2024-03-25T18:26:53.756 INFO:teuthology.misc.health.smithi001.stderr:1: v1:172.21.15.115:6789/0 mon.noname-c 2024-03-25T18:26:53.756 INFO:teuthology.misc.health.smithi001.stderr:2: v1:172.21.15.1:6790/0 mon.noname-b 2024-03-25T18:26:53.756 INFO:teuthology.misc.health.smithi001.stderr: 2024-03-25T18:26:53.756 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:26:53.754+0000 7ff188e2e640 1 Processor -- start 2024-03-25T18:26:53.756 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:26:53.754+0000 7ff188e2e640 1 -- start start 2024-03-25T18:26:53.756 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:26:53.754+0000 7ff188e2e640 10 monclient: init 2024-03-25T18:26:53.758 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:26:53.755+0000 7ff188e2e640 10 monclient: _reopen_session rank -1 2024-03-25T18:26:53.758 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:26:53.755+0000 7ff188e2e640 10 monclient: _add_conns ranks=[2,0,1] 2024-03-25T18:26:53.758 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:26:53.755+0000 7ff188e2e640 10 monclient(hunting): picked mon.noname-b con 0x7ff1840b4ff0 addr v1:172.21.15.1:6790/0 2024-03-25T18:26:53.758 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:26:53.755+0000 7ff188e2e640 10 monclient(hunting): picked mon.noname-a con 0x7ff1840b4670 addr v1:172.21.15.1:6789/0 2024-03-25T18:26:53.758 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:26:53.755+0000 7ff188e2e640 10 monclient(hunting): picked mon.noname-c con 0x7ff18413c480 addr v1:172.21.15.115:6789/0 2024-03-25T18:26:53.758 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:26:53.755+0000 7ff188e2e640 1 -- --> v1:172.21.15.1:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff1840ab0e0 con 0x7ff1840b4670 2024-03-25T18:26:53.758 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:26:53.755+0000 7ff188e2e640 1 -- --> v1:172.21.15.115:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff1840ab3c0 con 0x7ff18413c480 2024-03-25T18:26:53.758 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:26:53.755+0000 7ff188e2e640 1 -- --> v1:172.21.15.1:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff1840a9f30 con 0x7ff1840b4ff0 2024-03-25T18:26:53.758 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:26:53.755+0000 7ff188e2e640 10 monclient(hunting): _renew_subs 2024-03-25T18:26:53.759 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:26:53.755+0000 7ff188e2e640 10 monclient(hunting): authenticate will time out at 7494.225903s 2024-03-25T18:26:53.759 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:26:53.755+0000 7ff181d9b640 1 --1- >> v1:172.21.15.1:6789/0 conn(0x7ff1840b4670 0x7ff1841425c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.1:6789/0 says I am v1:172.21.15.1:51238/0 (socket says 172.21.15.1:51238) 2024-03-25T18:26:53.759 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:26:53.755+0000 7ff182d9d640 1 --1- >> v1:172.21.15.115:6789/0 conn(0x7ff18413c480 0x7ff18414dc00 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.115:6789/0 says I am v1:172.21.15.1:60698/0 (socket says 172.21.15.1:60698) 2024-03-25T18:26:53.759 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:26:53.755+0000 7ff181d9b640 1 -- 172.21.15.1:0/1357361869 learned_addr learned my addr 172.21.15.1:0/1357361869 (peer_addr_for_me v1:172.21.15.1:0/0) 2024-03-25T18:26:53.759 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:26:53.757+0000 7ff18159a640 1 -- 172.21.15.1:0/1357361869 <== mon.1 v1:172.21.15.115:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 768958796 0 0) 0x7ff1840ab3c0 con 0x7ff18413c480 2024-03-25T18:26:53.759 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:26:53.757+0000 7ff18159a640 10 monclient(hunting): _negotiate 2024-03-25T18:26:53.759 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:26:53.757+0000 7ff18159a640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:26:53.759 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:26:53.757+0000 7ff18159a640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:26:53.760 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:26:53.757+0000 7ff18159a640 10 monclient(hunting): my global_id is 4198 2024-03-25T18:26:53.760 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:26:53.757+0000 7ff18159a640 1 -- 172.21.15.1:0/1357361869 --> v1:172.21.15.115:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ff170003520 con 0x7ff18413c480 2024-03-25T18:26:53.760 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:26:53.758+0000 7ff18159a640 1 -- 172.21.15.1:0/1357361869 <== mon.0 v1:172.21.15.1:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 461915306 0 0) 0x7ff1840ab0e0 con 0x7ff1840b4670 2024-03-25T18:26:53.760 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:26:53.758+0000 7ff18159a640 10 monclient(hunting): _negotiate 2024-03-25T18:26:53.760 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:26:53.758+0000 7ff18159a640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:26:53.760 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:26:53.758+0000 7ff18159a640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:26:53.761 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:26:53.758+0000 7ff18159a640 10 monclient(hunting): my global_id is 4200 2024-03-25T18:26:53.761 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:26:53.758+0000 7ff18159a640 1 -- 172.21.15.1:0/1357361869 --> v1:172.21.15.1:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ff1840ab3c0 con 0x7ff1840b4670 2024-03-25T18:26:53.761 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:26:53.758+0000 7ff18159a640 1 -- 172.21.15.1:0/1357361869 <== mon.2 v1:172.21.15.1:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 485364932 0 0) 0x7ff1840a9f30 con 0x7ff1840b4ff0 2024-03-25T18:26:53.761 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:26:53.758+0000 7ff18159a640 10 monclient(hunting): _negotiate 2024-03-25T18:26:53.761 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:26:53.758+0000 7ff18159a640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:26:53.761 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:26:53.758+0000 7ff18159a640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:26:53.761 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:26:53.758+0000 7ff18159a640 10 monclient(hunting): my global_id is 4205 2024-03-25T18:26:53.761 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:26:53.758+0000 7ff18159a640 1 -- 172.21.15.1:0/1357361869 --> v1:172.21.15.1:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ff1840ab0e0 con 0x7ff1840b4ff0 2024-03-25T18:26:53.762 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:26:53.761+0000 7ff18159a640 1 -- 172.21.15.1:0/1357361869 <== mon.1 v1:172.21.15.115:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1527532459 0 0) 0x7ff170003520 con 0x7ff18413c480 2024-03-25T18:26:53.763 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:26:53.761+0000 7ff18159a640 1 -- 172.21.15.1:0/1357361869 >> v1:172.21.15.1:6790/0 conn(0x7ff1840b4ff0 legacy=0x7ff18413bd70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:26:53.763 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:26:53.761+0000 7ff18159a640 1 -- 172.21.15.1:0/1357361869 >> v1:172.21.15.1:6789/0 conn(0x7ff1840b4670 legacy=0x7ff1841425c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:26:53.763 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:26:53.761+0000 7ff18159a640 10 monclient: _finish_hunting 0 2024-03-25T18:26:53.763 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:26:53.761+0000 7ff18159a640 1 monclient: found mon.noname-c 2024-03-25T18:26:53.763 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:26:53.761+0000 7ff18159a640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:26:53.763 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:26:53.761+0000 7ff18159a640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.115:6789/0 2024-03-25T18:26:53.763 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:26:53.761+0000 7ff18159a640 1 -- 172.21.15.1:0/1357361869 --> v1:172.21.15.115:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff1840a9b70 con 0x7ff18413c480 2024-03-25T18:26:53.763 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:26:53.761+0000 7ff18159a640 10 monclient: _finish_auth 0 2024-03-25T18:26:53.763 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:26:53.761+0000 7ff18159a640 10 monclient: _check_auth_tickets 2024-03-25T18:26:53.763 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:26:53.761+0000 7ff18159a640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:26:53.763 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:26:53.761+0000 7ff188e2e640 5 monclient: authenticate success, global_id 4198 2024-03-25T18:26:53.763 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:26:53.761+0000 7ff18159a640 1 -- 172.21.15.1:0/1357361869 <== mon.1 v1:172.21.15.115:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1586965025 0 0) 0x7ff174001690 con 0x7ff18413c480 2024-03-25T18:26:53.763 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:26:53.761+0000 7ff18159a640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:26:53.764 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:26:53.761+0000 7ff18159a640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-03-25T18:26:53.764 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:26:53.761+0000 7ff18159a640 10 monclient: dump: 2024-03-25T18:26:53.764 INFO:teuthology.misc.health.smithi001.stderr:epoch 1 2024-03-25T18:26:53.764 INFO:teuthology.misc.health.smithi001.stderr:fsid c8ea72cf-9e99-4383-9b52-6c23d365fdfc 2024-03-25T18:26:53.764 INFO:teuthology.misc.health.smithi001.stderr:last_changed 2024-03-25T18:25:50.510519+0000 2024-03-25T18:26:53.764 INFO:teuthology.misc.health.smithi001.stderr:created 2024-03-25T18:25:50.510519+0000 2024-03-25T18:26:53.764 INFO:teuthology.misc.health.smithi001.stderr:min_mon_release 19 (squid) 2024-03-25T18:26:53.764 INFO:teuthology.misc.health.smithi001.stderr:election_strategy: 3 2024-03-25T18:26:53.764 INFO:teuthology.misc.health.smithi001.stderr:0: v1:172.21.15.1:6789/0 mon.a 2024-03-25T18:26:53.765 INFO:teuthology.misc.health.smithi001.stderr:1: v1:172.21.15.115:6789/0 mon.b 2024-03-25T18:26:53.765 INFO:teuthology.misc.health.smithi001.stderr:2: v1:172.21.15.1:6790/0 mon.c 2024-03-25T18:26:53.765 INFO:teuthology.misc.health.smithi001.stderr: 2024-03-25T18:26:53.765 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:26:53.761+0000 7ff188e2e640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T18:26:53.768 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:26:53.762+0000 7ff18159a640 1 -- 172.21.15.1:0/1357361869 <== mon.1 v1:172.21.15.115:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7ff1740018b0 con 0x7ff18413c480 2024-03-25T18:26:53.768 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:26:53.762+0000 7ff18159a640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:26:53.768 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:26:53.762+0000 7ff18159a640 1 -- 172.21.15.1:0/1357361869 <== mon.1 v1:172.21.15.115:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1586965025 0 0) 0x7ff174001c40 con 0x7ff18413c480 2024-03-25T18:26:53.768 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:26:53.762+0000 7ff188e2e640 10 monclient: get_monmap_and_config success 2024-03-25T18:26:53.768 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:26:53.762+0000 7ff188e2e640 4 set_mon_vals no callback set 2024-03-25T18:26:53.768 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:26:53.762+0000 7ff18159a640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:26:53.768 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:26:53.762+0000 7ff18159a640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-25T18:26:53.768 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:26:53.762+0000 7ff18159a640 10 monclient: dump: 2024-03-25T18:26:53.768 INFO:teuthology.misc.health.smithi001.stderr:epoch 1 2024-03-25T18:26:53.768 INFO:teuthology.misc.health.smithi001.stderr:fsid c8ea72cf-9e99-4383-9b52-6c23d365fdfc 2024-03-25T18:26:53.768 INFO:teuthology.misc.health.smithi001.stderr:last_changed 2024-03-25T18:25:50.510519+0000 2024-03-25T18:26:53.768 INFO:teuthology.misc.health.smithi001.stderr:created 2024-03-25T18:25:50.510519+0000 2024-03-25T18:26:53.769 INFO:teuthology.misc.health.smithi001.stderr:min_mon_release 19 (squid) 2024-03-25T18:26:53.769 INFO:teuthology.misc.health.smithi001.stderr:election_strategy: 3 2024-03-25T18:26:53.769 INFO:teuthology.misc.health.smithi001.stderr:0: v1:172.21.15.1:6789/0 mon.a 2024-03-25T18:26:53.769 INFO:teuthology.misc.health.smithi001.stderr:1: v1:172.21.15.115:6789/0 mon.b 2024-03-25T18:26:53.769 INFO:teuthology.misc.health.smithi001.stderr:2: v1:172.21.15.1:6790/0 mon.c 2024-03-25T18:26:53.769 INFO:teuthology.misc.health.smithi001.stderr: 2024-03-25T18:26:53.769 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:26:53.763+0000 7ff188e2e640 10 monclient: shutdown 2024-03-25T18:26:53.769 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:26:53.763+0000 7ff188e2e640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:26:53.769 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:26:53.763+0000 7ff188e2e640 1 -- 172.21.15.1:0/1357361869 >> v1:172.21.15.115:6789/0 conn(0x7ff18413c480 legacy=0x7ff18414dc00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:26:53.769 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:26:53.763+0000 7ff188e2e640 1 -- 172.21.15.1:0/1357361869 shutdown_connections 2024-03-25T18:26:53.769 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:26:53.763+0000 7ff188e2e640 1 -- 172.21.15.1:0/1357361869 >> 172.21.15.1:0/1357361869 conn(0x7ff1840617a0 msgr2=0x7ff184061bb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:26:53.769 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:26:53.763+0000 7ff188e2e640 1 -- 172.21.15.1:0/1357361869 shutdown_connections 2024-03-25T18:26:53.769 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:26:53.763+0000 7ff188e2e640 1 -- 172.21.15.1:0/1357361869 wait complete. 2024-03-25T18:26:53.769 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:26:53.764+0000 7ff188e2e640 10 monclient: build_initial_monmap 2024-03-25T18:26:53.770 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:26:53.764+0000 7ff188e2e640 10 monclient: monmap: 2024-03-25T18:26:53.770 INFO:teuthology.misc.health.smithi001.stderr:epoch 0 2024-03-25T18:26:53.770 INFO:teuthology.misc.health.smithi001.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T18:26:53.770 INFO:teuthology.misc.health.smithi001.stderr:last_changed 0.000000 2024-03-25T18:26:53.770 INFO:teuthology.misc.health.smithi001.stderr:created 0.000000 2024-03-25T18:26:53.770 INFO:teuthology.misc.health.smithi001.stderr:min_mon_release 0 (unknown) 2024-03-25T18:26:53.770 INFO:teuthology.misc.health.smithi001.stderr:election_strategy: 1 2024-03-25T18:26:53.770 INFO:teuthology.misc.health.smithi001.stderr:0: v1:172.21.15.1:6789/0 mon.noname-a 2024-03-25T18:26:53.770 INFO:teuthology.misc.health.smithi001.stderr:1: v1:172.21.15.115:6789/0 mon.noname-b 2024-03-25T18:26:53.770 INFO:teuthology.misc.health.smithi001.stderr:2: v1:172.21.15.1:6790/0 mon.noname-c 2024-03-25T18:26:53.770 INFO:teuthology.misc.health.smithi001.stderr: 2024-03-25T18:26:53.770 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:26:53.764+0000 7ff188e2e640 1 librados: starting msgr at 2024-03-25T18:26:53.770 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:26:53.764+0000 7ff188e2e640 1 librados: starting objecter 2024-03-25T18:26:53.770 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:26:53.764+0000 7ff188e2e640 1 Processor -- start 2024-03-25T18:26:53.771 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:26:53.765+0000 7ff188e2e640 1 -- start start 2024-03-25T18:26:53.771 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:26:53.765+0000 7ff188e2e640 1 librados: setting wanted keys 2024-03-25T18:26:53.771 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:26:53.765+0000 7ff188e2e640 1 librados: calling monclient init 2024-03-25T18:26:53.771 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:26:53.765+0000 7ff188e2e640 10 monclient: init 2024-03-25T18:26:53.771 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:26:53.765+0000 7ff188e2e640 10 monclient: _reopen_session rank -1 2024-03-25T18:26:53.771 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:26:53.765+0000 7ff188e2e640 10 monclient: _add_conns ranks=[2,1,0] 2024-03-25T18:26:53.771 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:26:53.765+0000 7ff188e2e640 10 monclient(hunting): picked mon.noname-c con 0x7ff1840b4670 addr v1:172.21.15.1:6790/0 2024-03-25T18:26:53.771 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:26:53.765+0000 7ff188e2e640 10 monclient(hunting): picked mon.noname-b con 0x7ff1840b4ff0 addr v1:172.21.15.115:6789/0 2024-03-25T18:26:53.771 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:26:53.765+0000 7ff188e2e640 10 monclient(hunting): picked mon.noname-a con 0x7ff18413c480 addr v1:172.21.15.1:6789/0 2024-03-25T18:26:53.771 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:26:53.765+0000 7ff188e2e640 1 -- --> v1:172.21.15.1:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff1840b0580 con 0x7ff18413c480 2024-03-25T18:26:53.771 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:26:53.765+0000 7ff188e2e640 1 -- --> v1:172.21.15.115:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff1840b07f0 con 0x7ff1840b4ff0 2024-03-25T18:26:53.771 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:26:53.765+0000 7ff188e2e640 1 -- --> v1:172.21.15.1:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff1840b0ad0 con 0x7ff1840b4670 2024-03-25T18:26:53.771 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:26:53.765+0000 7ff188e2e640 10 monclient(hunting): _renew_subs 2024-03-25T18:26:53.771 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:26:53.765+0000 7ff188e2e640 10 monclient(hunting): authenticate will time out at 7494.236272s 2024-03-25T18:26:53.771 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:26:53.765+0000 7ff182d9d640 1 --1- >> v1:172.21.15.1:6789/0 conn(0x7ff18413c480 0x7ff1841e2380 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.1:6789/0 says I am v1:172.21.15.1:51250/0 (socket says 172.21.15.1:51250) 2024-03-25T18:26:53.772 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:26:53.766+0000 7ff182d9d640 1 -- 172.21.15.1:0/2301587571 learned_addr learned my addr 172.21.15.1:0/2301587571 (peer_addr_for_me v1:172.21.15.1:0/0) 2024-03-25T18:26:53.772 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:26:53.766+0000 7ff17affd640 10 client.?.objecter ms_handle_connect 0x7ff1840b4ff0 2024-03-25T18:26:53.772 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:26:53.766+0000 7ff17affd640 10 client.?.objecter resend_mon_ops 2024-03-25T18:26:53.772 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:26:53.766+0000 7ff17affd640 10 client.?.objecter ms_handle_connect 0x7ff18413c480 2024-03-25T18:26:53.772 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:26:53.766+0000 7ff17affd640 10 client.?.objecter resend_mon_ops 2024-03-25T18:26:53.772 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:26:53.766+0000 7ff17affd640 10 client.?.objecter ms_handle_connect 0x7ff1840b4670 2024-03-25T18:26:53.772 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:26:53.766+0000 7ff17affd640 10 client.?.objecter resend_mon_ops 2024-03-25T18:26:53.773 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:26:53.768+0000 7ff17affd640 1 -- 172.21.15.1:0/2301587571 <== mon.1 v1:172.21.15.115:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2069017114 0 0) 0x7ff1840b07f0 con 0x7ff1840b4ff0 2024-03-25T18:26:53.773 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:26:53.768+0000 7ff17affd640 10 monclient(hunting): _negotiate 2024-03-25T18:26:53.773 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:26:53.768+0000 7ff17affd640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:26:53.773 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:26:53.768+0000 7ff17affd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:26:53.773 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:26:53.768+0000 7ff17affd640 10 monclient(hunting): my global_id is 4201 2024-03-25T18:26:53.773 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:26:53.768+0000 7ff17affd640 1 -- 172.21.15.1:0/2301587571 --> v1:172.21.15.115:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ff164004eb0 con 0x7ff1840b4ff0 2024-03-25T18:26:53.773 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:26:53.768+0000 7ff17affd640 1 -- 172.21.15.1:0/2301587571 <== mon.0 v1:172.21.15.1:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 722361189 0 0) 0x7ff1840b0580 con 0x7ff18413c480 2024-03-25T18:26:53.774 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:26:53.768+0000 7ff17affd640 10 monclient(hunting): _negotiate 2024-03-25T18:26:53.774 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:26:53.768+0000 7ff17affd640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:26:53.774 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:26:53.768+0000 7ff17affd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:26:53.774 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:26:53.768+0000 7ff17affd640 10 monclient(hunting): my global_id is 4203 2024-03-25T18:26:53.774 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:26:53.768+0000 7ff17affd640 1 -- 172.21.15.1:0/2301587571 --> v1:172.21.15.1:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ff1840b07f0 con 0x7ff18413c480 2024-03-25T18:26:53.774 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:26:53.768+0000 7ff17affd640 1 -- 172.21.15.1:0/2301587571 <== mon.2 v1:172.21.15.1:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3042782787 0 0) 0x7ff1840b0ad0 con 0x7ff1840b4670 2024-03-25T18:26:53.774 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:26:53.768+0000 7ff17affd640 10 monclient(hunting): _negotiate 2024-03-25T18:26:53.774 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:26:53.768+0000 7ff17affd640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:26:53.774 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:26:53.768+0000 7ff17affd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:26:53.774 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:26:53.768+0000 7ff17affd640 10 monclient(hunting): my global_id is 4208 2024-03-25T18:26:53.774 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:26:53.768+0000 7ff17affd640 1 -- 172.21.15.1:0/2301587571 --> v1:172.21.15.1:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ff1840b0580 con 0x7ff1840b4670 2024-03-25T18:26:53.776 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:26:53.772+0000 7ff17affd640 1 -- 172.21.15.1:0/2301587571 <== mon.1 v1:172.21.15.115:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2436378210 0 0) 0x7ff164004eb0 con 0x7ff1840b4ff0 2024-03-25T18:26:53.776 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:26:53.772+0000 7ff17affd640 1 -- 172.21.15.1:0/2301587571 --> v1:172.21.15.115:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7ff1840b0ad0 con 0x7ff1840b4ff0 2024-03-25T18:26:53.776 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:26:53.772+0000 7ff17affd640 1 -- 172.21.15.1:0/2301587571 <== mon.1 v1:172.21.15.115:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1586965025 0 0) 0x7ff16c001970 con 0x7ff1840b4ff0 2024-03-25T18:26:53.776 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:26:53.772+0000 7ff17affd640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:26:53.776 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:26:53.772+0000 7ff17affd640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-03-25T18:26:53.776 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:26:53.772+0000 7ff17affd640 10 monclient(hunting): dump: 2024-03-25T18:26:53.776 INFO:teuthology.misc.health.smithi001.stderr:epoch 1 2024-03-25T18:26:53.776 INFO:teuthology.misc.health.smithi001.stderr:fsid c8ea72cf-9e99-4383-9b52-6c23d365fdfc 2024-03-25T18:26:53.776 INFO:teuthology.misc.health.smithi001.stderr:last_changed 2024-03-25T18:25:50.510519+0000 2024-03-25T18:26:53.776 INFO:teuthology.misc.health.smithi001.stderr:created 2024-03-25T18:25:50.510519+0000 2024-03-25T18:26:53.776 INFO:teuthology.misc.health.smithi001.stderr:min_mon_release 19 (squid) 2024-03-25T18:26:53.776 INFO:teuthology.misc.health.smithi001.stderr:election_strategy: 3 2024-03-25T18:26:53.777 INFO:teuthology.misc.health.smithi001.stderr:0: v1:172.21.15.1:6789/0 mon.a 2024-03-25T18:26:53.777 INFO:teuthology.misc.health.smithi001.stderr:1: v1:172.21.15.115:6789/0 mon.b 2024-03-25T18:26:53.777 INFO:teuthology.misc.health.smithi001.stderr:2: v1:172.21.15.1:6790/0 mon.c 2024-03-25T18:26:53.777 INFO:teuthology.misc.health.smithi001.stderr: 2024-03-25T18:26:53.777 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:26:53.772+0000 7ff17affd640 10 monclient(hunting): _finish_auth 0 2024-03-25T18:26:53.777 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:26:53.773+0000 7ff17affd640 1 -- 172.21.15.1:0/2301587571 <== mon.0 v1:172.21.15.1:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 214709482 0 0) 0x7ff1840b07f0 con 0x7ff18413c480 2024-03-25T18:26:53.777 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:26:53.773+0000 7ff17affd640 1 -- 172.21.15.1:0/2301587571 --> v1:172.21.15.1:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7ff164004eb0 con 0x7ff18413c480 2024-03-25T18:26:53.777 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:26:53.773+0000 7ff17affd640 1 -- 172.21.15.1:0/2301587571 <== mon.0 v1:172.21.15.1:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1586965025 0 0) 0x7ff174001bc0 con 0x7ff18413c480 2024-03-25T18:26:53.777 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:26:53.773+0000 7ff17affd640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:26:53.777 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:26:53.773+0000 7ff17affd640 10 monclient(hunting): got monmap 1 from mon.a (according to old e1) 2024-03-25T18:26:53.777 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:26:53.773+0000 7ff17affd640 10 monclient(hunting): dump: 2024-03-25T18:26:53.777 INFO:teuthology.misc.health.smithi001.stderr:epoch 1 2024-03-25T18:26:53.778 INFO:teuthology.misc.health.smithi001.stderr:fsid c8ea72cf-9e99-4383-9b52-6c23d365fdfc 2024-03-25T18:26:53.778 INFO:teuthology.misc.health.smithi001.stderr:last_changed 2024-03-25T18:25:50.510519+0000 2024-03-25T18:26:53.778 INFO:teuthology.misc.health.smithi001.stderr:created 2024-03-25T18:25:50.510519+0000 2024-03-25T18:26:53.778 INFO:teuthology.misc.health.smithi001.stderr:min_mon_release 19 (squid) 2024-03-25T18:26:53.778 INFO:teuthology.misc.health.smithi001.stderr:election_strategy: 3 2024-03-25T18:26:53.778 INFO:teuthology.misc.health.smithi001.stderr:0: v1:172.21.15.1:6789/0 mon.a 2024-03-25T18:26:53.778 INFO:teuthology.misc.health.smithi001.stderr:1: v1:172.21.15.115:6789/0 mon.b 2024-03-25T18:26:53.778 INFO:teuthology.misc.health.smithi001.stderr:2: v1:172.21.15.1:6790/0 mon.c 2024-03-25T18:26:53.778 INFO:teuthology.misc.health.smithi001.stderr: 2024-03-25T18:26:53.778 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:26:53.773+0000 7ff17affd640 1 -- 172.21.15.1:0/2301587571 <== mon.2 v1:172.21.15.1:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 215338395 0 0) 0x7ff1840b0580 con 0x7ff1840b4670 2024-03-25T18:26:53.779 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:26:53.773+0000 7ff17affd640 1 -- 172.21.15.1:0/2301587571 --> v1:172.21.15.1:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7ff1840b07f0 con 0x7ff1840b4670 2024-03-25T18:26:53.779 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:26:53.773+0000 7ff17affd640 1 -- 172.21.15.1:0/2301587571 <== mon.2 v1:172.21.15.1:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1586965025 0 0) 0x7ff17c001bb0 con 0x7ff1840b4670 2024-03-25T18:26:53.779 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:26:53.773+0000 7ff17affd640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:26:53.779 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:26:53.773+0000 7ff17affd640 10 monclient(hunting): got monmap 1 from mon.c (according to old e1) 2024-03-25T18:26:53.779 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:26:53.773+0000 7ff17affd640 10 monclient(hunting): dump: 2024-03-25T18:26:53.779 INFO:teuthology.misc.health.smithi001.stderr:epoch 1 2024-03-25T18:26:53.779 INFO:teuthology.misc.health.smithi001.stderr:fsid c8ea72cf-9e99-4383-9b52-6c23d365fdfc 2024-03-25T18:26:53.779 INFO:teuthology.misc.health.smithi001.stderr:last_changed 2024-03-25T18:25:50.510519+0000 2024-03-25T18:26:53.779 INFO:teuthology.misc.health.smithi001.stderr:created 2024-03-25T18:25:50.510519+0000 2024-03-25T18:26:53.779 INFO:teuthology.misc.health.smithi001.stderr:min_mon_release 19 (squid) 2024-03-25T18:26:53.779 INFO:teuthology.misc.health.smithi001.stderr:election_strategy: 3 2024-03-25T18:26:53.779 INFO:teuthology.misc.health.smithi001.stderr:0: v1:172.21.15.1:6789/0 mon.a 2024-03-25T18:26:53.780 INFO:teuthology.misc.health.smithi001.stderr:1: v1:172.21.15.115:6789/0 mon.b 2024-03-25T18:26:53.780 INFO:teuthology.misc.health.smithi001.stderr:2: v1:172.21.15.1:6790/0 mon.c 2024-03-25T18:26:53.780 INFO:teuthology.misc.health.smithi001.stderr: 2024-03-25T18:26:53.781 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:26:53.775+0000 7ff17affd640 1 -- 172.21.15.1:0/2301587571 <== mon.1 v1:172.21.15.115:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 2006359211 0 0) 0x7ff1840b0ad0 con 0x7ff1840b4ff0 2024-03-25T18:26:53.781 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:26:53.775+0000 7ff17affd640 1 -- 172.21.15.1:0/2301587571 >> v1:172.21.15.1:6790/0 conn(0x7ff1840b4670 legacy=0x7ff1840ae6e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:26:53.781 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:26:53.775+0000 7ff17affd640 1 -- 172.21.15.1:0/2301587571 >> v1:172.21.15.1:6789/0 conn(0x7ff18413c480 legacy=0x7ff1841e2380 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:26:53.781 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:26:53.775+0000 7ff17affd640 10 monclient: _finish_hunting 0 2024-03-25T18:26:53.781 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:26:53.775+0000 7ff17affd640 1 monclient: found mon.b 2024-03-25T18:26:53.781 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:26:53.775+0000 7ff17affd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:26:53.782 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:26:53.775+0000 7ff17affd640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.115:6789/0 2024-03-25T18:26:53.782 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:26:53.775+0000 7ff17affd640 1 -- 172.21.15.1:0/2301587571 --> v1:172.21.15.115:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff1840b1360 con 0x7ff1840b4ff0 2024-03-25T18:26:53.782 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:26:53.775+0000 7ff17affd640 10 monclient: _finish_auth 0 2024-03-25T18:26:53.782 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:26:53.775+0000 7ff17affd640 10 monclient: _check_auth_tickets 2024-03-25T18:26:53.782 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:26:53.775+0000 7ff17affd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:26:53.782 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:26:53.775+0000 7ff188e2e640 5 monclient: authenticate success, global_id 4201 2024-03-25T18:26:53.782 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:26:53.775+0000 7ff188e2e640 10 monclient: _renew_subs 2024-03-25T18:26:53.782 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:26:53.775+0000 7ff188e2e640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.115:6789/0 2024-03-25T18:26:53.782 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:26:53.775+0000 7ff188e2e640 1 -- 172.21.15.1:0/2301587571 --> v1:172.21.15.115:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff1840b1880 con 0x7ff1840b4ff0 2024-03-25T18:26:53.782 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:26:53.776+0000 7ff188e2e640 10 client.4201.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T18:26:53.782 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:26:53.776+0000 7ff188e2e640 10 monclient: _renew_subs 2024-03-25T18:26:53.783 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:26:53.776+0000 7ff188e2e640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.115:6789/0 2024-03-25T18:26:53.783 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:26:53.776+0000 7ff188e2e640 1 -- 172.21.15.1:0/2301587571 --> v1:172.21.15.115:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7ff1840b10d0 con 0x7ff1840b4ff0 2024-03-25T18:26:53.783 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:26:53.776+0000 7ff188e2e640 1 librados: init done 2024-03-25T18:26:53.783 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:26:53.776+0000 7ff188e2e640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T18:26:53.783 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:26:53.776+0000 7ff188e2e640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:26:53.783 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:26:53.776+0000 7ff188e2e640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.115:6789/0 2024-03-25T18:26:53.783 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:26:53.776+0000 7ff188e2e640 1 -- 172.21.15.1:0/2301587571 --> v1:172.21.15.115:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff18406dbd0 con 0x7ff1840b4ff0 2024-03-25T18:26:53.794 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:26:53.793+0000 7ff17affd640 1 -- 172.21.15.1:0/2301587571 <== mon.1 v1:172.21.15.115:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7ff16c0047e0 con 0x7ff1840b4ff0 2024-03-25T18:26:53.794 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:26:53.793+0000 7ff17affd640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:26:53.795 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:26:53.793+0000 7ff17affd640 1 -- 172.21.15.1:0/2301587571 <== mon.1 v1:172.21.15.115:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1586965025 0 0) 0x7ff16c004af0 con 0x7ff1840b4ff0 2024-03-25T18:26:53.795 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:26:53.793+0000 7ff17affd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:26:53.795 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:26:53.793+0000 7ff17affd640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-25T18:26:53.795 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:26:53.793+0000 7ff17affd640 10 monclient: dump: 2024-03-25T18:26:53.795 INFO:teuthology.misc.health.smithi001.stderr:epoch 1 2024-03-25T18:26:53.795 INFO:teuthology.misc.health.smithi001.stderr:fsid c8ea72cf-9e99-4383-9b52-6c23d365fdfc 2024-03-25T18:26:53.795 INFO:teuthology.misc.health.smithi001.stderr:last_changed 2024-03-25T18:25:50.510519+0000 2024-03-25T18:26:53.795 INFO:teuthology.misc.health.smithi001.stderr:created 2024-03-25T18:25:50.510519+0000 2024-03-25T18:26:53.795 INFO:teuthology.misc.health.smithi001.stderr:min_mon_release 19 (squid) 2024-03-25T18:26:53.795 INFO:teuthology.misc.health.smithi001.stderr:election_strategy: 3 2024-03-25T18:26:53.795 INFO:teuthology.misc.health.smithi001.stderr:0: v1:172.21.15.1:6789/0 mon.a 2024-03-25T18:26:53.795 INFO:teuthology.misc.health.smithi001.stderr:1: v1:172.21.15.115:6789/0 mon.b 2024-03-25T18:26:53.795 INFO:teuthology.misc.health.smithi001.stderr:2: v1:172.21.15.1:6790/0 mon.c 2024-03-25T18:26:53.795 INFO:teuthology.misc.health.smithi001.stderr: 2024-03-25T18:26:53.796 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:26:53.793+0000 7ff180d99640 4 set_mon_vals no callback set 2024-03-25T18:26:53.796 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:26:53.794+0000 7ff17affd640 1 -- 172.21.15.1:0/2301587571 <== mon.1 v1:172.21.15.115:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 21787975 0 0) 0x7ff16c01ab00 con 0x7ff1840b4ff0 2024-03-25T18:26:53.796 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:26:53.794+0000 7ff18259c640 10 monclient: get_auth_request con 0x7ff16406e0b0 auth_method 0 2024-03-25T18:26:53.796 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:26:53.795+0000 7ff17affd640 10 client.4201.objecter ms_handle_connect 0x7ff16406e0b0 2024-03-25T18:26:53.797 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:26:53.796+0000 7ff17affd640 1 -- 172.21.15.1:0/2301587571 <== mon.1 v1:172.21.15.115:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 996254230 0 0) 0x7ff16c054740 con 0x7ff1840b4ff0 2024-03-25T18:26:53.797 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:26:53.796+0000 7ff17affd640 10 client.4201.objecter ms_dispatch 0x7ff184000b90 osd_map(10..10 src has 1..10) v4 2024-03-25T18:26:53.797 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:26:53.796+0000 7ff17affd640 3 client.4201.objecter handle_osd_map got epochs [10,10] > 0 2024-03-25T18:26:53.797 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:26:53.796+0000 7ff17affd640 3 client.4201.objecter handle_osd_map decoding full epoch 10 2024-03-25T18:26:53.797 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:26:53.796+0000 7ff17affd640 20 client.4201.objecter dump_active .. 0 homeless 2024-03-25T18:26:53.810 INFO:tasks.ceph.osd.2.smithi001.stderr:==00:00:00:09.402 35192== Warning: unimplemented fcntl command: 1038 2024-03-25T18:26:53.812 INFO:tasks.ceph.osd.7.smithi115.stderr:==00:00:00:09.424 35383== Warning: unimplemented fcntl command: 1038 2024-03-25T18:26:53.853 INFO:tasks.ceph.osd.4.smithi115.stderr:==00:00:00:09.471 35380== Warning: unimplemented fcntl command: 1038 2024-03-25T18:26:53.881 INFO:tasks.ceph.osd.6.smithi115.stderr:==00:00:00:09.497 35382== Warning: unimplemented fcntl command: 1038 2024-03-25T18:26:53.886 INFO:tasks.ceph.osd.3.smithi001.stderr:==00:00:00:09.488 35191== Warning: unimplemented fcntl command: 1038 2024-03-25T18:26:53.908 INFO:tasks.ceph.osd.5.smithi115.stderr:==00:00:00:09.525 35381== Warning: unimplemented fcntl command: 1038 2024-03-25T18:26:53.926 INFO:tasks.ceph.osd.0.smithi001.stderr:==00:00:00:09.533 35190== Warning: unimplemented fcntl command: 1038 2024-03-25T18:26:54.032 INFO:tasks.ceph.osd.1.smithi001.stderr:==00:00:00:09.649 35189== Warning: unimplemented fcntl command: 1038 2024-03-25T18:26:54.042 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:26:54.041+0000 7ff17affd640 1 -- 172.21.15.1:0/2301587571 <== mon.1 v1:172.21.15.115:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7ff16c050080 con 0x7ff1840b4ff0 2024-03-25T18:26:54.042 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:26:54.041+0000 7ff17affd640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:26:54.042 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:26:54.041+0000 7ff17affd640 10 monclient: _finish_command 1 = system:0 2024-03-25T18:26:54.081 INFO:tasks.ceph.osd.7.smithi115.stderr:==00:00:00:09.693 35383== Warning: unimplemented fcntl command: 1038 2024-03-25T18:26:54.102 INFO:tasks.ceph.osd.2.smithi001.stderr:==00:00:00:09.695 35192== Warning: unimplemented fcntl command: 1038 2024-03-25T18:26:54.145 INFO:tasks.ceph.osd.6.smithi115.stderr:==00:00:00:09.762 35382== Warning: unimplemented fcntl command: 1038 2024-03-25T18:26:54.153 INFO:tasks.ceph.osd.3.smithi001.stderr:==00:00:00:09.756 35191== Warning: unimplemented fcntl command: 1038 2024-03-25T18:26:54.169 INFO:tasks.ceph.osd.4.smithi115.stderr:==00:00:00:09.786 35380== Warning: unimplemented fcntl command: 1038 2024-03-25T18:26:54.196 INFO:tasks.ceph.osd.5.smithi115.stderr:==00:00:00:09.812 35381== Warning: unimplemented fcntl command: 1038 2024-03-25T18:26:54.198 INFO:tasks.ceph.osd.0.smithi001.stderr:==00:00:00:09.805 35190== Warning: unimplemented fcntl command: 1038 2024-03-25T18:26:54.227 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:26:54.225+0000 7ff188e2e640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:26:54.227 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:26:54.225+0000 7ff188e2e640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:26:54.227 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:26:54.225+0000 7ff188e2e640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.115:6789/0 2024-03-25T18:26:54.227 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:26:54.225+0000 7ff188e2e640 1 -- 172.21.15.1:0/2301587571 --> v1:172.21.15.115:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7ff1840688c0 con 0x7ff1840b4ff0 2024-03-25T18:26:54.270 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:26:54.269+0000 7ff17affd640 1 -- 172.21.15.1:0/2301587571 <== mon.1 v1:172.21.15.115:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 4092213348) 0x7ff16c052180 con 0x7ff1840b4ff0 2024-03-25T18:26:54.270 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:26:54.269+0000 7ff17affd640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:26:54.270 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:26:54.269+0000 7ff17affd640 10 monclient: _finish_command 2 = system:0 2024-03-25T18:26:54.270 INFO:teuthology.misc.health.smithi001.stdout: 2024-03-25T18:26:54.270 INFO:teuthology.misc.health.smithi001.stdout:{"epoch":10,"fsid":"c8ea72cf-9e99-4383-9b52-6c23d365fdfc","created":"2024-03-25T18:26:38.856638+0000","modified":"2024-03-25T18:26:44.224208+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T18:26:44.224208+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"b9f34922-3a84-49be-8426-807c1eceaefc","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"42bf306f-55e3-4634-9874-8b4b73ac6644","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"292cb23a-18a9-4207-a7cc-4d5727feaec0","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"45e0e9ab-898d-4673-9590-b4117bc5e84a","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"831a5a09-73ab-4a61-9872-088e76bdbc66","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"88cd903d-3b55-4840-9396-6e3cfe005282","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"89db7355-bc9d-46f4-8ab0-e8d9e744e026","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"cac47bee-0914-4f08-896f-348838882866","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T18:26:54.273 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:26:54.270+0000 7ff188e2e640 10 librados: watch_flush enter 2024-03-25T18:26:54.273 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:26:54.270+0000 7ff188e2e640 10 librados: watch_flush exit 2024-03-25T18:26:54.273 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:26:54.270+0000 7ff188e2e640 20 client.4201.objecter shutdown clearing up homeless session... 2024-03-25T18:26:54.273 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:26:54.270+0000 7ff188e2e640 10 client.4201.objecter successfully canceled tick 2024-03-25T18:26:54.273 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:26:54.270+0000 7ff188e2e640 1 -- 172.21.15.1:0/2301587571 >> v1:172.21.15.115:6800/4237379176 conn(0x7ff16406e0b0 legacy=0x7ff164070570 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:26:54.273 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:26:54.270+0000 7ff188e2e640 10 monclient: shutdown 2024-03-25T18:26:54.273 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:26:54.270+0000 7ff188e2e640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:26:54.273 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:26:54.270+0000 7ff188e2e640 1 -- 172.21.15.1:0/2301587571 >> v1:172.21.15.115:6789/0 conn(0x7ff1840b4ff0 legacy=0x7ff1840aef10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:26:54.273 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:26:54.270+0000 7ff188e2e640 1 -- 172.21.15.1:0/2301587571 shutdown_connections 2024-03-25T18:26:54.273 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:26:54.270+0000 7ff188e2e640 1 -- 172.21.15.1:0/2301587571 >> 172.21.15.1:0/2301587571 conn(0x7ff1840617a0 msgr2=0x7ff1841407f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:26:54.273 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:26:54.271+0000 7ff188e2e640 1 -- 172.21.15.1:0/2301587571 shutdown_connections 2024-03-25T18:26:54.273 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:26:54.271+0000 7ff188e2e640 1 -- 172.21.15.1:0/2301587571 wait complete. 2024-03-25T18:26:54.273 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:26:54.271+0000 7ff188e2e640 1 librados: shutdown 2024-03-25T18:26:54.281 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T18:26:54.322 INFO:tasks.ceph.osd.1.smithi001.stderr:==00:00:00:09.938 35189== Warning: unimplemented fcntl command: 1038 2024-03-25T18:26:54.353 INFO:tasks.ceph.osd.7.smithi115.stderr:==00:00:00:09.965 35383== Warning: unimplemented fcntl command: 1038 2024-03-25T18:26:54.396 INFO:tasks.ceph.osd.2.smithi001.stderr:==00:00:00:09.988 35192== Warning: unimplemented fcntl command: 1038 2024-03-25T18:26:54.414 INFO:tasks.ceph.osd.6.smithi115.stderr:==00:00:00:10.030 35382== Warning: unimplemented fcntl command: 1038 2024-03-25T18:26:54.430 INFO:tasks.ceph.osd.3.smithi001.stderr:==00:00:00:10.032 35191== Warning: unimplemented fcntl command: 1038 2024-03-25T18:26:54.457 INFO:tasks.ceph.osd.4.smithi115.stderr:==00:00:00:10.074 35380== Warning: unimplemented fcntl command: 1038 2024-03-25T18:26:54.465 INFO:tasks.ceph.osd.0.smithi001.stderr:==00:00:00:10.071 35190== Warning: unimplemented fcntl command: 1038 2024-03-25T18:26:54.492 INFO:tasks.ceph.osd.5.smithi115.stderr:==00:00:00:10.109 35381== Warning: unimplemented fcntl command: 1038 2024-03-25T18:26:54.612 INFO:tasks.ceph.osd.1.smithi001.stderr:==00:00:00:10.229 35189== Warning: unimplemented fcntl command: 1038 2024-03-25T18:26:54.617 INFO:tasks.ceph.osd.7.smithi115.stderr:==00:00:00:10.229 35383== Warning: unimplemented fcntl command: 1038 2024-03-25T18:26:54.678 INFO:tasks.ceph.osd.6.smithi115.stderr:==00:00:00:10.294 35382== Warning: unimplemented fcntl command: 1038 2024-03-25T18:26:54.697 INFO:tasks.ceph.osd.2.smithi001.stderr:==00:00:00:10.289 35192== Warning: unimplemented fcntl command: 1038 2024-03-25T18:26:54.700 INFO:tasks.ceph.osd.3.smithi001.stderr:==00:00:00:10.303 35191== Warning: unimplemented fcntl command: 1038 2024-03-25T18:26:54.735 INFO:tasks.ceph.osd.0.smithi001.stderr:==00:00:00:10.342 35190== Warning: unimplemented fcntl command: 1038 2024-03-25T18:26:54.745 INFO:tasks.ceph.osd.4.smithi115.stderr:==00:00:00:10.362 35380== Warning: unimplemented fcntl command: 1038 2024-03-25T18:26:54.787 INFO:tasks.ceph.osd.5.smithi115.stderr:==00:00:00:10.404 35381== Warning: unimplemented fcntl command: 1038 2024-03-25T18:26:54.881 INFO:tasks.ceph.osd.7.smithi115.stderr:==00:00:00:10.493 35383== Warning: unimplemented fcntl command: 1038 2024-03-25T18:26:54.901 INFO:tasks.ceph.osd.1.smithi001.stderr:==00:00:00:10.518 35189== Warning: unimplemented fcntl command: 1038 2024-03-25T18:26:54.965 INFO:tasks.ceph.osd.3.smithi001.stderr:==00:00:00:10.567 35191== Warning: unimplemented fcntl command: 1038 2024-03-25T18:26:54.989 INFO:tasks.ceph.osd.2.smithi001.stderr:==00:00:00:10.582 35192== Warning: unimplemented fcntl command: 1038 2024-03-25T18:26:55.004 INFO:tasks.ceph.osd.0.smithi001.stderr:==00:00:00:10.611 35190== Warning: unimplemented fcntl command: 1038 2024-03-25T18:26:55.190 INFO:tasks.ceph.osd.1.smithi001.stderr:==00:00:00:10.807 35189== Warning: unimplemented fcntl command: 1038 2024-03-25T18:26:55.197 INFO:tasks.ceph.osd.6.smithi115.stderr:==00:00:00:10.813 35382== Warning: unimplemented fcntl command: 1038 2024-03-25T18:26:55.229 INFO:tasks.ceph.osd.3.smithi001.stderr:==00:00:00:10.831 35191== Warning: unimplemented fcntl command: 1038 2024-03-25T18:26:55.252 INFO:tasks.ceph.osd.6.smithi115.stderr:==00:00:00:10.868 35382== Warning: unimplemented fcntl command: 1038 2024-03-25T18:26:55.271 INFO:tasks.ceph.osd.4.smithi115.stderr:==00:00:00:10.888 35380== Warning: unimplemented fcntl command: 1038 2024-03-25T18:26:55.271 INFO:tasks.ceph.osd.0.smithi001.stderr:==00:00:00:10.879 35190== Warning: unimplemented fcntl command: 1038 2024-03-25T18:26:55.289 INFO:tasks.ceph.osd.2.smithi001.stderr:==00:00:00:10.882 35192== Warning: unimplemented fcntl command: 1038 2024-03-25T18:26:55.328 INFO:tasks.ceph.osd.4.smithi115.stderr:==00:00:00:10.945 35380== Warning: unimplemented fcntl command: 1038 2024-03-25T18:26:55.346 INFO:tasks.ceph.osd.5.smithi115.stderr:==00:00:00:10.963 35381== Warning: unimplemented fcntl command: 1038 2024-03-25T18:26:55.363 INFO:tasks.ceph.osd.7.smithi115.stderr:==00:00:00:10.975 35383== Warning: unimplemented fcntl command: 1038 2024-03-25T18:26:55.401 INFO:tasks.ceph.osd.5.smithi115.stderr:==00:00:00:11.018 35381== Warning: unimplemented fcntl command: 1038 2024-03-25T18:26:55.417 INFO:tasks.ceph.osd.7.smithi115.stderr:==00:00:00:11.029 35383== Warning: unimplemented fcntl command: 1038 2024-03-25T18:26:55.479 INFO:tasks.ceph.osd.1.smithi001.stderr:==00:00:00:11.096 35189== Warning: unimplemented fcntl command: 1038 2024-03-25T18:26:55.547 INFO:tasks.ceph.osd.0.smithi001.stderr:==00:00:00:11.154 35190== Warning: unimplemented fcntl command: 1038 2024-03-25T18:26:55.583 INFO:tasks.ceph.osd.2.smithi001.stderr:==00:00:00:11.175 35192== Warning: unimplemented fcntl command: 1038 2024-03-25T18:26:55.719 INFO:tasks.ceph.osd.3.smithi001.stderr:==00:00:00:11.321 35191== Warning: unimplemented fcntl command: 1038 2024-03-25T18:26:55.774 INFO:tasks.ceph.osd.3.smithi001.stderr:==00:00:00:11.376 35191== Warning: unimplemented fcntl command: 1038 2024-03-25T18:26:56.004 INFO:tasks.ceph.osd.1.smithi001.stderr:==00:00:00:11.620 35189== Warning: unimplemented fcntl command: 1038 2024-03-25T18:26:56.029 INFO:tasks.ceph.osd.0.smithi001.stderr:==00:00:00:11.636 35190== Warning: unimplemented fcntl command: 1038 2024-03-25T18:26:56.060 INFO:tasks.ceph.osd.1.smithi001.stderr:==00:00:00:11.677 35189== Warning: unimplemented fcntl command: 1038 2024-03-25T18:26:56.087 INFO:tasks.ceph.osd.0.smithi001.stderr:==00:00:00:11.694 35190== Warning: unimplemented fcntl command: 1038 2024-03-25T18:26:56.143 INFO:tasks.ceph.osd.2.smithi001.stderr:==00:00:00:11.735 35192== Warning: unimplemented fcntl command: 1038 2024-03-25T18:26:56.227 INFO:tasks.ceph.osd.2.smithi001.stderr:==00:00:00:11.820 35192== Warning: unimplemented fcntl command: 1038 2024-03-25T18:27:00.825 INFO:tasks.ceph.osd.5.smithi115.stderr:==00:00:00:16.442 35381== Warning: unimplemented fcntl command: 1038 2024-03-25T18:27:00.893 INFO:tasks.ceph.osd.4.smithi115.stderr:==00:00:00:16.510 35380== Warning: unimplemented fcntl command: 1038 2024-03-25T18:27:00.946 INFO:tasks.ceph.osd.7.smithi115.stderr:==00:00:00:16.558 35383== Warning: unimplemented fcntl command: 1038 2024-03-25T18:27:00.965 INFO:tasks.ceph.osd.6.smithi115.stderr:==00:00:00:16.581 35382== Warning: unimplemented fcntl command: 1038 2024-03-25T18:27:01.088 DEBUG:teuthology.orchestra.run.smithi001:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T18:27:01.194 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:27:01.193+0000 7fa75df56640 10 monclient: get_monmap_and_config 2024-03-25T18:27:01.194 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:27:01.193+0000 7fa75df56640 10 monclient: build_initial_monmap 2024-03-25T18:27:01.194 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:27:01.193+0000 7fa75df56640 10 monclient: monmap: 2024-03-25T18:27:01.194 INFO:teuthology.misc.health.smithi001.stderr:epoch 0 2024-03-25T18:27:01.194 INFO:teuthology.misc.health.smithi001.stderr:fsid c8ea72cf-9e99-4383-9b52-6c23d365fdfc 2024-03-25T18:27:01.194 INFO:teuthology.misc.health.smithi001.stderr:last_changed 2024-03-25T18:27:01.194060+0000 2024-03-25T18:27:01.194 INFO:teuthology.misc.health.smithi001.stderr:created 2024-03-25T18:27:01.194060+0000 2024-03-25T18:27:01.194 INFO:teuthology.misc.health.smithi001.stderr:min_mon_release 0 (unknown) 2024-03-25T18:27:01.195 INFO:teuthology.misc.health.smithi001.stderr:election_strategy: 3 2024-03-25T18:27:01.195 INFO:teuthology.misc.health.smithi001.stderr:0: v1:172.21.15.1:6789/0 mon.noname-a 2024-03-25T18:27:01.195 INFO:teuthology.misc.health.smithi001.stderr:1: v1:172.21.15.115:6789/0 mon.noname-c 2024-03-25T18:27:01.195 INFO:teuthology.misc.health.smithi001.stderr:2: v1:172.21.15.1:6790/0 mon.noname-b 2024-03-25T18:27:01.195 INFO:teuthology.misc.health.smithi001.stderr: 2024-03-25T18:27:01.197 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:27:01.194+0000 7fa75df56640 1 Processor -- start 2024-03-25T18:27:01.197 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:27:01.194+0000 7fa75df56640 1 -- start start 2024-03-25T18:27:01.197 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:27:01.194+0000 7fa75df56640 10 monclient: init 2024-03-25T18:27:01.197 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:27:01.194+0000 7fa75df56640 10 monclient: _reopen_session rank -1 2024-03-25T18:27:01.197 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:27:01.194+0000 7fa75df56640 10 monclient: _add_conns ranks=[1,0,2] 2024-03-25T18:27:01.197 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:27:01.194+0000 7fa75df56640 10 monclient(hunting): picked mon.noname-c con 0x7fa7580a98d0 addr v1:172.21.15.115:6789/0 2024-03-25T18:27:01.197 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:27:01.194+0000 7fa75df56640 10 monclient(hunting): picked mon.noname-a con 0x7fa7580b2a60 addr v1:172.21.15.1:6789/0 2024-03-25T18:27:01.197 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:27:01.194+0000 7fa75df56640 10 monclient(hunting): picked mon.noname-b con 0x7fa75814bc00 addr v1:172.21.15.1:6790/0 2024-03-25T18:27:01.197 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:27:01.195+0000 7fa75df56640 1 -- --> v1:172.21.15.1:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa7580b2e80 con 0x7fa7580b2a60 2024-03-25T18:27:01.198 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:27:01.195+0000 7fa75df56640 1 -- --> v1:172.21.15.115:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa7581431c0 con 0x7fa7580a98d0 2024-03-25T18:27:01.198 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:27:01.195+0000 7fa75df56640 1 -- --> v1:172.21.15.1:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa7581422f0 con 0x7fa75814bc00 2024-03-25T18:27:01.198 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:27:01.195+0000 7fa75df56640 10 monclient(hunting): _renew_subs 2024-03-25T18:27:01.198 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:27:01.195+0000 7fa75df56640 10 monclient(hunting): authenticate will time out at 7501.664690s 2024-03-25T18:27:01.198 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:27:01.195+0000 7fa756ffd640 1 --1- >> v1:172.21.15.1:6789/0 conn(0x7fa7580b2a60 0x7fa75813f440 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.1:6789/0 says I am v1:172.21.15.1:51260/0 (socket says 172.21.15.1:51260) 2024-03-25T18:27:01.198 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:27:01.195+0000 7fa756ffd640 1 -- 172.21.15.1:0/343725758 learned_addr learned my addr 172.21.15.1:0/343725758 (peer_addr_for_me v1:172.21.15.1:0/0) 2024-03-25T18:27:01.198 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:27:01.197+0000 7fa7567fc640 1 -- 172.21.15.1:0/343725758 <== mon.0 v1:172.21.15.1:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3857300441 0 0) 0x7fa7580b2e80 con 0x7fa7580b2a60 2024-03-25T18:27:01.198 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:27:01.197+0000 7fa7567fc640 10 monclient(hunting): _negotiate 2024-03-25T18:27:01.199 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:27:01.197+0000 7fa7567fc640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:27:01.199 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:27:01.197+0000 7fa7567fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:27:01.199 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:27:01.197+0000 7fa7567fc640 10 monclient(hunting): my global_id is 4206 2024-03-25T18:27:01.199 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:27:01.197+0000 7fa7567fc640 1 -- 172.21.15.1:0/343725758 --> v1:172.21.15.1:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fa740003520 con 0x7fa7580b2a60 2024-03-25T18:27:01.199 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:27:01.199+0000 7fa7567fc640 1 -- 172.21.15.1:0/343725758 <== mon.1 v1:172.21.15.115:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3025199986 0 0) 0x7fa7581431c0 con 0x7fa7580a98d0 2024-03-25T18:27:01.199 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:27:01.199+0000 7fa7567fc640 10 monclient(hunting): _negotiate 2024-03-25T18:27:01.199 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:27:01.199+0000 7fa7567fc640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:27:01.199 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:27:01.199+0000 7fa7567fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:27:01.199 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:27:01.199+0000 7fa7567fc640 10 monclient(hunting): my global_id is 4204 2024-03-25T18:27:01.200 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:27:01.199+0000 7fa7567fc640 1 -- 172.21.15.1:0/343725758 --> v1:172.21.15.115:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fa7580b2e80 con 0x7fa7580a98d0 2024-03-25T18:27:01.200 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:27:01.200+0000 7fa7567fc640 1 -- 172.21.15.1:0/343725758 <== mon.2 v1:172.21.15.1:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3652416953 0 0) 0x7fa7581422f0 con 0x7fa75814bc00 2024-03-25T18:27:01.200 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:27:01.200+0000 7fa7567fc640 10 monclient(hunting): _negotiate 2024-03-25T18:27:01.200 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:27:01.200+0000 7fa7567fc640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:27:01.200 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:27:01.200+0000 7fa7567fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:27:01.200 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:27:01.200+0000 7fa7567fc640 10 monclient(hunting): my global_id is 4211 2024-03-25T18:27:01.200 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:27:01.200+0000 7fa7567fc640 1 -- 172.21.15.1:0/343725758 --> v1:172.21.15.1:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fa7581431c0 con 0x7fa75814bc00 2024-03-25T18:27:01.203 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:27:01.202+0000 7fa7567fc640 1 -- 172.21.15.1:0/343725758 <== mon.1 v1:172.21.15.115:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 4185833441 0 0) 0x7fa7580b2e80 con 0x7fa7580a98d0 2024-03-25T18:27:01.203 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:27:01.202+0000 7fa7567fc640 1 -- 172.21.15.1:0/343725758 >> v1:172.21.15.1:6790/0 conn(0x7fa75814bc00 legacy=0x7fa75814dff0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:27:01.203 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:27:01.202+0000 7fa7567fc640 1 -- 172.21.15.1:0/343725758 >> v1:172.21.15.1:6789/0 conn(0x7fa7580b2a60 legacy=0x7fa75813f440 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:27:01.203 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:27:01.202+0000 7fa7567fc640 10 monclient: _finish_hunting 0 2024-03-25T18:27:01.203 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:27:01.202+0000 7fa7567fc640 1 monclient: found mon.noname-c 2024-03-25T18:27:01.203 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:27:01.202+0000 7fa7567fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:27:01.203 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:27:01.202+0000 7fa7567fc640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.115:6789/0 2024-03-25T18:27:01.203 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:27:01.202+0000 7fa7567fc640 1 -- 172.21.15.1:0/343725758 --> v1:172.21.15.115:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa758142590 con 0x7fa7580a98d0 2024-03-25T18:27:01.203 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:27:01.202+0000 7fa7567fc640 10 monclient: _finish_auth 0 2024-03-25T18:27:01.203 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:27:01.202+0000 7fa7567fc640 10 monclient: _check_auth_tickets 2024-03-25T18:27:01.204 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:27:01.202+0000 7fa7567fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:27:01.204 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:27:01.202+0000 7fa75df56640 5 monclient: authenticate success, global_id 4204 2024-03-25T18:27:01.204 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:27:01.202+0000 7fa75df56640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T18:27:01.204 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:27:01.202+0000 7fa7567fc640 1 -- 172.21.15.1:0/343725758 <== mon.1 v1:172.21.15.115:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1586965025 0 0) 0x7fa748001670 con 0x7fa7580a98d0 2024-03-25T18:27:01.204 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:27:01.202+0000 7fa7567fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:27:01.204 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:27:01.202+0000 7fa7567fc640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-03-25T18:27:01.204 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:27:01.202+0000 7fa7567fc640 10 monclient: dump: 2024-03-25T18:27:01.204 INFO:teuthology.misc.health.smithi001.stderr:epoch 1 2024-03-25T18:27:01.204 INFO:teuthology.misc.health.smithi001.stderr:fsid c8ea72cf-9e99-4383-9b52-6c23d365fdfc 2024-03-25T18:27:01.204 INFO:teuthology.misc.health.smithi001.stderr:last_changed 2024-03-25T18:25:50.510519+0000 2024-03-25T18:27:01.204 INFO:teuthology.misc.health.smithi001.stderr:created 2024-03-25T18:25:50.510519+0000 2024-03-25T18:27:01.204 INFO:teuthology.misc.health.smithi001.stderr:min_mon_release 19 (squid) 2024-03-25T18:27:01.204 INFO:teuthology.misc.health.smithi001.stderr:election_strategy: 3 2024-03-25T18:27:01.205 INFO:teuthology.misc.health.smithi001.stderr:0: v1:172.21.15.1:6789/0 mon.a 2024-03-25T18:27:01.205 INFO:teuthology.misc.health.smithi001.stderr:1: v1:172.21.15.115:6789/0 mon.b 2024-03-25T18:27:01.205 INFO:teuthology.misc.health.smithi001.stderr:2: v1:172.21.15.1:6790/0 mon.c 2024-03-25T18:27:01.205 INFO:teuthology.misc.health.smithi001.stderr: 2024-03-25T18:27:01.205 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:27:01.202+0000 7fa75df56640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T18:27:01.206 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:27:01.204+0000 7fa7567fc640 1 -- 172.21.15.1:0/343725758 <== mon.1 v1:172.21.15.115:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fa748001850 con 0x7fa7580a98d0 2024-03-25T18:27:01.206 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:27:01.204+0000 7fa7567fc640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:27:01.206 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:27:01.204+0000 7fa75df56640 10 monclient: get_monmap_and_config success 2024-03-25T18:27:01.206 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:27:01.204+0000 7fa75df56640 4 set_mon_vals no callback set 2024-03-25T18:27:01.206 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:27:01.205+0000 7fa7567fc640 1 -- 172.21.15.1:0/343725758 <== mon.1 v1:172.21.15.115:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1586965025 0 0) 0x7fa748001670 con 0x7fa7580a98d0 2024-03-25T18:27:01.207 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:27:01.205+0000 7fa7567fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:27:01.207 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:27:01.205+0000 7fa7567fc640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-25T18:27:01.207 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:27:01.205+0000 7fa7567fc640 10 monclient: dump: 2024-03-25T18:27:01.207 INFO:teuthology.misc.health.smithi001.stderr:epoch 1 2024-03-25T18:27:01.207 INFO:teuthology.misc.health.smithi001.stderr:fsid c8ea72cf-9e99-4383-9b52-6c23d365fdfc 2024-03-25T18:27:01.207 INFO:teuthology.misc.health.smithi001.stderr:last_changed 2024-03-25T18:25:50.510519+0000 2024-03-25T18:27:01.207 INFO:teuthology.misc.health.smithi001.stderr:created 2024-03-25T18:25:50.510519+0000 2024-03-25T18:27:01.207 INFO:teuthology.misc.health.smithi001.stderr:min_mon_release 19 (squid) 2024-03-25T18:27:01.207 INFO:teuthology.misc.health.smithi001.stderr:election_strategy: 3 2024-03-25T18:27:01.207 INFO:teuthology.misc.health.smithi001.stderr:0: v1:172.21.15.1:6789/0 mon.a 2024-03-25T18:27:01.207 INFO:teuthology.misc.health.smithi001.stderr:1: v1:172.21.15.115:6789/0 mon.b 2024-03-25T18:27:01.207 INFO:teuthology.misc.health.smithi001.stderr:2: v1:172.21.15.1:6790/0 mon.c 2024-03-25T18:27:01.207 INFO:teuthology.misc.health.smithi001.stderr: 2024-03-25T18:27:01.208 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:27:01.205+0000 7fa75df56640 10 monclient: shutdown 2024-03-25T18:27:01.208 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:27:01.205+0000 7fa75df56640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:27:01.208 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:27:01.205+0000 7fa75df56640 1 -- 172.21.15.1:0/343725758 >> v1:172.21.15.115:6789/0 conn(0x7fa7580a98d0 legacy=0x7fa758142ab0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:27:01.208 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:27:01.205+0000 7fa75df56640 1 -- 172.21.15.1:0/343725758 shutdown_connections 2024-03-25T18:27:01.208 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:27:01.205+0000 7fa75df56640 1 -- 172.21.15.1:0/343725758 >> 172.21.15.1:0/343725758 conn(0x7fa7580b4b60 msgr2=0x7fa7580b4f70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:27:01.208 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:27:01.205+0000 7fa75df56640 1 -- 172.21.15.1:0/343725758 shutdown_connections 2024-03-25T18:27:01.208 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:27:01.206+0000 7fa75df56640 1 -- 172.21.15.1:0/343725758 wait complete. 2024-03-25T18:27:01.211 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:27:01.206+0000 7fa75df56640 10 monclient: build_initial_monmap 2024-03-25T18:27:01.211 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:27:01.206+0000 7fa75df56640 10 monclient: monmap: 2024-03-25T18:27:01.211 INFO:teuthology.misc.health.smithi001.stderr:epoch 0 2024-03-25T18:27:01.211 INFO:teuthology.misc.health.smithi001.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T18:27:01.211 INFO:teuthology.misc.health.smithi001.stderr:last_changed 0.000000 2024-03-25T18:27:01.211 INFO:teuthology.misc.health.smithi001.stderr:created 0.000000 2024-03-25T18:27:01.211 INFO:teuthology.misc.health.smithi001.stderr:min_mon_release 0 (unknown) 2024-03-25T18:27:01.211 INFO:teuthology.misc.health.smithi001.stderr:election_strategy: 1 2024-03-25T18:27:01.211 INFO:teuthology.misc.health.smithi001.stderr:0: v1:172.21.15.1:6789/0 mon.noname-a 2024-03-25T18:27:01.211 INFO:teuthology.misc.health.smithi001.stderr:1: v1:172.21.15.115:6789/0 mon.noname-b 2024-03-25T18:27:01.211 INFO:teuthology.misc.health.smithi001.stderr:2: v1:172.21.15.1:6790/0 mon.noname-c 2024-03-25T18:27:01.211 INFO:teuthology.misc.health.smithi001.stderr: 2024-03-25T18:27:01.212 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:27:01.207+0000 7fa75df56640 1 librados: starting msgr at 2024-03-25T18:27:01.212 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:27:01.207+0000 7fa75df56640 1 librados: starting objecter 2024-03-25T18:27:01.212 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:27:01.207+0000 7fa75df56640 1 Processor -- start 2024-03-25T18:27:01.212 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:27:01.207+0000 7fa75df56640 1 -- start start 2024-03-25T18:27:01.212 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:27:01.207+0000 7fa75df56640 1 librados: setting wanted keys 2024-03-25T18:27:01.212 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:27:01.207+0000 7fa75df56640 1 librados: calling monclient init 2024-03-25T18:27:01.212 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:27:01.207+0000 7fa75df56640 10 monclient: init 2024-03-25T18:27:01.212 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:27:01.208+0000 7fa75df56640 10 monclient: _reopen_session rank -1 2024-03-25T18:27:01.212 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:27:01.208+0000 7fa75df56640 10 monclient: _add_conns ranks=[2,1,0] 2024-03-25T18:27:01.212 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:27:01.208+0000 7fa75df56640 10 monclient(hunting): picked mon.noname-c con 0x7fa7580a98d0 addr v1:172.21.15.1:6790/0 2024-03-25T18:27:01.212 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:27:01.208+0000 7fa75df56640 10 monclient(hunting): picked mon.noname-b con 0x7fa7580b2a60 addr v1:172.21.15.115:6789/0 2024-03-25T18:27:01.212 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:27:01.208+0000 7fa75df56640 10 monclient(hunting): picked mon.noname-a con 0x7fa75814bc00 addr v1:172.21.15.1:6789/0 2024-03-25T18:27:01.212 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:27:01.208+0000 7fa75df56640 1 -- --> v1:172.21.15.1:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa758147c30 con 0x7fa75814bc00 2024-03-25T18:27:01.213 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:27:01.208+0000 7fa75df56640 1 -- --> v1:172.21.15.115:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa75813e310 con 0x7fa7580b2a60 2024-03-25T18:27:01.213 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:27:01.209+0000 7fa75df56640 1 -- --> v1:172.21.15.1:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa75813ee00 con 0x7fa7580a98d0 2024-03-25T18:27:01.213 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:27:01.209+0000 7fa7577fe640 1 --1- >> v1:172.21.15.1:6790/0 conn(0x7fa7580a98d0 0x7fa75813e5a0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.1:6790/0 says I am v1:172.21.15.1:36448/0 (socket says 172.21.15.1:36448) 2024-03-25T18:27:01.213 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:27:01.209+0000 7fa7577fe640 1 -- 172.21.15.1:0/2930706795 learned_addr learned my addr 172.21.15.1:0/2930706795 (peer_addr_for_me v1:172.21.15.1:0/0) 2024-03-25T18:27:01.213 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:27:01.209+0000 7fa75df56640 10 monclient(hunting): _renew_subs 2024-03-25T18:27:01.213 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:27:01.209+0000 7fa75df56640 10 monclient(hunting): authenticate will time out at 7501.678998s 2024-03-25T18:27:01.213 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:27:01.209+0000 7fa73ffff640 10 client.?.objecter ms_handle_connect 0x7fa7580a98d0 2024-03-25T18:27:01.213 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:27:01.209+0000 7fa73ffff640 10 client.?.objecter resend_mon_ops 2024-03-25T18:27:01.213 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:27:01.209+0000 7fa73ffff640 10 client.?.objecter ms_handle_connect 0x7fa7580b2a60 2024-03-25T18:27:01.213 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:27:01.209+0000 7fa73ffff640 10 client.?.objecter resend_mon_ops 2024-03-25T18:27:01.213 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:27:01.209+0000 7fa73ffff640 10 client.?.objecter ms_handle_connect 0x7fa75814bc00 2024-03-25T18:27:01.213 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:27:01.209+0000 7fa73ffff640 10 client.?.objecter resend_mon_ops 2024-03-25T18:27:01.214 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:27:01.211+0000 7fa73ffff640 1 -- 172.21.15.1:0/2930706795 <== mon.1 v1:172.21.15.115:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 946850138 0 0) 0x7fa75813e310 con 0x7fa7580b2a60 2024-03-25T18:27:01.214 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:27:01.211+0000 7fa73ffff640 10 monclient(hunting): _negotiate 2024-03-25T18:27:01.214 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:27:01.211+0000 7fa73ffff640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:27:01.214 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:27:01.211+0000 7fa73ffff640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:27:01.214 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:27:01.211+0000 7fa73ffff640 10 monclient(hunting): my global_id is 4207 2024-03-25T18:27:01.214 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:27:01.211+0000 7fa73ffff640 1 -- 172.21.15.1:0/2930706795 --> v1:172.21.15.115:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fa744004eb0 con 0x7fa7580b2a60 2024-03-25T18:27:01.214 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:27:01.212+0000 7fa73ffff640 1 -- 172.21.15.1:0/2930706795 <== mon.2 v1:172.21.15.1:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1385207935 0 0) 0x7fa75813ee00 con 0x7fa7580a98d0 2024-03-25T18:27:01.215 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:27:01.212+0000 7fa73ffff640 10 monclient(hunting): _negotiate 2024-03-25T18:27:01.215 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:27:01.212+0000 7fa73ffff640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:27:01.215 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:27:01.212+0000 7fa73ffff640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:27:01.215 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:27:01.212+0000 7fa73ffff640 10 monclient(hunting): my global_id is 4214 2024-03-25T18:27:01.215 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:27:01.212+0000 7fa73ffff640 1 -- 172.21.15.1:0/2930706795 --> v1:172.21.15.1:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fa75813e310 con 0x7fa7580a98d0 2024-03-25T18:27:01.215 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:27:01.212+0000 7fa73ffff640 1 -- 172.21.15.1:0/2930706795 <== mon.0 v1:172.21.15.1:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2696699111 0 0) 0x7fa758147c30 con 0x7fa75814bc00 2024-03-25T18:27:01.215 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:27:01.212+0000 7fa73ffff640 10 monclient(hunting): _negotiate 2024-03-25T18:27:01.215 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:27:01.212+0000 7fa73ffff640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:27:01.215 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:27:01.212+0000 7fa73ffff640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:27:01.215 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:27:01.212+0000 7fa73ffff640 10 monclient(hunting): my global_id is 4209 2024-03-25T18:27:01.215 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:27:01.212+0000 7fa73ffff640 1 -- 172.21.15.1:0/2930706795 --> v1:172.21.15.1:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fa75813ee00 con 0x7fa75814bc00 2024-03-25T18:27:01.218 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:27:01.218+0000 7fa73ffff640 1 -- 172.21.15.1:0/2930706795 <== mon.1 v1:172.21.15.115:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1076262450 0 0) 0x7fa744004eb0 con 0x7fa7580b2a60 2024-03-25T18:27:01.218 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:27:01.218+0000 7fa73ffff640 1 -- 172.21.15.1:0/2930706795 --> v1:172.21.15.115:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fa758147c30 con 0x7fa7580b2a60 2024-03-25T18:27:01.219 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:27:01.218+0000 7fa73ffff640 1 -- 172.21.15.1:0/2930706795 <== mon.1 v1:172.21.15.115:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1586965025 0 0) 0x7fa74c001910 con 0x7fa7580b2a60 2024-03-25T18:27:01.219 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:27:01.218+0000 7fa73ffff640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:27:01.219 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:27:01.218+0000 7fa73ffff640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-03-25T18:27:01.219 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:27:01.218+0000 7fa73ffff640 10 monclient(hunting): dump: 2024-03-25T18:27:01.219 INFO:teuthology.misc.health.smithi001.stderr:epoch 1 2024-03-25T18:27:01.219 INFO:teuthology.misc.health.smithi001.stderr:fsid c8ea72cf-9e99-4383-9b52-6c23d365fdfc 2024-03-25T18:27:01.219 INFO:teuthology.misc.health.smithi001.stderr:last_changed 2024-03-25T18:25:50.510519+0000 2024-03-25T18:27:01.219 INFO:teuthology.misc.health.smithi001.stderr:created 2024-03-25T18:25:50.510519+0000 2024-03-25T18:27:01.219 INFO:teuthology.misc.health.smithi001.stderr:min_mon_release 19 (squid) 2024-03-25T18:27:01.219 INFO:teuthology.misc.health.smithi001.stderr:election_strategy: 3 2024-03-25T18:27:01.219 INFO:teuthology.misc.health.smithi001.stderr:0: v1:172.21.15.1:6789/0 mon.a 2024-03-25T18:27:01.219 INFO:teuthology.misc.health.smithi001.stderr:1: v1:172.21.15.115:6789/0 mon.b 2024-03-25T18:27:01.220 INFO:teuthology.misc.health.smithi001.stderr:2: v1:172.21.15.1:6790/0 mon.c 2024-03-25T18:27:01.220 INFO:teuthology.misc.health.smithi001.stderr: 2024-03-25T18:27:01.221 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:27:01.218+0000 7fa73ffff640 10 monclient(hunting): _finish_auth 0 2024-03-25T18:27:01.221 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:27:01.221+0000 7fa73ffff640 1 -- 172.21.15.1:0/2930706795 <== mon.0 v1:172.21.15.1:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1905376259 0 0) 0x7fa75813ee00 con 0x7fa75814bc00 2024-03-25T18:27:01.221 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:27:01.221+0000 7fa73ffff640 1 -- 172.21.15.1:0/2930706795 --> v1:172.21.15.1:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fa744004eb0 con 0x7fa75814bc00 2024-03-25T18:27:01.222 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:27:01.221+0000 7fa73ffff640 1 -- 172.21.15.1:0/2930706795 <== mon.0 v1:172.21.15.1:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1586965025 0 0) 0x7fa750001b90 con 0x7fa75814bc00 2024-03-25T18:27:01.223 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:27:01.221+0000 7fa73ffff640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:27:01.223 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:27:01.221+0000 7fa73ffff640 10 monclient(hunting): got monmap 1 from mon.a (according to old e1) 2024-03-25T18:27:01.223 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:27:01.221+0000 7fa73ffff640 10 monclient(hunting): dump: 2024-03-25T18:27:01.223 INFO:teuthology.misc.health.smithi001.stderr:epoch 1 2024-03-25T18:27:01.223 INFO:teuthology.misc.health.smithi001.stderr:fsid c8ea72cf-9e99-4383-9b52-6c23d365fdfc 2024-03-25T18:27:01.223 INFO:teuthology.misc.health.smithi001.stderr:last_changed 2024-03-25T18:25:50.510519+0000 2024-03-25T18:27:01.223 INFO:teuthology.misc.health.smithi001.stderr:created 2024-03-25T18:25:50.510519+0000 2024-03-25T18:27:01.223 INFO:teuthology.misc.health.smithi001.stderr:min_mon_release 19 (squid) 2024-03-25T18:27:01.223 INFO:teuthology.misc.health.smithi001.stderr:election_strategy: 3 2024-03-25T18:27:01.223 INFO:teuthology.misc.health.smithi001.stderr:0: v1:172.21.15.1:6789/0 mon.a 2024-03-25T18:27:01.223 INFO:teuthology.misc.health.smithi001.stderr:1: v1:172.21.15.115:6789/0 mon.b 2024-03-25T18:27:01.224 INFO:teuthology.misc.health.smithi001.stderr:2: v1:172.21.15.1:6790/0 mon.c 2024-03-25T18:27:01.224 INFO:teuthology.misc.health.smithi001.stderr: 2024-03-25T18:27:01.224 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:27:01.221+0000 7fa73ffff640 1 -- 172.21.15.1:0/2930706795 <== mon.2 v1:172.21.15.1:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 406989220 0 0) 0x7fa75813e310 con 0x7fa7580a98d0 2024-03-25T18:27:01.224 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:27:01.221+0000 7fa73ffff640 1 -- 172.21.15.1:0/2930706795 --> v1:172.21.15.1:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fa75813ee00 con 0x7fa7580a98d0 2024-03-25T18:27:01.227 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:27:01.223+0000 7fa73ffff640 1 -- 172.21.15.1:0/2930706795 <== mon.2 v1:172.21.15.1:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1586965025 0 0) 0x7fa748001670 con 0x7fa7580a98d0 2024-03-25T18:27:01.227 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:27:01.223+0000 7fa73ffff640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:27:01.227 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:27:01.223+0000 7fa73ffff640 10 monclient(hunting): got monmap 1 from mon.c (according to old e1) 2024-03-25T18:27:01.227 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:27:01.223+0000 7fa73ffff640 10 monclient(hunting): dump: 2024-03-25T18:27:01.227 INFO:teuthology.misc.health.smithi001.stderr:epoch 1 2024-03-25T18:27:01.227 INFO:teuthology.misc.health.smithi001.stderr:fsid c8ea72cf-9e99-4383-9b52-6c23d365fdfc 2024-03-25T18:27:01.228 INFO:teuthology.misc.health.smithi001.stderr:last_changed 2024-03-25T18:25:50.510519+0000 2024-03-25T18:27:01.228 INFO:teuthology.misc.health.smithi001.stderr:created 2024-03-25T18:25:50.510519+0000 2024-03-25T18:27:01.228 INFO:teuthology.misc.health.smithi001.stderr:min_mon_release 19 (squid) 2024-03-25T18:27:01.228 INFO:teuthology.misc.health.smithi001.stderr:election_strategy: 3 2024-03-25T18:27:01.228 INFO:teuthology.misc.health.smithi001.stderr:0: v1:172.21.15.1:6789/0 mon.a 2024-03-25T18:27:01.228 INFO:teuthology.misc.health.smithi001.stderr:1: v1:172.21.15.115:6789/0 mon.b 2024-03-25T18:27:01.228 INFO:teuthology.misc.health.smithi001.stderr:2: v1:172.21.15.1:6790/0 mon.c 2024-03-25T18:27:01.228 INFO:teuthology.misc.health.smithi001.stderr: 2024-03-25T18:27:01.228 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:27:01.223+0000 7fa73ffff640 1 -- 172.21.15.1:0/2930706795 <== mon.1 v1:172.21.15.115:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 3640146220 0 0) 0x7fa758147c30 con 0x7fa7580b2a60 2024-03-25T18:27:01.228 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:27:01.223+0000 7fa73ffff640 1 -- 172.21.15.1:0/2930706795 >> v1:172.21.15.1:6790/0 conn(0x7fa7580a98d0 legacy=0x7fa75813e5a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:27:01.228 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:27:01.223+0000 7fa73ffff640 1 -- 172.21.15.1:0/2930706795 >> v1:172.21.15.1:6789/0 conn(0x7fa75814bc00 legacy=0x7fa7581da6b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:27:01.228 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:27:01.223+0000 7fa73ffff640 10 monclient: _finish_hunting 0 2024-03-25T18:27:01.228 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:27:01.223+0000 7fa73ffff640 1 monclient: found mon.b 2024-03-25T18:27:01.228 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:27:01.223+0000 7fa73ffff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:27:01.229 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:27:01.223+0000 7fa73ffff640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.115:6789/0 2024-03-25T18:27:01.229 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:27:01.223+0000 7fa73ffff640 1 -- 172.21.15.1:0/2930706795 --> v1:172.21.15.115:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa7581461c0 con 0x7fa7580b2a60 2024-03-25T18:27:01.229 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:27:01.223+0000 7fa73ffff640 10 monclient: _finish_auth 0 2024-03-25T18:27:01.229 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:27:01.223+0000 7fa73ffff640 10 monclient: _check_auth_tickets 2024-03-25T18:27:01.229 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:27:01.223+0000 7fa73ffff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:27:01.229 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:27:01.224+0000 7fa75df56640 5 monclient: authenticate success, global_id 4207 2024-03-25T18:27:01.229 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:27:01.224+0000 7fa75df56640 10 monclient: _renew_subs 2024-03-25T18:27:01.229 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:27:01.224+0000 7fa75df56640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.115:6789/0 2024-03-25T18:27:01.229 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:27:01.224+0000 7fa75df56640 1 -- 172.21.15.1:0/2930706795 --> v1:172.21.15.115:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa758146470 con 0x7fa7580b2a60 2024-03-25T18:27:01.229 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:27:01.224+0000 7fa75df56640 10 client.4207.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T18:27:01.229 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:27:01.224+0000 7fa75df56640 10 monclient: _renew_subs 2024-03-25T18:27:01.229 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:27:01.224+0000 7fa75df56640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.115:6789/0 2024-03-25T18:27:01.229 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:27:01.224+0000 7fa75df56640 1 -- 172.21.15.1:0/2930706795 --> v1:172.21.15.115:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7fa758138b00 con 0x7fa7580b2a60 2024-03-25T18:27:01.229 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:27:01.224+0000 7fa75df56640 1 librados: init done 2024-03-25T18:27:01.230 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:27:01.225+0000 7fa75df56640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T18:27:01.230 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:27:01.225+0000 7fa75df56640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:27:01.230 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:27:01.225+0000 7fa75df56640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.115:6789/0 2024-03-25T18:27:01.230 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:27:01.225+0000 7fa75df56640 1 -- 172.21.15.1:0/2930706795 --> v1:172.21.15.115:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa7580ab020 con 0x7fa7580b2a60 2024-03-25T18:27:01.412 INFO:tasks.ceph.osd.3.smithi001.stderr:==00:00:00:17.014 35191== Warning: unimplemented fcntl command: 1038 2024-03-25T18:27:01.485 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:27:01.484+0000 7fa73ffff640 1 -- 172.21.15.1:0/2930706795 <== mon.1 v1:172.21.15.115:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fa74c0047e0 con 0x7fa7580b2a60 2024-03-25T18:27:01.485 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:27:01.484+0000 7fa73ffff640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:27:01.485 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:27:01.484+0000 7fa73ffff640 1 -- 172.21.15.1:0/2930706795 <== mon.1 v1:172.21.15.115:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1586965025 0 0) 0x7fa74c004af0 con 0x7fa7580b2a60 2024-03-25T18:27:01.485 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:27:01.484+0000 7fa73ffff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:27:01.485 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:27:01.484+0000 7fa73ffff640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-25T18:27:01.485 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:27:01.484+0000 7fa73ffff640 10 monclient: dump: 2024-03-25T18:27:01.485 INFO:teuthology.misc.health.smithi001.stderr:epoch 1 2024-03-25T18:27:01.485 INFO:teuthology.misc.health.smithi001.stderr:fsid c8ea72cf-9e99-4383-9b52-6c23d365fdfc 2024-03-25T18:27:01.486 INFO:teuthology.misc.health.smithi001.stderr:last_changed 2024-03-25T18:25:50.510519+0000 2024-03-25T18:27:01.486 INFO:teuthology.misc.health.smithi001.stderr:created 2024-03-25T18:25:50.510519+0000 2024-03-25T18:27:01.486 INFO:teuthology.misc.health.smithi001.stderr:min_mon_release 19 (squid) 2024-03-25T18:27:01.486 INFO:teuthology.misc.health.smithi001.stderr:election_strategy: 3 2024-03-25T18:27:01.486 INFO:teuthology.misc.health.smithi001.stderr:0: v1:172.21.15.1:6789/0 mon.a 2024-03-25T18:27:01.486 INFO:teuthology.misc.health.smithi001.stderr:1: v1:172.21.15.115:6789/0 mon.b 2024-03-25T18:27:01.486 INFO:teuthology.misc.health.smithi001.stderr:2: v1:172.21.15.1:6790/0 mon.c 2024-03-25T18:27:01.486 INFO:teuthology.misc.health.smithi001.stderr: 2024-03-25T18:27:01.486 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:27:01.484+0000 7fa755ffb640 4 set_mon_vals no callback set 2024-03-25T18:27:01.487 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:27:01.485+0000 7fa73ffff640 1 -- 172.21.15.1:0/2930706795 <== mon.1 v1:172.21.15.115:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 21787975 0 0) 0x7fa74c01aac0 con 0x7fa7580b2a60 2024-03-25T18:27:01.487 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:27:01.486+0000 7fa7577fe640 10 monclient: get_auth_request con 0x7fa74406e110 auth_method 0 2024-03-25T18:27:01.487 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:27:01.486+0000 7fa73ffff640 10 client.4207.objecter ms_handle_connect 0x7fa74406e110 2024-03-25T18:27:01.490 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:27:01.490+0000 7fa73ffff640 1 -- 172.21.15.1:0/2930706795 <== mon.1 v1:172.21.15.115:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 996254230 0 0) 0x7fa74c054740 con 0x7fa7580b2a60 2024-03-25T18:27:01.490 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:27:01.490+0000 7fa73ffff640 10 client.4207.objecter ms_dispatch 0x7fa758000b90 osd_map(10..10 src has 1..10) v4 2024-03-25T18:27:01.491 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:27:01.490+0000 7fa73ffff640 3 client.4207.objecter handle_osd_map got epochs [10,10] > 0 2024-03-25T18:27:01.491 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:27:01.490+0000 7fa73ffff640 3 client.4207.objecter handle_osd_map decoding full epoch 10 2024-03-25T18:27:01.491 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:27:01.490+0000 7fa73ffff640 20 client.4207.objecter dump_active .. 0 homeless 2024-03-25T18:27:01.491 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:27:01.490+0000 7fa73ffff640 1 -- 172.21.15.1:0/2930706795 <== mon.1 v1:172.21.15.115:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7fa74c050080 con 0x7fa7580b2a60 2024-03-25T18:27:01.491 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:27:01.490+0000 7fa73ffff640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:27:01.491 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:27:01.490+0000 7fa73ffff640 10 monclient: _finish_command 1 = system:0 2024-03-25T18:27:01.834 INFO:tasks.ceph.osd.1.smithi001.stderr:==00:00:00:17.449 35189== Warning: unimplemented fcntl command: 1038 2024-03-25T18:27:01.835 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:27:01.834+0000 7fa75df56640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:27:01.835 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:27:01.834+0000 7fa75df56640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:27:01.835 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:27:01.834+0000 7fa75df56640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.115:6789/0 2024-03-25T18:27:01.835 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:27:01.834+0000 7fa75df56640 1 -- 172.21.15.1:0/2930706795 --> v1:172.21.15.115:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fa7580b4780 con 0x7fa7580b2a60 2024-03-25T18:27:01.860 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:27:01.859+0000 7fa73ffff640 1 -- 172.21.15.1:0/2930706795 <== mon.1 v1:172.21.15.115:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 4092213348) 0x7fa7580b4780 con 0x7fa7580b2a60 2024-03-25T18:27:01.860 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:27:01.859+0000 7fa73ffff640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:27:01.860 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:27:01.859+0000 7fa73ffff640 10 monclient: _finish_command 2 = system:0 2024-03-25T18:27:01.860 INFO:teuthology.misc.health.smithi001.stdout: 2024-03-25T18:27:01.861 INFO:teuthology.misc.health.smithi001.stdout:{"epoch":10,"fsid":"c8ea72cf-9e99-4383-9b52-6c23d365fdfc","created":"2024-03-25T18:26:38.856638+0000","modified":"2024-03-25T18:26:44.224208+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T18:26:44.224208+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"b9f34922-3a84-49be-8426-807c1eceaefc","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"42bf306f-55e3-4634-9874-8b4b73ac6644","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"292cb23a-18a9-4207-a7cc-4d5727feaec0","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"45e0e9ab-898d-4673-9590-b4117bc5e84a","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"831a5a09-73ab-4a61-9872-088e76bdbc66","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"88cd903d-3b55-4840-9396-6e3cfe005282","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"89db7355-bc9d-46f4-8ab0-e8d9e744e026","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"cac47bee-0914-4f08-896f-348838882866","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T18:27:01.863 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:27:01.862+0000 7fa75df56640 10 librados: watch_flush enter 2024-03-25T18:27:01.863 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:27:01.862+0000 7fa75df56640 10 librados: watch_flush exit 2024-03-25T18:27:01.863 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:27:01.862+0000 7fa75df56640 20 client.4207.objecter shutdown clearing up homeless session... 2024-03-25T18:27:01.863 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:27:01.862+0000 7fa75df56640 10 client.4207.objecter successfully canceled tick 2024-03-25T18:27:01.863 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:27:01.862+0000 7fa75df56640 1 -- 172.21.15.1:0/2930706795 >> v1:172.21.15.115:6800/4237379176 conn(0x7fa74406e110 legacy=0x7fa7440705d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:27:01.863 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:27:01.862+0000 7fa75df56640 10 monclient: shutdown 2024-03-25T18:27:01.864 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:27:01.862+0000 7fa75df56640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:27:01.864 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:27:01.862+0000 7fa75df56640 1 -- 172.21.15.1:0/2930706795 >> v1:172.21.15.115:6789/0 conn(0x7fa7580b2a60 legacy=0x7fa758138210 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:27:01.864 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:27:01.862+0000 7fa75df56640 1 -- 172.21.15.1:0/2930706795 shutdown_connections 2024-03-25T18:27:01.864 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:27:01.862+0000 7fa75df56640 1 -- 172.21.15.1:0/2930706795 >> 172.21.15.1:0/2930706795 conn(0x7fa7580b4b60 msgr2=0x7fa7580b4f40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:27:01.864 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:27:01.862+0000 7fa75df56640 1 -- 172.21.15.1:0/2930706795 shutdown_connections 2024-03-25T18:27:01.864 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:27:01.862+0000 7fa75df56640 1 -- 172.21.15.1:0/2930706795 wait complete. 2024-03-25T18:27:01.864 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:27:01.863+0000 7fa75df56640 1 librados: shutdown 2024-03-25T18:27:01.877 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T18:27:02.168 INFO:tasks.ceph.osd.0.smithi001.stderr:==00:00:00:17.775 35190== Warning: unimplemented fcntl command: 1038 2024-03-25T18:27:02.242 INFO:tasks.ceph.osd.2.smithi001.stderr:==00:00:00:17.833 35192== Warning: unimplemented fcntl command: 1038 2024-03-25T18:27:06.627 INFO:tasks.ceph.osd.4.smithi115.stderr:==00:00:00:22.244 35380== Warning: unimplemented fcntl command: 1038 2024-03-25T18:27:06.631 INFO:tasks.ceph.osd.4.smithi115.stderr:==00:00:00:22.248 35380== Warning: unimplemented fcntl command: 1038 2024-03-25T18:27:06.739 INFO:tasks.ceph.osd.7.smithi115.stderr:==00:00:00:22.351 35383== Warning: unimplemented fcntl command: 1038 2024-03-25T18:27:06.743 INFO:tasks.ceph.osd.7.smithi115.stderr:==00:00:00:22.355 35383== Warning: unimplemented fcntl command: 1038 2024-03-25T18:27:06.848 INFO:tasks.ceph.osd.5.smithi115.stderr:==00:00:00:22.465 35381== Warning: unimplemented fcntl command: 1038 2024-03-25T18:27:06.852 INFO:tasks.ceph.osd.5.smithi115.stderr:==00:00:00:22.469 35381== Warning: unimplemented fcntl command: 1038 2024-03-25T18:27:07.098 INFO:tasks.ceph.osd.3.smithi001.stderr:==00:00:00:22.700 35191== Warning: unimplemented fcntl command: 1038 2024-03-25T18:27:07.104 INFO:tasks.ceph.osd.3.smithi001.stderr:==00:00:00:22.706 35191== Warning: unimplemented fcntl command: 1038 2024-03-25T18:27:07.225 INFO:tasks.ceph.osd.6.smithi115.stderr:==00:00:00:22.841 35382== Warning: unimplemented fcntl command: 1038 2024-03-25T18:27:07.229 INFO:tasks.ceph.osd.6.smithi115.stderr:==00:00:00:22.846 35382== Warning: unimplemented fcntl command: 1038 2024-03-25T18:27:07.560 INFO:tasks.ceph.osd.0.smithi001.stderr:==00:00:00:23.167 35190== Warning: unimplemented fcntl command: 1038 2024-03-25T18:27:07.564 INFO:tasks.ceph.osd.0.smithi001.stderr:==00:00:00:23.171 35190== Warning: unimplemented fcntl command: 1038 2024-03-25T18:27:07.807 INFO:tasks.ceph.osd.1.smithi001.stderr:==00:00:00:23.424 35189== Warning: unimplemented fcntl command: 1038 2024-03-25T18:27:07.813 INFO:tasks.ceph.osd.1.smithi001.stderr:==00:00:00:23.430 35189== Warning: unimplemented fcntl command: 1038 2024-03-25T18:27:08.164 INFO:tasks.ceph.osd.2.smithi001.stderr:==00:00:00:23.756 35192== Warning: unimplemented fcntl command: 1038 2024-03-25T18:27:08.168 INFO:tasks.ceph.osd.2.smithi001.stderr:==00:00:00:23.760 35192== Warning: unimplemented fcntl command: 1038 2024-03-25T18:27:08.685 DEBUG:teuthology.orchestra.run.smithi001:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T18:27:08.806 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:27:08.806+0000 7f0c5d674640 10 monclient: get_monmap_and_config 2024-03-25T18:27:08.807 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:27:08.806+0000 7f0c5d674640 10 monclient: build_initial_monmap 2024-03-25T18:27:08.807 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:27:08.806+0000 7f0c5d674640 10 monclient: monmap: 2024-03-25T18:27:08.807 INFO:teuthology.misc.health.smithi001.stderr:epoch 0 2024-03-25T18:27:08.807 INFO:teuthology.misc.health.smithi001.stderr:fsid c8ea72cf-9e99-4383-9b52-6c23d365fdfc 2024-03-25T18:27:08.807 INFO:teuthology.misc.health.smithi001.stderr:last_changed 2024-03-25T18:27:08.806923+0000 2024-03-25T18:27:08.807 INFO:teuthology.misc.health.smithi001.stderr:created 2024-03-25T18:27:08.806923+0000 2024-03-25T18:27:08.809 INFO:teuthology.misc.health.smithi001.stderr:min_mon_release 0 (unknown) 2024-03-25T18:27:08.809 INFO:teuthology.misc.health.smithi001.stderr:election_strategy: 3 2024-03-25T18:27:08.809 INFO:teuthology.misc.health.smithi001.stderr:0: v1:172.21.15.1:6789/0 mon.noname-a 2024-03-25T18:27:08.809 INFO:teuthology.misc.health.smithi001.stderr:1: v1:172.21.15.115:6789/0 mon.noname-c 2024-03-25T18:27:08.810 INFO:teuthology.misc.health.smithi001.stderr:2: v1:172.21.15.1:6790/0 mon.noname-b 2024-03-25T18:27:08.810 INFO:teuthology.misc.health.smithi001.stderr: 2024-03-25T18:27:08.811 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:27:08.806+0000 7f0c5d674640 1 Processor -- start 2024-03-25T18:27:08.811 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:27:08.807+0000 7f0c5d674640 1 -- start start 2024-03-25T18:27:08.811 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:27:08.807+0000 7f0c5d674640 10 monclient: init 2024-03-25T18:27:08.811 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:27:08.807+0000 7f0c5d674640 10 monclient: _reopen_session rank -1 2024-03-25T18:27:08.811 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:27:08.807+0000 7f0c5d674640 10 monclient: _add_conns ranks=[2,1,0] 2024-03-25T18:27:08.811 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:27:08.807+0000 7f0c5d674640 10 monclient(hunting): picked mon.noname-b con 0x7f0c580b4e70 addr v1:172.21.15.1:6790/0 2024-03-25T18:27:08.812 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:27:08.807+0000 7f0c5d674640 10 monclient(hunting): picked mon.noname-c con 0x7f0c580b44b0 addr v1:172.21.15.115:6789/0 2024-03-25T18:27:08.812 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:27:08.807+0000 7f0c5d674640 10 monclient(hunting): picked mon.noname-a con 0x7f0c580b1b00 addr v1:172.21.15.1:6789/0 2024-03-25T18:27:08.812 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:27:08.807+0000 7f0c5d674640 1 -- --> v1:172.21.15.1:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0c580a96f0 con 0x7f0c580b1b00 2024-03-25T18:27:08.812 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:27:08.807+0000 7f0c5d674640 1 -- --> v1:172.21.15.115:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0c580ab0e0 con 0x7f0c580b44b0 2024-03-25T18:27:08.812 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:27:08.807+0000 7f0c5d674640 1 -- --> v1:172.21.15.1:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0c58145f70 con 0x7f0c580b4e70 2024-03-25T18:27:08.812 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:27:08.807+0000 7f0c5d674640 10 monclient(hunting): _renew_subs 2024-03-25T18:27:08.812 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:27:08.807+0000 7f0c5d674640 10 monclient(hunting): authenticate will time out at 7509.277629s 2024-03-25T18:27:08.812 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:27:08.808+0000 7f0c577fe640 1 --1- >> v1:172.21.15.1:6789/0 conn(0x7f0c580b1b00 0x7f0c5814dc00 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.1:6789/0 says I am v1:172.21.15.1:38870/0 (socket says 172.21.15.1:38870) 2024-03-25T18:27:08.812 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:27:08.808+0000 7f0c577fe640 1 -- 172.21.15.1:0/3592286154 learned_addr learned my addr 172.21.15.1:0/3592286154 (peer_addr_for_me v1:172.21.15.1:0/0) 2024-03-25T18:27:08.812 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:27:08.808+0000 7f0c567fc640 1 --1- 172.21.15.1:0/3592286154 >> v1:172.21.15.115:6789/0 conn(0x7f0c580b44b0 0x7f0c58145860 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.115:6789/0 says I am v1:172.21.15.1:54414/0 (socket says 172.21.15.1:54414) 2024-03-25T18:27:08.813 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:27:08.811+0000 7f0c55ffb640 1 -- 172.21.15.1:0/3592286154 <== mon.1 v1:172.21.15.115:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3887617165 0 0) 0x7f0c580ab0e0 con 0x7f0c580b44b0 2024-03-25T18:27:08.813 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:27:08.811+0000 7f0c55ffb640 10 monclient(hunting): _negotiate 2024-03-25T18:27:08.813 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:27:08.811+0000 7f0c55ffb640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:27:08.814 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:27:08.811+0000 7f0c55ffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:27:08.814 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:27:08.811+0000 7f0c55ffb640 10 monclient(hunting): my global_id is 4210 2024-03-25T18:27:08.814 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:27:08.811+0000 7f0c55ffb640 1 -- 172.21.15.1:0/3592286154 --> v1:172.21.15.115:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f0c40003520 con 0x7f0c580b44b0 2024-03-25T18:27:08.814 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:27:08.811+0000 7f0c55ffb640 1 -- 172.21.15.1:0/3592286154 <== mon.2 v1:172.21.15.1:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 654076653 0 0) 0x7f0c58145f70 con 0x7f0c580b4e70 2024-03-25T18:27:08.814 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:27:08.811+0000 7f0c55ffb640 10 monclient(hunting): _negotiate 2024-03-25T18:27:08.814 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:27:08.811+0000 7f0c55ffb640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:27:08.814 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:27:08.811+0000 7f0c55ffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:27:08.814 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:27:08.811+0000 7f0c55ffb640 10 monclient(hunting): my global_id is 4217 2024-03-25T18:27:08.814 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:27:08.811+0000 7f0c55ffb640 1 -- 172.21.15.1:0/3592286154 --> v1:172.21.15.1:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f0c40003840 con 0x7f0c580b4e70 2024-03-25T18:27:08.814 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:27:08.812+0000 7f0c55ffb640 1 -- 172.21.15.1:0/3592286154 <== mon.0 v1:172.21.15.1:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3232588072 0 0) 0x7f0c580a96f0 con 0x7f0c580b1b00 2024-03-25T18:27:08.814 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:27:08.812+0000 7f0c55ffb640 10 monclient(hunting): _negotiate 2024-03-25T18:27:08.814 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:27:08.812+0000 7f0c55ffb640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:27:08.815 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:27:08.812+0000 7f0c55ffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:27:08.815 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:27:08.812+0000 7f0c55ffb640 10 monclient(hunting): my global_id is 4212 2024-03-25T18:27:08.815 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:27:08.812+0000 7f0c55ffb640 1 -- 172.21.15.1:0/3592286154 --> v1:172.21.15.1:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f0c58145f70 con 0x7f0c580b1b00 2024-03-25T18:27:08.816 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:27:08.815+0000 7f0c55ffb640 1 -- 172.21.15.1:0/3592286154 <== mon.2 v1:172.21.15.1:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 4096716908 0 0) 0x7f0c40003840 con 0x7f0c580b4e70 2024-03-25T18:27:08.816 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:27:08.815+0000 7f0c55ffb640 1 -- 172.21.15.1:0/3592286154 >> v1:172.21.15.115:6789/0 conn(0x7f0c580b44b0 legacy=0x7f0c58145860 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:27:08.816 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:27:08.815+0000 7f0c55ffb640 1 -- 172.21.15.1:0/3592286154 >> v1:172.21.15.1:6789/0 conn(0x7f0c580b1b00 legacy=0x7f0c5814dc00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:27:08.817 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:27:08.815+0000 7f0c55ffb640 10 monclient: _finish_hunting 0 2024-03-25T18:27:08.817 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:27:08.815+0000 7f0c55ffb640 1 monclient: found mon.noname-b 2024-03-25T18:27:08.817 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:27:08.815+0000 7f0c55ffb640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:27:08.817 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:27:08.815+0000 7f0c55ffb640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.1:6790/0 2024-03-25T18:27:08.817 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:27:08.815+0000 7f0c55ffb640 1 -- 172.21.15.1:0/3592286154 --> v1:172.21.15.1:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0c580a9b80 con 0x7f0c580b4e70 2024-03-25T18:27:08.817 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:27:08.815+0000 7f0c55ffb640 10 monclient: _finish_auth 0 2024-03-25T18:27:08.817 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:27:08.815+0000 7f0c55ffb640 10 monclient: _check_auth_tickets 2024-03-25T18:27:08.817 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:27:08.815+0000 7f0c55ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:27:08.817 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:27:08.815+0000 7f0c55ffb640 1 -- 172.21.15.1:0/3592286154 <== mon.2 v1:172.21.15.1:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1586965025 0 0) 0x7f0c48001670 con 0x7f0c580b4e70 2024-03-25T18:27:08.817 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:27:08.815+0000 7f0c55ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:27:08.817 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:27:08.815+0000 7f0c55ffb640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-03-25T18:27:08.817 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:27:08.815+0000 7f0c55ffb640 10 monclient: dump: 2024-03-25T18:27:08.817 INFO:teuthology.misc.health.smithi001.stderr:epoch 1 2024-03-25T18:27:08.817 INFO:teuthology.misc.health.smithi001.stderr:fsid c8ea72cf-9e99-4383-9b52-6c23d365fdfc 2024-03-25T18:27:08.818 INFO:teuthology.misc.health.smithi001.stderr:last_changed 2024-03-25T18:25:50.510519+0000 2024-03-25T18:27:08.818 INFO:teuthology.misc.health.smithi001.stderr:created 2024-03-25T18:25:50.510519+0000 2024-03-25T18:27:08.818 INFO:teuthology.misc.health.smithi001.stderr:min_mon_release 19 (squid) 2024-03-25T18:27:08.818 INFO:teuthology.misc.health.smithi001.stderr:election_strategy: 3 2024-03-25T18:27:08.818 INFO:teuthology.misc.health.smithi001.stderr:0: v1:172.21.15.1:6789/0 mon.a 2024-03-25T18:27:08.818 INFO:teuthology.misc.health.smithi001.stderr:1: v1:172.21.15.115:6789/0 mon.b 2024-03-25T18:27:08.818 INFO:teuthology.misc.health.smithi001.stderr:2: v1:172.21.15.1:6790/0 mon.c 2024-03-25T18:27:08.818 INFO:teuthology.misc.health.smithi001.stderr: 2024-03-25T18:27:08.818 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:27:08.815+0000 7f0c5d674640 5 monclient: authenticate success, global_id 4217 2024-03-25T18:27:08.818 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:27:08.815+0000 7f0c5d674640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T18:27:08.821 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:27:08.818+0000 7f0c55ffb640 1 -- 172.21.15.1:0/3592286154 <== mon.2 v1:172.21.15.1:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f0c480017d0 con 0x7f0c580b4e70 2024-03-25T18:27:08.821 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:27:08.818+0000 7f0c55ffb640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:27:08.821 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:27:08.818+0000 7f0c5d674640 10 monclient: get_monmap_and_config success 2024-03-25T18:27:08.821 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:27:08.818+0000 7f0c5d674640 4 set_mon_vals no callback set 2024-03-25T18:27:08.821 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:27:08.818+0000 7f0c55ffb640 1 -- 172.21.15.1:0/3592286154 <== mon.2 v1:172.21.15.1:6790/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1586965025 0 0) 0x7f0c48001be0 con 0x7f0c580b4e70 2024-03-25T18:27:08.822 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:27:08.818+0000 7f0c55ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:27:08.822 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:27:08.818+0000 7f0c55ffb640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-25T18:27:08.822 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:27:08.818+0000 7f0c55ffb640 10 monclient: dump: 2024-03-25T18:27:08.822 INFO:teuthology.misc.health.smithi001.stderr:epoch 1 2024-03-25T18:27:08.822 INFO:teuthology.misc.health.smithi001.stderr:fsid c8ea72cf-9e99-4383-9b52-6c23d365fdfc 2024-03-25T18:27:08.822 INFO:teuthology.misc.health.smithi001.stderr:last_changed 2024-03-25T18:25:50.510519+0000 2024-03-25T18:27:08.822 INFO:teuthology.misc.health.smithi001.stderr:created 2024-03-25T18:25:50.510519+0000 2024-03-25T18:27:08.822 INFO:teuthology.misc.health.smithi001.stderr:min_mon_release 19 (squid) 2024-03-25T18:27:08.822 INFO:teuthology.misc.health.smithi001.stderr:election_strategy: 3 2024-03-25T18:27:08.822 INFO:teuthology.misc.health.smithi001.stderr:0: v1:172.21.15.1:6789/0 mon.a 2024-03-25T18:27:08.822 INFO:teuthology.misc.health.smithi001.stderr:1: v1:172.21.15.115:6789/0 mon.b 2024-03-25T18:27:08.822 INFO:teuthology.misc.health.smithi001.stderr:2: v1:172.21.15.1:6790/0 mon.c 2024-03-25T18:27:08.822 INFO:teuthology.misc.health.smithi001.stderr: 2024-03-25T18:27:08.823 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:27:08.819+0000 7f0c5d674640 10 monclient: shutdown 2024-03-25T18:27:08.823 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:27:08.819+0000 7f0c5d674640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:27:08.823 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:27:08.819+0000 7f0c5d674640 1 -- 172.21.15.1:0/3592286154 >> v1:172.21.15.1:6790/0 conn(0x7f0c580b4e70 legacy=0x7f0c5813f940 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:27:08.823 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:27:08.819+0000 7f0c5d674640 1 -- 172.21.15.1:0/3592286154 shutdown_connections 2024-03-25T18:27:08.823 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:27:08.819+0000 7f0c5d674640 1 -- 172.21.15.1:0/3592286154 >> 172.21.15.1:0/3592286154 conn(0x7f0c580617a0 msgr2=0x7f0c58061bb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:27:08.823 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:27:08.820+0000 7f0c5d674640 1 -- 172.21.15.1:0/3592286154 shutdown_connections 2024-03-25T18:27:08.823 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:27:08.820+0000 7f0c5d674640 1 -- 172.21.15.1:0/3592286154 wait complete. 2024-03-25T18:27:08.823 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:27:08.820+0000 7f0c5d674640 10 monclient: build_initial_monmap 2024-03-25T18:27:08.823 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:27:08.821+0000 7f0c5d674640 10 monclient: monmap: 2024-03-25T18:27:08.823 INFO:teuthology.misc.health.smithi001.stderr:epoch 0 2024-03-25T18:27:08.823 INFO:teuthology.misc.health.smithi001.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T18:27:08.823 INFO:teuthology.misc.health.smithi001.stderr:last_changed 0.000000 2024-03-25T18:27:08.823 INFO:teuthology.misc.health.smithi001.stderr:created 0.000000 2024-03-25T18:27:08.824 INFO:teuthology.misc.health.smithi001.stderr:min_mon_release 0 (unknown) 2024-03-25T18:27:08.824 INFO:teuthology.misc.health.smithi001.stderr:election_strategy: 1 2024-03-25T18:27:08.824 INFO:teuthology.misc.health.smithi001.stderr:0: v1:172.21.15.1:6789/0 mon.noname-a 2024-03-25T18:27:08.824 INFO:teuthology.misc.health.smithi001.stderr:1: v1:172.21.15.115:6789/0 mon.noname-b 2024-03-25T18:27:08.824 INFO:teuthology.misc.health.smithi001.stderr:2: v1:172.21.15.1:6790/0 mon.noname-c 2024-03-25T18:27:08.824 INFO:teuthology.misc.health.smithi001.stderr: 2024-03-25T18:27:08.826 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:27:08.821+0000 7f0c5d674640 1 librados: starting msgr at 2024-03-25T18:27:08.826 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:27:08.821+0000 7f0c5d674640 1 librados: starting objecter 2024-03-25T18:27:08.826 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:27:08.821+0000 7f0c5d674640 1 Processor -- start 2024-03-25T18:27:08.826 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:27:08.822+0000 7f0c5d674640 1 -- start start 2024-03-25T18:27:08.826 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:27:08.822+0000 7f0c5d674640 1 librados: setting wanted keys 2024-03-25T18:27:08.826 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:27:08.822+0000 7f0c5d674640 1 librados: calling monclient init 2024-03-25T18:27:08.826 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:27:08.822+0000 7f0c5d674640 10 monclient: init 2024-03-25T18:27:08.826 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:27:08.822+0000 7f0c5d674640 10 monclient: _reopen_session rank -1 2024-03-25T18:27:08.827 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:27:08.822+0000 7f0c5d674640 10 monclient: _add_conns ranks=[0,1,2] 2024-03-25T18:27:08.827 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:27:08.822+0000 7f0c5d674640 10 monclient(hunting): picked mon.noname-a con 0x7f0c580b1b00 addr v1:172.21.15.1:6789/0 2024-03-25T18:27:08.827 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:27:08.822+0000 7f0c5d674640 10 monclient(hunting): picked mon.noname-b con 0x7f0c580b44b0 addr v1:172.21.15.115:6789/0 2024-03-25T18:27:08.827 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:27:08.823+0000 7f0c5d674640 10 monclient(hunting): picked mon.noname-c con 0x7f0c580b4e70 addr v1:172.21.15.1:6790/0 2024-03-25T18:27:08.827 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:27:08.823+0000 7f0c5d674640 1 -- --> v1:172.21.15.1:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0c581d85f0 con 0x7f0c580b1b00 2024-03-25T18:27:08.827 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:27:08.823+0000 7f0c5d674640 1 -- --> v1:172.21.15.115:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0c581d8780 con 0x7f0c580b44b0 2024-03-25T18:27:08.827 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:27:08.823+0000 7f0c5d674640 1 -- --> v1:172.21.15.1:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0c58140d90 con 0x7f0c580b4e70 2024-03-25T18:27:08.827 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:27:08.823+0000 7f0c5d674640 10 monclient(hunting): _renew_subs 2024-03-25T18:27:08.827 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:27:08.823+0000 7f0c5d674640 10 monclient(hunting): authenticate will time out at 7509.292857s 2024-03-25T18:27:08.827 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:27:08.823+0000 7f0c567fc640 1 --1- >> v1:172.21.15.115:6789/0 conn(0x7f0c580b44b0 0x7f0c58130ee0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.115:6789/0 says I am v1:172.21.15.1:54428/0 (socket says 172.21.15.1:54428) 2024-03-25T18:27:08.827 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:27:08.823+0000 7f0c567fc640 1 -- 172.21.15.1:0/1021873481 learned_addr learned my addr 172.21.15.1:0/1021873481 (peer_addr_for_me v1:172.21.15.1:0/0) 2024-03-25T18:27:08.827 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:27:08.823+0000 7f0c3f7fe640 10 client.?.objecter ms_handle_connect 0x7f0c580b44b0 2024-03-25T18:27:08.827 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:27:08.823+0000 7f0c3f7fe640 10 client.?.objecter resend_mon_ops 2024-03-25T18:27:08.827 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:27:08.824+0000 7f0c3f7fe640 10 client.?.objecter ms_handle_connect 0x7f0c580b1b00 2024-03-25T18:27:08.828 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:27:08.824+0000 7f0c3f7fe640 10 client.?.objecter resend_mon_ops 2024-03-25T18:27:08.828 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:27:08.824+0000 7f0c3f7fe640 10 client.?.objecter ms_handle_connect 0x7f0c580b4e70 2024-03-25T18:27:08.828 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:27:08.824+0000 7f0c3f7fe640 10 client.?.objecter resend_mon_ops 2024-03-25T18:27:08.828 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:27:08.826+0000 7f0c3f7fe640 1 -- 172.21.15.1:0/1021873481 <== mon.1 v1:172.21.15.115:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1600764418 0 0) 0x7f0c40003520 con 0x7f0c580b44b0 2024-03-25T18:27:08.828 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:27:08.826+0000 7f0c3f7fe640 10 monclient(hunting): _negotiate 2024-03-25T18:27:08.828 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:27:08.826+0000 7f0c3f7fe640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:27:08.829 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:27:08.826+0000 7f0c3f7fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:27:08.829 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:27:08.826+0000 7f0c3f7fe640 10 monclient(hunting): my global_id is 4213 2024-03-25T18:27:08.829 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:27:08.826+0000 7f0c3f7fe640 1 -- 172.21.15.1:0/1021873481 --> v1:172.21.15.115:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f0c44004eb0 con 0x7f0c580b44b0 2024-03-25T18:27:08.829 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:27:08.826+0000 7f0c3f7fe640 1 -- 172.21.15.1:0/1021873481 <== mon.2 v1:172.21.15.1:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 524587504 0 0) 0x7f0c58140d90 con 0x7f0c580b4e70 2024-03-25T18:27:08.829 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:27:08.826+0000 7f0c3f7fe640 10 monclient(hunting): _negotiate 2024-03-25T18:27:08.829 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:27:08.826+0000 7f0c3f7fe640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:27:08.829 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:27:08.826+0000 7f0c3f7fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:27:08.829 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:27:08.826+0000 7f0c3f7fe640 10 monclient(hunting): my global_id is 4220 2024-03-25T18:27:08.829 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:27:08.826+0000 7f0c3f7fe640 1 -- 172.21.15.1:0/1021873481 --> v1:172.21.15.1:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f0c40003520 con 0x7f0c580b4e70 2024-03-25T18:27:08.829 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:27:08.827+0000 7f0c3f7fe640 1 -- 172.21.15.1:0/1021873481 <== mon.0 v1:172.21.15.1:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3231901001 0 0) 0x7f0c581d85f0 con 0x7f0c580b1b00 2024-03-25T18:27:08.829 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:27:08.827+0000 7f0c3f7fe640 10 monclient(hunting): _negotiate 2024-03-25T18:27:08.829 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:27:08.827+0000 7f0c3f7fe640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:27:08.829 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:27:08.827+0000 7f0c3f7fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:27:08.829 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:27:08.827+0000 7f0c3f7fe640 10 monclient(hunting): my global_id is 4215 2024-03-25T18:27:08.830 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:27:08.827+0000 7f0c3f7fe640 1 -- 172.21.15.1:0/1021873481 --> v1:172.21.15.1:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f0c58140d90 con 0x7f0c580b1b00 2024-03-25T18:27:08.833 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:27:08.833+0000 7f0c3f7fe640 1 -- 172.21.15.1:0/1021873481 <== mon.1 v1:172.21.15.115:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 4290696956 0 0) 0x7f0c44004eb0 con 0x7f0c580b44b0 2024-03-25T18:27:08.833 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:27:08.833+0000 7f0c3f7fe640 1 -- 172.21.15.1:0/1021873481 --> v1:172.21.15.115:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f0c581d85f0 con 0x7f0c580b44b0 2024-03-25T18:27:08.833 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:27:08.833+0000 7f0c3f7fe640 1 -- 172.21.15.1:0/1021873481 <== mon.1 v1:172.21.15.115:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1586965025 0 0) 0x7f0c4c0017d0 con 0x7f0c580b44b0 2024-03-25T18:27:08.833 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:27:08.833+0000 7f0c3f7fe640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:27:08.833 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:27:08.833+0000 7f0c3f7fe640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-03-25T18:27:08.833 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:27:08.833+0000 7f0c3f7fe640 10 monclient(hunting): dump: 2024-03-25T18:27:08.834 INFO:teuthology.misc.health.smithi001.stderr:epoch 1 2024-03-25T18:27:08.834 INFO:teuthology.misc.health.smithi001.stderr:fsid c8ea72cf-9e99-4383-9b52-6c23d365fdfc 2024-03-25T18:27:08.834 INFO:teuthology.misc.health.smithi001.stderr:last_changed 2024-03-25T18:25:50.510519+0000 2024-03-25T18:27:08.834 INFO:teuthology.misc.health.smithi001.stderr:created 2024-03-25T18:25:50.510519+0000 2024-03-25T18:27:08.834 INFO:teuthology.misc.health.smithi001.stderr:min_mon_release 19 (squid) 2024-03-25T18:27:08.834 INFO:teuthology.misc.health.smithi001.stderr:election_strategy: 3 2024-03-25T18:27:08.834 INFO:teuthology.misc.health.smithi001.stderr:0: v1:172.21.15.1:6789/0 mon.a 2024-03-25T18:27:08.834 INFO:teuthology.misc.health.smithi001.stderr:1: v1:172.21.15.115:6789/0 mon.b 2024-03-25T18:27:08.834 INFO:teuthology.misc.health.smithi001.stderr:2: v1:172.21.15.1:6790/0 mon.c 2024-03-25T18:27:08.834 INFO:teuthology.misc.health.smithi001.stderr: 2024-03-25T18:27:08.834 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:27:08.833+0000 7f0c3f7fe640 10 monclient(hunting): _finish_auth 0 2024-03-25T18:27:08.835 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:27:08.834+0000 7f0c3f7fe640 1 -- 172.21.15.1:0/1021873481 <== mon.2 v1:172.21.15.1:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2266323832 0 0) 0x7f0c40003520 con 0x7f0c580b4e70 2024-03-25T18:27:08.836 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:27:08.834+0000 7f0c3f7fe640 1 -- 172.21.15.1:0/1021873481 --> v1:172.21.15.1:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f0c44004eb0 con 0x7f0c580b4e70 2024-03-25T18:27:08.836 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:27:08.834+0000 7f0c3f7fe640 1 -- 172.21.15.1:0/1021873481 <== mon.2 v1:172.21.15.1:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1586965025 0 0) 0x7f0c50001c30 con 0x7f0c580b4e70 2024-03-25T18:27:08.836 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:27:08.834+0000 7f0c3f7fe640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:27:08.836 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:27:08.834+0000 7f0c3f7fe640 10 monclient(hunting): got monmap 1 from mon.c (according to old e1) 2024-03-25T18:27:08.836 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:27:08.834+0000 7f0c3f7fe640 10 monclient(hunting): dump: 2024-03-25T18:27:08.836 INFO:teuthology.misc.health.smithi001.stderr:epoch 1 2024-03-25T18:27:08.836 INFO:teuthology.misc.health.smithi001.stderr:fsid c8ea72cf-9e99-4383-9b52-6c23d365fdfc 2024-03-25T18:27:08.836 INFO:teuthology.misc.health.smithi001.stderr:last_changed 2024-03-25T18:25:50.510519+0000 2024-03-25T18:27:08.836 INFO:teuthology.misc.health.smithi001.stderr:created 2024-03-25T18:25:50.510519+0000 2024-03-25T18:27:08.836 INFO:teuthology.misc.health.smithi001.stderr:min_mon_release 19 (squid) 2024-03-25T18:27:08.836 INFO:teuthology.misc.health.smithi001.stderr:election_strategy: 3 2024-03-25T18:27:08.836 INFO:teuthology.misc.health.smithi001.stderr:0: v1:172.21.15.1:6789/0 mon.a 2024-03-25T18:27:08.836 INFO:teuthology.misc.health.smithi001.stderr:1: v1:172.21.15.115:6789/0 mon.b 2024-03-25T18:27:08.836 INFO:teuthology.misc.health.smithi001.stderr:2: v1:172.21.15.1:6790/0 mon.c 2024-03-25T18:27:08.837 INFO:teuthology.misc.health.smithi001.stderr: 2024-03-25T18:27:08.837 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:27:08.834+0000 7f0c3f7fe640 1 -- 172.21.15.1:0/1021873481 <== mon.0 v1:172.21.15.1:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1497895084 0 0) 0x7f0c58140d90 con 0x7f0c580b1b00 2024-03-25T18:27:08.837 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:27:08.834+0000 7f0c3f7fe640 1 -- 172.21.15.1:0/1021873481 --> v1:172.21.15.1:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f0c40003520 con 0x7f0c580b1b00 2024-03-25T18:27:08.837 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:27:08.834+0000 7f0c3f7fe640 1 -- 172.21.15.1:0/1021873481 <== mon.0 v1:172.21.15.1:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1586965025 0 0) 0x7f0c48001be0 con 0x7f0c580b1b00 2024-03-25T18:27:08.837 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:27:08.834+0000 7f0c3f7fe640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:27:08.837 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:27:08.834+0000 7f0c3f7fe640 10 monclient(hunting): got monmap 1 from mon.a (according to old e1) 2024-03-25T18:27:08.837 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:27:08.834+0000 7f0c3f7fe640 10 monclient(hunting): dump: 2024-03-25T18:27:08.837 INFO:teuthology.misc.health.smithi001.stderr:epoch 1 2024-03-25T18:27:08.837 INFO:teuthology.misc.health.smithi001.stderr:fsid c8ea72cf-9e99-4383-9b52-6c23d365fdfc 2024-03-25T18:27:08.837 INFO:teuthology.misc.health.smithi001.stderr:last_changed 2024-03-25T18:25:50.510519+0000 2024-03-25T18:27:08.837 INFO:teuthology.misc.health.smithi001.stderr:created 2024-03-25T18:25:50.510519+0000 2024-03-25T18:27:08.837 INFO:teuthology.misc.health.smithi001.stderr:min_mon_release 19 (squid) 2024-03-25T18:27:08.837 INFO:teuthology.misc.health.smithi001.stderr:election_strategy: 3 2024-03-25T18:27:08.837 INFO:teuthology.misc.health.smithi001.stderr:0: v1:172.21.15.1:6789/0 mon.a 2024-03-25T18:27:08.838 INFO:teuthology.misc.health.smithi001.stderr:1: v1:172.21.15.115:6789/0 mon.b 2024-03-25T18:27:08.838 INFO:teuthology.misc.health.smithi001.stderr:2: v1:172.21.15.1:6790/0 mon.c 2024-03-25T18:27:08.838 INFO:teuthology.misc.health.smithi001.stderr: 2024-03-25T18:27:08.840 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:27:08.837+0000 7f0c3f7fe640 1 -- 172.21.15.1:0/1021873481 <== mon.1 v1:172.21.15.115:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 39593324 0 0) 0x7f0c581d85f0 con 0x7f0c580b44b0 2024-03-25T18:27:08.840 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:27:08.837+0000 7f0c3f7fe640 1 -- 172.21.15.1:0/1021873481 >> v1:172.21.15.1:6790/0 conn(0x7f0c580b4e70 legacy=0x7f0c581e30a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:27:08.840 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:27:08.837+0000 7f0c3f7fe640 1 -- 172.21.15.1:0/1021873481 >> v1:172.21.15.1:6789/0 conn(0x7f0c580b1b00 legacy=0x7f0c581d7ee0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:27:08.840 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:27:08.837+0000 7f0c3f7fe640 10 monclient: _finish_hunting 0 2024-03-25T18:27:08.840 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:27:08.837+0000 7f0c3f7fe640 1 monclient: found mon.b 2024-03-25T18:27:08.840 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:27:08.837+0000 7f0c3f7fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:27:08.840 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:27:08.837+0000 7f0c3f7fe640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.115:6789/0 2024-03-25T18:27:08.840 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:27:08.837+0000 7f0c3f7fe640 1 -- 172.21.15.1:0/1021873481 --> v1:172.21.15.115:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0c58140f20 con 0x7f0c580b44b0 2024-03-25T18:27:08.841 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:27:08.837+0000 7f0c3f7fe640 10 monclient: _finish_auth 0 2024-03-25T18:27:08.841 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:27:08.837+0000 7f0c3f7fe640 10 monclient: _check_auth_tickets 2024-03-25T18:27:08.841 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:27:08.837+0000 7f0c3f7fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:27:08.841 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:27:08.837+0000 7f0c5d674640 5 monclient: authenticate success, global_id 4213 2024-03-25T18:27:08.841 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:27:08.837+0000 7f0c5d674640 10 monclient: _renew_subs 2024-03-25T18:27:08.841 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:27:08.837+0000 7f0c5d674640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.115:6789/0 2024-03-25T18:27:08.841 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:27:08.837+0000 7f0c5d674640 1 -- 172.21.15.1:0/1021873481 --> v1:172.21.15.115:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0c58144050 con 0x7f0c580b44b0 2024-03-25T18:27:08.841 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:27:08.838+0000 7f0c5d674640 10 client.4213.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T18:27:08.841 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:27:08.838+0000 7f0c5d674640 10 monclient: _renew_subs 2024-03-25T18:27:08.841 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:27:08.838+0000 7f0c5d674640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.115:6789/0 2024-03-25T18:27:08.841 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:27:08.838+0000 7f0c5d674640 1 -- 172.21.15.1:0/1021873481 --> v1:172.21.15.115:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f0c581317d0 con 0x7f0c580b44b0 2024-03-25T18:27:08.841 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:27:08.838+0000 7f0c5d674640 1 librados: init done 2024-03-25T18:27:08.841 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:27:08.839+0000 7f0c5d674640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T18:27:08.842 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:27:08.839+0000 7f0c5d674640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:27:08.842 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:27:08.839+0000 7f0c5d674640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.115:6789/0 2024-03-25T18:27:08.842 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:27:08.839+0000 7f0c5d674640 1 -- 172.21.15.1:0/1021873481 --> v1:172.21.15.115:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0c581d7860 con 0x7f0c580b44b0 2024-03-25T18:27:08.866 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:27:08.865+0000 7f0c3f7fe640 1 -- 172.21.15.1:0/1021873481 <== mon.1 v1:172.21.15.115:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f0c4c001ce0 con 0x7f0c580b44b0 2024-03-25T18:27:08.866 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:27:08.865+0000 7f0c3f7fe640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:27:08.866 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:27:08.865+0000 7f0c3f7fe640 1 -- 172.21.15.1:0/1021873481 <== mon.1 v1:172.21.15.115:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1586965025 0 0) 0x7f0c4c004980 con 0x7f0c580b44b0 2024-03-25T18:27:08.866 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:27:08.865+0000 7f0c3f7fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:27:08.866 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:27:08.865+0000 7f0c3f7fe640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-25T18:27:08.866 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:27:08.865+0000 7f0c3f7fe640 10 monclient: dump: 2024-03-25T18:27:08.866 INFO:teuthology.misc.health.smithi001.stderr:epoch 1 2024-03-25T18:27:08.866 INFO:teuthology.misc.health.smithi001.stderr:fsid c8ea72cf-9e99-4383-9b52-6c23d365fdfc 2024-03-25T18:27:08.866 INFO:teuthology.misc.health.smithi001.stderr:last_changed 2024-03-25T18:25:50.510519+0000 2024-03-25T18:27:08.867 INFO:teuthology.misc.health.smithi001.stderr:created 2024-03-25T18:25:50.510519+0000 2024-03-25T18:27:08.867 INFO:teuthology.misc.health.smithi001.stderr:min_mon_release 19 (squid) 2024-03-25T18:27:08.867 INFO:teuthology.misc.health.smithi001.stderr:election_strategy: 3 2024-03-25T18:27:08.867 INFO:teuthology.misc.health.smithi001.stderr:0: v1:172.21.15.1:6789/0 mon.a 2024-03-25T18:27:08.867 INFO:teuthology.misc.health.smithi001.stderr:1: v1:172.21.15.115:6789/0 mon.b 2024-03-25T18:27:08.867 INFO:teuthology.misc.health.smithi001.stderr:2: v1:172.21.15.1:6790/0 mon.c 2024-03-25T18:27:08.867 INFO:teuthology.misc.health.smithi001.stderr: 2024-03-25T18:27:08.867 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:27:08.865+0000 7f0c557fa640 4 set_mon_vals no callback set 2024-03-25T18:27:08.868 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:27:08.866+0000 7f0c3f7fe640 1 -- 172.21.15.1:0/1021873481 <== mon.1 v1:172.21.15.115:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 21787975 0 0) 0x7f0c4c01a950 con 0x7f0c580b44b0 2024-03-25T18:27:08.868 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:27:08.867+0000 7f0c56ffd640 10 monclient: get_auth_request con 0x7f0c4406e0b0 auth_method 0 2024-03-25T18:27:08.868 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:27:08.867+0000 7f0c3f7fe640 10 client.4213.objecter ms_handle_connect 0x7f0c4406e0b0 2024-03-25T18:27:08.870 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:27:08.869+0000 7f0c3f7fe640 1 -- 172.21.15.1:0/1021873481 <== mon.1 v1:172.21.15.115:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 996254230 0 0) 0x7f0c4c0546d0 con 0x7f0c580b44b0 2024-03-25T18:27:08.870 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:27:08.869+0000 7f0c3f7fe640 10 client.4213.objecter ms_dispatch 0x7f0c58000b90 osd_map(10..10 src has 1..10) v4 2024-03-25T18:27:08.870 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:27:08.869+0000 7f0c3f7fe640 3 client.4213.objecter handle_osd_map got epochs [10,10] > 0 2024-03-25T18:27:08.870 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:27:08.869+0000 7f0c3f7fe640 3 client.4213.objecter handle_osd_map decoding full epoch 10 2024-03-25T18:27:08.870 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:27:08.869+0000 7f0c3f7fe640 20 client.4213.objecter dump_active .. 0 homeless 2024-03-25T18:27:09.136 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:27:09.136+0000 7f0c3f7fe640 1 -- 172.21.15.1:0/1021873481 <== mon.1 v1:172.21.15.115:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f0c4c050010 con 0x7f0c580b44b0 2024-03-25T18:27:09.137 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:27:09.136+0000 7f0c3f7fe640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:27:09.137 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:27:09.136+0000 7f0c3f7fe640 10 monclient: _finish_command 1 = system:0 2024-03-25T18:27:09.356 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:27:09.355+0000 7f0c5d674640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:27:09.356 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:27:09.355+0000 7f0c5d674640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:27:09.356 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:27:09.355+0000 7f0c5d674640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.115:6789/0 2024-03-25T18:27:09.356 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:27:09.355+0000 7f0c5d674640 1 -- 172.21.15.1:0/1021873481 --> v1:172.21.15.115:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f0c5804f730 con 0x7f0c580b44b0 2024-03-25T18:27:09.370 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:27:09.369+0000 7f0c3f7fe640 1 -- 172.21.15.1:0/1021873481 <== mon.1 v1:172.21.15.115:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 4092213348) 0x7f0c4c052110 con 0x7f0c580b44b0 2024-03-25T18:27:09.370 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:27:09.369+0000 7f0c3f7fe640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:27:09.370 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:27:09.369+0000 7f0c3f7fe640 10 monclient: _finish_command 2 = system:0 2024-03-25T18:27:09.370 INFO:teuthology.misc.health.smithi001.stdout: 2024-03-25T18:27:09.370 INFO:teuthology.misc.health.smithi001.stdout:{"epoch":10,"fsid":"c8ea72cf-9e99-4383-9b52-6c23d365fdfc","created":"2024-03-25T18:26:38.856638+0000","modified":"2024-03-25T18:26:44.224208+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T18:26:44.224208+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"b9f34922-3a84-49be-8426-807c1eceaefc","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"42bf306f-55e3-4634-9874-8b4b73ac6644","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"292cb23a-18a9-4207-a7cc-4d5727feaec0","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"45e0e9ab-898d-4673-9590-b4117bc5e84a","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"831a5a09-73ab-4a61-9872-088e76bdbc66","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"88cd903d-3b55-4840-9396-6e3cfe005282","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"89db7355-bc9d-46f4-8ab0-e8d9e744e026","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"cac47bee-0914-4f08-896f-348838882866","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T18:27:09.372 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:27:09.372+0000 7f0c5d674640 10 librados: watch_flush enter 2024-03-25T18:27:09.373 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:27:09.372+0000 7f0c5d674640 10 librados: watch_flush exit 2024-03-25T18:27:09.373 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:27:09.372+0000 7f0c5d674640 20 client.4213.objecter shutdown clearing up homeless session... 2024-03-25T18:27:09.373 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:27:09.372+0000 7f0c5d674640 10 client.4213.objecter successfully canceled tick 2024-03-25T18:27:09.373 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:27:09.372+0000 7f0c5d674640 1 -- 172.21.15.1:0/1021873481 >> v1:172.21.15.115:6800/4237379176 conn(0x7f0c4406e0b0 legacy=0x7f0c44070570 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:27:09.373 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:27:09.372+0000 7f0c5d674640 10 monclient: shutdown 2024-03-25T18:27:09.373 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:27:09.372+0000 7f0c5d674640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:27:09.373 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:27:09.372+0000 7f0c5d674640 1 -- 172.21.15.1:0/1021873481 >> v1:172.21.15.115:6789/0 conn(0x7f0c580b44b0 legacy=0x7f0c58130ee0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:27:09.373 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:27:09.372+0000 7f0c5d674640 1 -- 172.21.15.1:0/1021873481 shutdown_connections 2024-03-25T18:27:09.373 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:27:09.372+0000 7f0c5d674640 1 -- 172.21.15.1:0/1021873481 >> 172.21.15.1:0/1021873481 conn(0x7f0c580617a0 msgr2=0x7f0c5813eba0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:27:09.373 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:27:09.372+0000 7f0c5d674640 1 -- 172.21.15.1:0/1021873481 shutdown_connections 2024-03-25T18:27:09.373 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:27:09.372+0000 7f0c5d674640 1 -- 172.21.15.1:0/1021873481 wait complete. 2024-03-25T18:27:09.373 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:27:09.372+0000 7f0c5d674640 1 librados: shutdown 2024-03-25T18:27:09.383 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T18:27:10.898 INFO:tasks.ceph.osd.5.smithi115.stderr:==00:00:00:26.515 35381== Warning: unimplemented fcntl command: 1038 2024-03-25T18:27:11.135 INFO:tasks.ceph.osd.7.smithi115.stderr:==00:00:00:26.747 35383== Warning: unimplemented fcntl command: 1038 2024-03-25T18:27:11.310 INFO:tasks.ceph.osd.4.smithi115.stderr:==00:00:00:26.927 35380== Warning: unimplemented fcntl command: 1038 2024-03-25T18:27:11.564 INFO:tasks.ceph.osd.6.smithi115.stderr:==00:00:00:27.180 35382== Warning: unimplemented fcntl command: 1038 2024-03-25T18:27:11.834 INFO:tasks.ceph.osd.3.smithi001.stderr:==00:00:00:27.437 35191== Warning: unimplemented fcntl command: 1038 2024-03-25T18:27:12.287 INFO:tasks.ceph.osd.0.smithi001.stderr:==00:00:00:27.895 35190== Warning: unimplemented fcntl command: 1038 2024-03-25T18:27:12.308 INFO:tasks.ceph.osd.1.smithi001.stderr:==00:00:00:27.925 35189== Warning: unimplemented fcntl command: 1038 2024-03-25T18:27:13.062 INFO:tasks.ceph.osd.2.smithi001.stderr:==00:00:00:28.654 35192== Warning: unimplemented fcntl command: 1038 2024-03-25T18:27:16.192 DEBUG:teuthology.orchestra.run.smithi001:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T18:27:16.282 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:27:16.281+0000 7f030b932640 10 monclient: get_monmap_and_config 2024-03-25T18:27:16.282 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:27:16.281+0000 7f030b932640 10 monclient: build_initial_monmap 2024-03-25T18:27:16.282 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:27:16.281+0000 7f030b932640 10 monclient: monmap: 2024-03-25T18:27:16.282 INFO:teuthology.misc.health.smithi001.stderr:epoch 0 2024-03-25T18:27:16.282 INFO:teuthology.misc.health.smithi001.stderr:fsid c8ea72cf-9e99-4383-9b52-6c23d365fdfc 2024-03-25T18:27:16.282 INFO:teuthology.misc.health.smithi001.stderr:last_changed 2024-03-25T18:27:16.282227+0000 2024-03-25T18:27:16.283 INFO:teuthology.misc.health.smithi001.stderr:created 2024-03-25T18:27:16.282227+0000 2024-03-25T18:27:16.283 INFO:teuthology.misc.health.smithi001.stderr:min_mon_release 0 (unknown) 2024-03-25T18:27:16.283 INFO:teuthology.misc.health.smithi001.stderr:election_strategy: 3 2024-03-25T18:27:16.283 INFO:teuthology.misc.health.smithi001.stderr:0: v1:172.21.15.1:6789/0 mon.noname-a 2024-03-25T18:27:16.283 INFO:teuthology.misc.health.smithi001.stderr:1: v1:172.21.15.115:6789/0 mon.noname-c 2024-03-25T18:27:16.283 INFO:teuthology.misc.health.smithi001.stderr:2: v1:172.21.15.1:6790/0 mon.noname-b 2024-03-25T18:27:16.283 INFO:teuthology.misc.health.smithi001.stderr: 2024-03-25T18:27:16.284 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:27:16.282+0000 7f030b932640 1 Processor -- start 2024-03-25T18:27:16.284 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:27:16.282+0000 7f030b932640 1 -- start start 2024-03-25T18:27:16.285 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:27:16.282+0000 7f030b932640 10 monclient: init 2024-03-25T18:27:16.285 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:27:16.283+0000 7f030b932640 10 monclient: _reopen_session rank -1 2024-03-25T18:27:16.285 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:27:16.283+0000 7f030b932640 10 monclient: _add_conns ranks=[1,0,2] 2024-03-25T18:27:16.285 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:27:16.283+0000 7f030b932640 10 monclient(hunting): picked mon.noname-c con 0x7f03040b4e70 addr v1:172.21.15.115:6789/0 2024-03-25T18:27:16.285 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:27:16.283+0000 7f030b932640 10 monclient(hunting): picked mon.noname-a con 0x7f03040b44b0 addr v1:172.21.15.1:6789/0 2024-03-25T18:27:16.285 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:27:16.283+0000 7f030b932640 10 monclient(hunting): picked mon.noname-b con 0x7f03040b1b00 addr v1:172.21.15.1:6790/0 2024-03-25T18:27:16.285 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:27:16.283+0000 7f030b932640 1 -- --> v1:172.21.15.1:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f03040a96f0 con 0x7f03040b44b0 2024-03-25T18:27:16.285 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:27:16.283+0000 7f030b932640 1 -- --> v1:172.21.15.115:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f03040ab0e0 con 0x7f03040b4e70 2024-03-25T18:27:16.285 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:27:16.283+0000 7f030b932640 1 -- --> v1:172.21.15.1:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0304145f70 con 0x7f03040b1b00 2024-03-25T18:27:16.285 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:27:16.283+0000 7f030b932640 10 monclient(hunting): _renew_subs 2024-03-25T18:27:16.285 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:27:16.283+0000 7f030b932640 10 monclient(hunting): authenticate will time out at 7516.752958s 2024-03-25T18:27:16.285 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:27:16.283+0000 7f0309ecf640 1 --1- >> v1:172.21.15.1:6790/0 conn(0x7f03040b1b00 0x7f030414dc00 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.1:6790/0 says I am v1:172.21.15.1:42418/0 (socket says 172.21.15.1:42418) 2024-03-25T18:27:16.285 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:27:16.283+0000 7f0309ecf640 1 -- 172.21.15.1:0/2924892433 learned_addr learned my addr 172.21.15.1:0/2924892433 (peer_addr_for_me v1:172.21.15.1:0/0) 2024-03-25T18:27:16.287 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:27:16.286+0000 7f02fbfff640 1 -- 172.21.15.1:0/2924892433 <== mon.2 v1:172.21.15.1:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3933618968 0 0) 0x7f0304145f70 con 0x7f03040b1b00 2024-03-25T18:27:16.287 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:27:16.286+0000 7f02fbfff640 10 monclient(hunting): _negotiate 2024-03-25T18:27:16.287 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:27:16.286+0000 7f02fbfff640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:27:16.287 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:27:16.286+0000 7f02fbfff640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:27:16.287 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:27:16.286+0000 7f02fbfff640 10 monclient(hunting): my global_id is 4223 2024-03-25T18:27:16.287 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:27:16.286+0000 7f02fbfff640 1 -- 172.21.15.1:0/2924892433 --> v1:172.21.15.1:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f02f0003520 con 0x7f03040b1b00 2024-03-25T18:27:16.289 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:27:16.287+0000 7f02fbfff640 1 -- 172.21.15.1:0/2924892433 <== mon.1 v1:172.21.15.115:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3844711397 0 0) 0x7f03040ab0e0 con 0x7f03040b4e70 2024-03-25T18:27:16.289 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:27:16.287+0000 7f02fbfff640 10 monclient(hunting): _negotiate 2024-03-25T18:27:16.289 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:27:16.287+0000 7f02fbfff640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:27:16.289 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:27:16.287+0000 7f02fbfff640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:27:16.289 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:27:16.287+0000 7f02fbfff640 10 monclient(hunting): my global_id is 4216 2024-03-25T18:27:16.289 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:27:16.287+0000 7f02fbfff640 1 -- 172.21.15.1:0/2924892433 --> v1:172.21.15.115:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f0304145f70 con 0x7f03040b4e70 2024-03-25T18:27:16.289 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:27:16.287+0000 7f02fbfff640 1 -- 172.21.15.1:0/2924892433 <== mon.0 v1:172.21.15.1:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1260518265 0 0) 0x7f03040a96f0 con 0x7f03040b44b0 2024-03-25T18:27:16.289 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:27:16.287+0000 7f02fbfff640 10 monclient(hunting): _negotiate 2024-03-25T18:27:16.289 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:27:16.287+0000 7f02fbfff640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:27:16.290 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:27:16.287+0000 7f02fbfff640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:27:16.290 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:27:16.287+0000 7f02fbfff640 10 monclient(hunting): my global_id is 4218 2024-03-25T18:27:16.290 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:27:16.287+0000 7f02fbfff640 1 -- 172.21.15.1:0/2924892433 --> v1:172.21.15.1:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f02f00039d0 con 0x7f03040b44b0 2024-03-25T18:27:16.292 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:27:16.290+0000 7f02fbfff640 1 -- 172.21.15.1:0/2924892433 <== mon.2 v1:172.21.15.1:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 4123759487 0 0) 0x7f02f0003520 con 0x7f03040b1b00 2024-03-25T18:27:16.292 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:27:16.290+0000 7f02fbfff640 1 -- 172.21.15.1:0/2924892433 >> v1:172.21.15.115:6789/0 conn(0x7f03040b4e70 legacy=0x7f030413f940 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:27:16.292 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:27:16.290+0000 7f02fbfff640 1 -- 172.21.15.1:0/2924892433 >> v1:172.21.15.1:6789/0 conn(0x7f03040b44b0 legacy=0x7f0304145860 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:27:16.293 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:27:16.290+0000 7f02fbfff640 10 monclient: _finish_hunting 0 2024-03-25T18:27:16.293 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:27:16.290+0000 7f02fbfff640 1 monclient: found mon.noname-b 2024-03-25T18:27:16.293 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:27:16.290+0000 7f02fbfff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:27:16.293 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:27:16.290+0000 7f02fbfff640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.1:6790/0 2024-03-25T18:27:16.293 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:27:16.290+0000 7f02fbfff640 1 -- 172.21.15.1:0/2924892433 --> v1:172.21.15.1:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f03040a9b80 con 0x7f03040b1b00 2024-03-25T18:27:16.293 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:27:16.290+0000 7f02fbfff640 10 monclient: _finish_auth 0 2024-03-25T18:27:16.293 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:27:16.290+0000 7f02fbfff640 10 monclient: _check_auth_tickets 2024-03-25T18:27:16.293 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:27:16.290+0000 7f02fbfff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:27:16.293 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:27:16.290+0000 7f02fbfff640 1 -- 172.21.15.1:0/2924892433 <== mon.2 v1:172.21.15.1:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1586965025 0 0) 0x7f02fc001a10 con 0x7f03040b1b00 2024-03-25T18:27:16.293 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:27:16.290+0000 7f02fbfff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:27:16.293 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:27:16.290+0000 7f02fbfff640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-03-25T18:27:16.293 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:27:16.290+0000 7f02fbfff640 10 monclient: dump: 2024-03-25T18:27:16.293 INFO:teuthology.misc.health.smithi001.stderr:epoch 1 2024-03-25T18:27:16.293 INFO:teuthology.misc.health.smithi001.stderr:fsid c8ea72cf-9e99-4383-9b52-6c23d365fdfc 2024-03-25T18:27:16.294 INFO:teuthology.misc.health.smithi001.stderr:last_changed 2024-03-25T18:25:50.510519+0000 2024-03-25T18:27:16.294 INFO:teuthology.misc.health.smithi001.stderr:created 2024-03-25T18:25:50.510519+0000 2024-03-25T18:27:16.294 INFO:teuthology.misc.health.smithi001.stderr:min_mon_release 19 (squid) 2024-03-25T18:27:16.294 INFO:teuthology.misc.health.smithi001.stderr:election_strategy: 3 2024-03-25T18:27:16.294 INFO:teuthology.misc.health.smithi001.stderr:0: v1:172.21.15.1:6789/0 mon.a 2024-03-25T18:27:16.294 INFO:teuthology.misc.health.smithi001.stderr:1: v1:172.21.15.115:6789/0 mon.b 2024-03-25T18:27:16.294 INFO:teuthology.misc.health.smithi001.stderr:2: v1:172.21.15.1:6790/0 mon.c 2024-03-25T18:27:16.294 INFO:teuthology.misc.health.smithi001.stderr: 2024-03-25T18:27:16.294 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:27:16.290+0000 7f030b932640 5 monclient: authenticate success, global_id 4223 2024-03-25T18:27:16.294 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:27:16.290+0000 7f030b932640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T18:27:16.294 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:27:16.292+0000 7f02fbfff640 1 -- 172.21.15.1:0/2924892433 <== mon.2 v1:172.21.15.1:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f02fc001bf0 con 0x7f03040b1b00 2024-03-25T18:27:16.294 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:27:16.292+0000 7f02fbfff640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:27:16.294 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:27:16.292+0000 7f030b932640 10 monclient: get_monmap_and_config success 2024-03-25T18:27:16.294 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:27:16.292+0000 7f030b932640 4 set_mon_vals no callback set 2024-03-25T18:27:16.298 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:27:16.293+0000 7f02fbfff640 1 -- 172.21.15.1:0/2924892433 <== mon.2 v1:172.21.15.1:6790/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1586965025 0 0) 0x7f02fc001a10 con 0x7f03040b1b00 2024-03-25T18:27:16.298 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:27:16.293+0000 7f02fbfff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:27:16.298 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:27:16.293+0000 7f02fbfff640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-25T18:27:16.298 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:27:16.293+0000 7f02fbfff640 10 monclient: dump: 2024-03-25T18:27:16.298 INFO:teuthology.misc.health.smithi001.stderr:epoch 1 2024-03-25T18:27:16.298 INFO:teuthology.misc.health.smithi001.stderr:fsid c8ea72cf-9e99-4383-9b52-6c23d365fdfc 2024-03-25T18:27:16.298 INFO:teuthology.misc.health.smithi001.stderr:last_changed 2024-03-25T18:25:50.510519+0000 2024-03-25T18:27:16.299 INFO:teuthology.misc.health.smithi001.stderr:created 2024-03-25T18:25:50.510519+0000 2024-03-25T18:27:16.299 INFO:teuthology.misc.health.smithi001.stderr:min_mon_release 19 (squid) 2024-03-25T18:27:16.299 INFO:teuthology.misc.health.smithi001.stderr:election_strategy: 3 2024-03-25T18:27:16.299 INFO:teuthology.misc.health.smithi001.stderr:0: v1:172.21.15.1:6789/0 mon.a 2024-03-25T18:27:16.299 INFO:teuthology.misc.health.smithi001.stderr:1: v1:172.21.15.115:6789/0 mon.b 2024-03-25T18:27:16.299 INFO:teuthology.misc.health.smithi001.stderr:2: v1:172.21.15.1:6790/0 mon.c 2024-03-25T18:27:16.299 INFO:teuthology.misc.health.smithi001.stderr: 2024-03-25T18:27:16.299 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:27:16.293+0000 7f030b932640 10 monclient: shutdown 2024-03-25T18:27:16.299 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:27:16.293+0000 7f030b932640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:27:16.299 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:27:16.293+0000 7f030b932640 1 -- 172.21.15.1:0/2924892433 >> v1:172.21.15.1:6790/0 conn(0x7f03040b1b00 legacy=0x7f030414dc00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:27:16.299 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:27:16.293+0000 7f030b932640 1 -- 172.21.15.1:0/2924892433 shutdown_connections 2024-03-25T18:27:16.299 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:27:16.293+0000 7f030b932640 1 -- 172.21.15.1:0/2924892433 >> 172.21.15.1:0/2924892433 conn(0x7f03040617a0 msgr2=0x7f0304061bb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:27:16.299 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:27:16.293+0000 7f030b932640 1 -- 172.21.15.1:0/2924892433 shutdown_connections 2024-03-25T18:27:16.299 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:27:16.293+0000 7f030b932640 1 -- 172.21.15.1:0/2924892433 wait complete. 2024-03-25T18:27:16.299 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:27:16.294+0000 7f030b932640 10 monclient: build_initial_monmap 2024-03-25T18:27:16.300 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:27:16.294+0000 7f030b932640 10 monclient: monmap: 2024-03-25T18:27:16.300 INFO:teuthology.misc.health.smithi001.stderr:epoch 0 2024-03-25T18:27:16.300 INFO:teuthology.misc.health.smithi001.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T18:27:16.300 INFO:teuthology.misc.health.smithi001.stderr:last_changed 0.000000 2024-03-25T18:27:16.300 INFO:teuthology.misc.health.smithi001.stderr:created 0.000000 2024-03-25T18:27:16.300 INFO:teuthology.misc.health.smithi001.stderr:min_mon_release 0 (unknown) 2024-03-25T18:27:16.300 INFO:teuthology.misc.health.smithi001.stderr:election_strategy: 1 2024-03-25T18:27:16.300 INFO:teuthology.misc.health.smithi001.stderr:0: v1:172.21.15.1:6789/0 mon.noname-a 2024-03-25T18:27:16.300 INFO:teuthology.misc.health.smithi001.stderr:1: v1:172.21.15.115:6789/0 mon.noname-b 2024-03-25T18:27:16.300 INFO:teuthology.misc.health.smithi001.stderr:2: v1:172.21.15.1:6790/0 mon.noname-c 2024-03-25T18:27:16.300 INFO:teuthology.misc.health.smithi001.stderr: 2024-03-25T18:27:16.300 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:27:16.295+0000 7f030b932640 1 librados: starting msgr at 2024-03-25T18:27:16.300 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:27:16.295+0000 7f030b932640 1 librados: starting objecter 2024-03-25T18:27:16.300 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:27:16.295+0000 7f030b932640 1 Processor -- start 2024-03-25T18:27:16.300 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:27:16.295+0000 7f030b932640 1 -- start start 2024-03-25T18:27:16.301 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:27:16.295+0000 7f030b932640 1 librados: setting wanted keys 2024-03-25T18:27:16.301 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:27:16.295+0000 7f030b932640 1 librados: calling monclient init 2024-03-25T18:27:16.301 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:27:16.295+0000 7f030b932640 10 monclient: init 2024-03-25T18:27:16.301 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:27:16.295+0000 7f030b932640 10 monclient: _reopen_session rank -1 2024-03-25T18:27:16.301 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:27:16.295+0000 7f030b932640 10 monclient: _add_conns ranks=[0,1,2] 2024-03-25T18:27:16.301 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:27:16.295+0000 7f030b932640 10 monclient(hunting): picked mon.noname-a con 0x7f03040b1b00 addr v1:172.21.15.1:6789/0 2024-03-25T18:27:16.301 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:27:16.295+0000 7f030b932640 10 monclient(hunting): picked mon.noname-b con 0x7f03040b44b0 addr v1:172.21.15.115:6789/0 2024-03-25T18:27:16.301 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:27:16.295+0000 7f030b932640 10 monclient(hunting): picked mon.noname-c con 0x7f03040b4e70 addr v1:172.21.15.1:6790/0 2024-03-25T18:27:16.301 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:27:16.295+0000 7f030b932640 1 -- --> v1:172.21.15.1:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0304142150 con 0x7f03040b1b00 2024-03-25T18:27:16.301 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:27:16.296+0000 7f030b932640 1 -- --> v1:172.21.15.115:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f03041435a0 con 0x7f03040b44b0 2024-03-25T18:27:16.301 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:27:16.296+0000 7f030b932640 1 -- --> v1:172.21.15.1:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0304143880 con 0x7f03040b4e70 2024-03-25T18:27:16.301 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:27:16.296+0000 7f030b932640 10 monclient(hunting): _renew_subs 2024-03-25T18:27:16.301 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:27:16.296+0000 7f030b932640 10 monclient(hunting): authenticate will time out at 7516.765679s 2024-03-25T18:27:16.301 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:27:16.296+0000 7f0309ecf640 1 --1- >> v1:172.21.15.1:6790/0 conn(0x7f03040b4e70 0x7f03041e2090 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.1:6790/0 says I am v1:172.21.15.1:42428/0 (socket says 172.21.15.1:42428) 2024-03-25T18:27:16.301 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:27:16.296+0000 7f0309ecf640 1 -- 172.21.15.1:0/3675566436 learned_addr learned my addr 172.21.15.1:0/3675566436 (peer_addr_for_me v1:172.21.15.1:0/0) 2024-03-25T18:27:16.302 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:27:16.296+0000 7f02f9ffb640 10 client.?.objecter ms_handle_connect 0x7f03040b4e70 2024-03-25T18:27:16.302 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:27:16.296+0000 7f02f9ffb640 10 client.?.objecter resend_mon_ops 2024-03-25T18:27:16.302 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:27:16.297+0000 7f02f9ffb640 10 client.?.objecter ms_handle_connect 0x7f03040b44b0 2024-03-25T18:27:16.302 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:27:16.297+0000 7f02f9ffb640 10 client.?.objecter resend_mon_ops 2024-03-25T18:27:16.302 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:27:16.297+0000 7f02f9ffb640 10 client.?.objecter ms_handle_connect 0x7f03040b1b00 2024-03-25T18:27:16.302 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:27:16.297+0000 7f02f9ffb640 10 client.?.objecter resend_mon_ops 2024-03-25T18:27:16.303 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:27:16.299+0000 7f02f9ffb640 1 -- 172.21.15.1:0/3675566436 <== mon.2 v1:172.21.15.1:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1286580610 0 0) 0x7f0304143880 con 0x7f03040b4e70 2024-03-25T18:27:16.303 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:27:16.299+0000 7f02f9ffb640 10 monclient(hunting): _negotiate 2024-03-25T18:27:16.303 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:27:16.299+0000 7f02f9ffb640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:27:16.303 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:27:16.299+0000 7f02f9ffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:27:16.304 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:27:16.299+0000 7f02f9ffb640 10 monclient(hunting): my global_id is 4226 2024-03-25T18:27:16.304 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:27:16.299+0000 7f02f9ffb640 1 -- 172.21.15.1:0/3675566436 --> v1:172.21.15.1:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f02e8004eb0 con 0x7f03040b4e70 2024-03-25T18:27:16.304 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:27:16.300+0000 7f02f9ffb640 1 -- 172.21.15.1:0/3675566436 <== mon.1 v1:172.21.15.115:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1301251497 0 0) 0x7f03041435a0 con 0x7f03040b44b0 2024-03-25T18:27:16.304 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:27:16.300+0000 7f02f9ffb640 10 monclient(hunting): _negotiate 2024-03-25T18:27:16.304 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:27:16.300+0000 7f02f9ffb640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:27:16.304 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:27:16.300+0000 7f02f9ffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:27:16.304 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:27:16.300+0000 7f02f9ffb640 10 monclient(hunting): my global_id is 4219 2024-03-25T18:27:16.304 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:27:16.300+0000 7f02f9ffb640 1 -- 172.21.15.1:0/3675566436 --> v1:172.21.15.115:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f0304143880 con 0x7f03040b44b0 2024-03-25T18:27:16.304 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:27:16.300+0000 7f02f9ffb640 1 -- 172.21.15.1:0/3675566436 <== mon.0 v1:172.21.15.1:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2398644015 0 0) 0x7f0304142150 con 0x7f03040b1b00 2024-03-25T18:27:16.304 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:27:16.300+0000 7f02f9ffb640 10 monclient(hunting): _negotiate 2024-03-25T18:27:16.304 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:27:16.300+0000 7f02f9ffb640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:27:16.304 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:27:16.300+0000 7f02f9ffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:27:16.304 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:27:16.300+0000 7f02f9ffb640 10 monclient(hunting): my global_id is 4221 2024-03-25T18:27:16.304 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:27:16.300+0000 7f02f9ffb640 1 -- 172.21.15.1:0/3675566436 --> v1:172.21.15.1:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f03041435a0 con 0x7f03040b1b00 2024-03-25T18:27:16.307 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:27:16.306+0000 7f02f9ffb640 1 -- 172.21.15.1:0/3675566436 <== mon.2 v1:172.21.15.1:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2619361044 0 0) 0x7f02e8004eb0 con 0x7f03040b4e70 2024-03-25T18:27:16.307 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:27:16.306+0000 7f02f9ffb640 1 -- 172.21.15.1:0/3675566436 --> v1:172.21.15.1:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f0304142150 con 0x7f03040b4e70 2024-03-25T18:27:16.307 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:27:16.306+0000 7f02f9ffb640 1 -- 172.21.15.1:0/3675566436 <== mon.2 v1:172.21.15.1:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1586965025 0 0) 0x7f02fc001a10 con 0x7f03040b4e70 2024-03-25T18:27:16.307 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:27:16.306+0000 7f02f9ffb640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:27:16.308 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:27:16.306+0000 7f02f9ffb640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-03-25T18:27:16.308 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:27:16.306+0000 7f02f9ffb640 10 monclient(hunting): dump: 2024-03-25T18:27:16.308 INFO:teuthology.misc.health.smithi001.stderr:epoch 1 2024-03-25T18:27:16.308 INFO:teuthology.misc.health.smithi001.stderr:fsid c8ea72cf-9e99-4383-9b52-6c23d365fdfc 2024-03-25T18:27:16.308 INFO:teuthology.misc.health.smithi001.stderr:last_changed 2024-03-25T18:25:50.510519+0000 2024-03-25T18:27:16.308 INFO:teuthology.misc.health.smithi001.stderr:created 2024-03-25T18:25:50.510519+0000 2024-03-25T18:27:16.308 INFO:teuthology.misc.health.smithi001.stderr:min_mon_release 19 (squid) 2024-03-25T18:27:16.308 INFO:teuthology.misc.health.smithi001.stderr:election_strategy: 3 2024-03-25T18:27:16.308 INFO:teuthology.misc.health.smithi001.stderr:0: v1:172.21.15.1:6789/0 mon.a 2024-03-25T18:27:16.308 INFO:teuthology.misc.health.smithi001.stderr:1: v1:172.21.15.115:6789/0 mon.b 2024-03-25T18:27:16.308 INFO:teuthology.misc.health.smithi001.stderr:2: v1:172.21.15.1:6790/0 mon.c 2024-03-25T18:27:16.308 INFO:teuthology.misc.health.smithi001.stderr: 2024-03-25T18:27:16.308 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:27:16.306+0000 7f02f9ffb640 10 monclient(hunting): _finish_auth 0 2024-03-25T18:27:16.310 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:27:16.307+0000 7f02f9ffb640 1 -- 172.21.15.1:0/3675566436 <== mon.1 v1:172.21.15.115:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2748737127 0 0) 0x7f0304143880 con 0x7f03040b44b0 2024-03-25T18:27:16.310 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:27:16.307+0000 7f02f9ffb640 1 -- 172.21.15.1:0/3675566436 --> v1:172.21.15.115:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f02e8004eb0 con 0x7f03040b44b0 2024-03-25T18:27:16.310 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:27:16.307+0000 7f02f9ffb640 1 -- 172.21.15.1:0/3675566436 <== mon.1 v1:172.21.15.115:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1586965025 0 0) 0x7f02f40018b0 con 0x7f03040b44b0 2024-03-25T18:27:16.310 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:27:16.307+0000 7f02f9ffb640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:27:16.310 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:27:16.307+0000 7f02f9ffb640 10 monclient(hunting): got monmap 1 from mon.b (according to old e1) 2024-03-25T18:27:16.310 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:27:16.307+0000 7f02f9ffb640 10 monclient(hunting): dump: 2024-03-25T18:27:16.310 INFO:teuthology.misc.health.smithi001.stderr:epoch 1 2024-03-25T18:27:16.310 INFO:teuthology.misc.health.smithi001.stderr:fsid c8ea72cf-9e99-4383-9b52-6c23d365fdfc 2024-03-25T18:27:16.310 INFO:teuthology.misc.health.smithi001.stderr:last_changed 2024-03-25T18:25:50.510519+0000 2024-03-25T18:27:16.310 INFO:teuthology.misc.health.smithi001.stderr:created 2024-03-25T18:25:50.510519+0000 2024-03-25T18:27:16.310 INFO:teuthology.misc.health.smithi001.stderr:min_mon_release 19 (squid) 2024-03-25T18:27:16.311 INFO:teuthology.misc.health.smithi001.stderr:election_strategy: 3 2024-03-25T18:27:16.311 INFO:teuthology.misc.health.smithi001.stderr:0: v1:172.21.15.1:6789/0 mon.a 2024-03-25T18:27:16.311 INFO:teuthology.misc.health.smithi001.stderr:1: v1:172.21.15.115:6789/0 mon.b 2024-03-25T18:27:16.311 INFO:teuthology.misc.health.smithi001.stderr:2: v1:172.21.15.1:6790/0 mon.c 2024-03-25T18:27:16.311 INFO:teuthology.misc.health.smithi001.stderr: 2024-03-25T18:27:16.311 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:27:16.307+0000 7f02f9ffb640 1 -- 172.21.15.1:0/3675566436 <== mon.0 v1:172.21.15.1:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 382395523 0 0) 0x7f03041435a0 con 0x7f03040b1b00 2024-03-25T18:27:16.311 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:27:16.307+0000 7f02f9ffb640 1 -- 172.21.15.1:0/3675566436 --> v1:172.21.15.1:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f0304143880 con 0x7f03040b1b00 2024-03-25T18:27:16.311 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:27:16.307+0000 7f02f9ffb640 1 -- 172.21.15.1:0/3675566436 <== mon.0 v1:172.21.15.1:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1586965025 0 0) 0x7f0300001700 con 0x7f03040b1b00 2024-03-25T18:27:16.311 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:27:16.307+0000 7f02f9ffb640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:27:16.311 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:27:16.307+0000 7f02f9ffb640 10 monclient(hunting): got monmap 1 from mon.a (according to old e1) 2024-03-25T18:27:16.311 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:27:16.307+0000 7f02f9ffb640 10 monclient(hunting): dump: 2024-03-25T18:27:16.311 INFO:teuthology.misc.health.smithi001.stderr:epoch 1 2024-03-25T18:27:16.311 INFO:teuthology.misc.health.smithi001.stderr:fsid c8ea72cf-9e99-4383-9b52-6c23d365fdfc 2024-03-25T18:27:16.311 INFO:teuthology.misc.health.smithi001.stderr:last_changed 2024-03-25T18:25:50.510519+0000 2024-03-25T18:27:16.311 INFO:teuthology.misc.health.smithi001.stderr:created 2024-03-25T18:25:50.510519+0000 2024-03-25T18:27:16.312 INFO:teuthology.misc.health.smithi001.stderr:min_mon_release 19 (squid) 2024-03-25T18:27:16.312 INFO:teuthology.misc.health.smithi001.stderr:election_strategy: 3 2024-03-25T18:27:16.312 INFO:teuthology.misc.health.smithi001.stderr:0: v1:172.21.15.1:6789/0 mon.a 2024-03-25T18:27:16.312 INFO:teuthology.misc.health.smithi001.stderr:1: v1:172.21.15.115:6789/0 mon.b 2024-03-25T18:27:16.312 INFO:teuthology.misc.health.smithi001.stderr:2: v1:172.21.15.1:6790/0 mon.c 2024-03-25T18:27:16.312 INFO:teuthology.misc.health.smithi001.stderr: 2024-03-25T18:27:16.315 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:27:16.310+0000 7f02f9ffb640 1 -- 172.21.15.1:0/3675566436 <== mon.2 v1:172.21.15.1:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 3638929174 0 0) 0x7f0304142150 con 0x7f03040b4e70 2024-03-25T18:27:16.315 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:27:16.310+0000 7f02f9ffb640 1 -- 172.21.15.1:0/3675566436 >> v1:172.21.15.115:6789/0 conn(0x7f03040b44b0 legacy=0x7f03041d75e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:27:16.315 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:27:16.310+0000 7f02f9ffb640 1 -- 172.21.15.1:0/3675566436 >> v1:172.21.15.1:6789/0 conn(0x7f03040b1b00 legacy=0x7f0304141420 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:27:16.315 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:27:16.310+0000 7f02f9ffb640 10 monclient: _finish_hunting 0 2024-03-25T18:27:16.315 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:27:16.310+0000 7f02f9ffb640 1 monclient: found mon.c 2024-03-25T18:27:16.315 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:27:16.310+0000 7f02f9ffb640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:27:16.315 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:27:16.310+0000 7f02f9ffb640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.1:6790/0 2024-03-25T18:27:16.315 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:27:16.310+0000 7f02f9ffb640 1 -- 172.21.15.1:0/3675566436 --> v1:172.21.15.1:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f03041425e0 con 0x7f03040b4e70 2024-03-25T18:27:16.315 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:27:16.310+0000 7f02f9ffb640 10 monclient: _finish_auth 0 2024-03-25T18:27:16.315 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:27:16.310+0000 7f02f9ffb640 10 monclient: _check_auth_tickets 2024-03-25T18:27:16.315 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:27:16.310+0000 7f02f9ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:27:16.315 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:27:16.310+0000 7f030b932640 5 monclient: authenticate success, global_id 4226 2024-03-25T18:27:16.315 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:27:16.310+0000 7f030b932640 10 monclient: _renew_subs 2024-03-25T18:27:16.316 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:27:16.310+0000 7f030b932640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.1:6790/0 2024-03-25T18:27:16.316 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:27:16.310+0000 7f030b932640 1 -- 172.21.15.1:0/3675566436 --> v1:172.21.15.1:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0304141be0 con 0x7f03040b4e70 2024-03-25T18:27:16.316 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:27:16.310+0000 7f030b932640 10 client.4226.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T18:27:16.316 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:27:16.310+0000 7f030b932640 10 monclient: _renew_subs 2024-03-25T18:27:16.316 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:27:16.310+0000 7f030b932640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.1:6790/0 2024-03-25T18:27:16.316 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:27:16.310+0000 7f030b932640 1 -- 172.21.15.1:0/3675566436 --> v1:172.21.15.1:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f03041d7ed0 con 0x7f03040b4e70 2024-03-25T18:27:16.316 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:27:16.310+0000 7f030b932640 1 librados: init done 2024-03-25T18:27:16.316 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:27:16.311+0000 7f02ef7fe640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T18:27:16.316 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:27:16.311+0000 7f02ef7fe640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:27:16.316 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:27:16.312+0000 7f02ef7fe640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.1:6790/0 2024-03-25T18:27:16.316 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:27:16.312+0000 7f02ef7fe640 1 -- 172.21.15.1:0/3675566436 --> v1:172.21.15.1:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f02e4005080 con 0x7f03040b4e70 2024-03-25T18:27:16.316 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:27:16.313+0000 7f02f9ffb640 1 -- 172.21.15.1:0/3675566436 <== mon.2 v1:172.21.15.1:6790/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f02fc001580 con 0x7f03040b4e70 2024-03-25T18:27:16.316 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:27:16.313+0000 7f02f9ffb640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:27:16.316 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:27:16.313+0000 7f02fb7fe640 4 set_mon_vals no callback set 2024-03-25T18:27:16.693 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:27:16.692+0000 7f02f9ffb640 1 -- 172.21.15.1:0/3675566436 <== mon.2 v1:172.21.15.1:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1586965025 0 0) 0x7f02fc005060 con 0x7f03040b4e70 2024-03-25T18:27:16.693 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:27:16.692+0000 7f02f9ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:27:16.694 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:27:16.692+0000 7f02f9ffb640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-25T18:27:16.694 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:27:16.692+0000 7f02f9ffb640 10 monclient: dump: 2024-03-25T18:27:16.694 INFO:teuthology.misc.health.smithi001.stderr:epoch 1 2024-03-25T18:27:16.694 INFO:teuthology.misc.health.smithi001.stderr:fsid c8ea72cf-9e99-4383-9b52-6c23d365fdfc 2024-03-25T18:27:16.694 INFO:teuthology.misc.health.smithi001.stderr:last_changed 2024-03-25T18:25:50.510519+0000 2024-03-25T18:27:16.694 INFO:teuthology.misc.health.smithi001.stderr:created 2024-03-25T18:25:50.510519+0000 2024-03-25T18:27:16.694 INFO:teuthology.misc.health.smithi001.stderr:min_mon_release 19 (squid) 2024-03-25T18:27:16.694 INFO:teuthology.misc.health.smithi001.stderr:election_strategy: 3 2024-03-25T18:27:16.694 INFO:teuthology.misc.health.smithi001.stderr:0: v1:172.21.15.1:6789/0 mon.a 2024-03-25T18:27:16.694 INFO:teuthology.misc.health.smithi001.stderr:1: v1:172.21.15.115:6789/0 mon.b 2024-03-25T18:27:16.694 INFO:teuthology.misc.health.smithi001.stderr:2: v1:172.21.15.1:6790/0 mon.c 2024-03-25T18:27:16.694 INFO:teuthology.misc.health.smithi001.stderr: 2024-03-25T18:27:16.695 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:27:16.693+0000 7f02f9ffb640 1 -- 172.21.15.1:0/3675566436 <== mon.2 v1:172.21.15.1:6790/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 21787975 0 0) 0x7f02fc01b030 con 0x7f03040b4e70 2024-03-25T18:27:16.695 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:27:16.694+0000 7f03096ce640 10 monclient: get_auth_request con 0x7f02e806e090 auth_method 0 2024-03-25T18:27:16.695 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:27:16.695+0000 7f02f9ffb640 10 client.4226.objecter ms_handle_connect 0x7f02e806e090 2024-03-25T18:27:16.725 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:27:16.724+0000 7f02f9ffb640 1 -- 172.21.15.1:0/3675566436 <== mon.2 v1:172.21.15.1:6790/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 996254230 0 0) 0x7f02fc054c40 con 0x7f03040b4e70 2024-03-25T18:27:16.725 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:27:16.724+0000 7f02f9ffb640 10 client.4226.objecter ms_dispatch 0x7f0304000b90 osd_map(10..10 src has 1..10) v4 2024-03-25T18:27:16.725 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:27:16.724+0000 7f02f9ffb640 3 client.4226.objecter handle_osd_map got epochs [10,10] > 0 2024-03-25T18:27:16.725 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:27:16.724+0000 7f02f9ffb640 3 client.4226.objecter handle_osd_map decoding full epoch 10 2024-03-25T18:27:16.725 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:27:16.724+0000 7f02f9ffb640 20 client.4226.objecter dump_active .. 0 homeless 2024-03-25T18:27:16.725 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:27:16.724+0000 7f02f9ffb640 1 -- 172.21.15.1:0/3675566436 <== mon.2 v1:172.21.15.1:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f02fc050580 con 0x7f03040b4e70 2024-03-25T18:27:16.725 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:27:16.724+0000 7f02f9ffb640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:27:16.725 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:27:16.724+0000 7f02f9ffb640 10 monclient: _finish_command 1 = system:0 2024-03-25T18:27:17.010 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:27:17.009+0000 7f02ef7fe640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:27:17.010 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:27:17.009+0000 7f02ef7fe640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:27:17.011 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:27:17.009+0000 7f02ef7fe640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.1:6790/0 2024-03-25T18:27:17.011 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:27:17.009+0000 7f02ef7fe640 1 -- 172.21.15.1:0/3675566436 --> v1:172.21.15.1:6790/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f02e4005330 con 0x7f03040b4e70 2024-03-25T18:27:17.138 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:27:17.138+0000 7f02f9ffb640 1 -- 172.21.15.1:0/3675566436 <== mon.2 v1:172.21.15.1:6790/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 4092213348) 0x7f02fc052680 con 0x7f03040b4e70 2024-03-25T18:27:17.139 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:27:17.138+0000 7f02f9ffb640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:27:17.139 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:27:17.138+0000 7f02f9ffb640 10 monclient: _finish_command 2 = system:0 2024-03-25T18:27:17.139 INFO:teuthology.misc.health.smithi001.stdout: 2024-03-25T18:27:17.139 INFO:teuthology.misc.health.smithi001.stdout:{"epoch":10,"fsid":"c8ea72cf-9e99-4383-9b52-6c23d365fdfc","created":"2024-03-25T18:26:38.856638+0000","modified":"2024-03-25T18:26:44.224208+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T18:26:44.224208+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"b9f34922-3a84-49be-8426-807c1eceaefc","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"42bf306f-55e3-4634-9874-8b4b73ac6644","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"292cb23a-18a9-4207-a7cc-4d5727feaec0","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"45e0e9ab-898d-4673-9590-b4117bc5e84a","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"831a5a09-73ab-4a61-9872-088e76bdbc66","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"88cd903d-3b55-4840-9396-6e3cfe005282","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"89db7355-bc9d-46f4-8ab0-e8d9e744e026","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"cac47bee-0914-4f08-896f-348838882866","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T18:27:17.142 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:27:17.140+0000 7f02ef7fe640 10 librados: watch_flush enter 2024-03-25T18:27:17.142 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:27:17.140+0000 7f02ef7fe640 10 librados: watch_flush exit 2024-03-25T18:27:17.142 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:27:17.140+0000 7f02ef7fe640 20 client.4226.objecter shutdown clearing up homeless session... 2024-03-25T18:27:17.142 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:27:17.140+0000 7f02ef7fe640 10 client.4226.objecter successfully canceled tick 2024-03-25T18:27:17.142 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:27:17.141+0000 7f02ef7fe640 1 -- 172.21.15.1:0/3675566436 >> v1:172.21.15.115:6800/4237379176 conn(0x7f02e806e090 legacy=0x7f02e8070550 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:27:17.142 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:27:17.141+0000 7f02ef7fe640 10 monclient: shutdown 2024-03-25T18:27:17.142 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:27:17.141+0000 7f02ef7fe640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:27:17.142 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:27:17.141+0000 7f02ef7fe640 1 -- 172.21.15.1:0/3675566436 >> v1:172.21.15.1:6790/0 conn(0x7f03040b4e70 legacy=0x7f03041e2090 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:27:17.143 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:27:17.141+0000 7f02ef7fe640 1 -- 172.21.15.1:0/3675566436 shutdown_connections 2024-03-25T18:27:17.143 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:27:17.141+0000 7f02ef7fe640 1 -- 172.21.15.1:0/3675566436 >> 172.21.15.1:0/3675566436 conn(0x7f03040617a0 msgr2=0x7f03040b59e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:27:17.143 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:27:17.141+0000 7f02ef7fe640 1 -- 172.21.15.1:0/3675566436 shutdown_connections 2024-03-25T18:27:17.143 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:27:17.141+0000 7f02ef7fe640 1 -- 172.21.15.1:0/3675566436 wait complete. 2024-03-25T18:27:17.143 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:27:17.141+0000 7f02ef7fe640 1 librados: shutdown 2024-03-25T18:27:17.156 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T18:27:19.439 INFO:tasks.ceph.osd.5.smithi115.stderr:2024-03-25T18:27:19.434+0000 5dc3480 -1 osd.5 0 log_to_monitors true 2024-03-25T18:27:19.766 INFO:tasks.ceph.osd.7.smithi115.stderr:2024-03-25T18:27:19.763+0000 5dc3480 -1 osd.7 0 log_to_monitors true 2024-03-25T18:27:20.194 INFO:tasks.ceph.osd.4.smithi115.stderr:2024-03-25T18:27:20.191+0000 5dc3480 -1 osd.4 0 log_to_monitors true 2024-03-25T18:27:20.298 INFO:tasks.ceph.osd.3.smithi001.stderr:2024-03-25T18:27:20.295+0000 5dc3480 -1 osd.3 0 log_to_monitors true 2024-03-25T18:27:20.406 INFO:tasks.ceph.osd.6.smithi115.stderr:2024-03-25T18:27:20.402+0000 5dc3480 -1 osd.6 0 log_to_monitors true 2024-03-25T18:27:21.367 INFO:tasks.ceph.osd.1.smithi001.stderr:2024-03-25T18:27:21.363+0000 5dc3480 -1 osd.1 0 log_to_monitors true 2024-03-25T18:27:21.545 INFO:tasks.ceph.osd.0.smithi001.stderr:2024-03-25T18:27:21.541+0000 5dc3480 -1 osd.0 0 log_to_monitors true 2024-03-25T18:27:22.577 INFO:tasks.ceph.osd.2.smithi001.stderr:2024-03-25T18:27:22.573+0000 5dc3480 -1 osd.2 0 log_to_monitors true 2024-03-25T18:27:23.963 DEBUG:teuthology.orchestra.run.smithi001:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T18:27:24.081 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:27:24.080+0000 7fa601c9d640 10 monclient: get_monmap_and_config 2024-03-25T18:27:24.081 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:27:24.080+0000 7fa601c9d640 10 monclient: build_initial_monmap 2024-03-25T18:27:24.081 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:27:24.080+0000 7fa601c9d640 10 monclient: monmap: 2024-03-25T18:27:24.081 INFO:teuthology.misc.health.smithi001.stderr:epoch 0 2024-03-25T18:27:24.081 INFO:teuthology.misc.health.smithi001.stderr:fsid c8ea72cf-9e99-4383-9b52-6c23d365fdfc 2024-03-25T18:27:24.081 INFO:teuthology.misc.health.smithi001.stderr:last_changed 2024-03-25T18:27:24.081095+0000 2024-03-25T18:27:24.081 INFO:teuthology.misc.health.smithi001.stderr:created 2024-03-25T18:27:24.081095+0000 2024-03-25T18:27:24.081 INFO:teuthology.misc.health.smithi001.stderr:min_mon_release 0 (unknown) 2024-03-25T18:27:24.082 INFO:teuthology.misc.health.smithi001.stderr:election_strategy: 3 2024-03-25T18:27:24.082 INFO:teuthology.misc.health.smithi001.stderr:0: v1:172.21.15.1:6789/0 mon.noname-a 2024-03-25T18:27:24.082 INFO:teuthology.misc.health.smithi001.stderr:1: v1:172.21.15.115:6789/0 mon.noname-c 2024-03-25T18:27:24.082 INFO:teuthology.misc.health.smithi001.stderr:2: v1:172.21.15.1:6790/0 mon.noname-b 2024-03-25T18:27:24.082 INFO:teuthology.misc.health.smithi001.stderr: 2024-03-25T18:27:24.083 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:27:24.081+0000 7fa601c9d640 1 Processor -- start 2024-03-25T18:27:24.083 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:27:24.081+0000 7fa601c9d640 1 -- start start 2024-03-25T18:27:24.083 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:27:24.081+0000 7fa601c9d640 10 monclient: init 2024-03-25T18:27:24.084 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:27:24.082+0000 7fa601c9d640 10 monclient: _reopen_session rank -1 2024-03-25T18:27:24.084 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:27:24.082+0000 7fa601c9d640 10 monclient: _add_conns ranks=[1,0,2] 2024-03-25T18:27:24.084 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:27:24.082+0000 7fa601c9d640 10 monclient(hunting): picked mon.noname-c con 0x7fa5fc0b4ff0 addr v1:172.21.15.115:6789/0 2024-03-25T18:27:24.084 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:27:24.082+0000 7fa601c9d640 10 monclient(hunting): picked mon.noname-a con 0x7fa5fc0b4670 addr v1:172.21.15.1:6789/0 2024-03-25T18:27:24.084 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:27:24.082+0000 7fa601c9d640 10 monclient(hunting): picked mon.noname-b con 0x7fa5fc13c480 addr v1:172.21.15.1:6790/0 2024-03-25T18:27:24.084 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:27:24.082+0000 7fa601c9d640 1 -- --> v1:172.21.15.1:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa5fc0ab0e0 con 0x7fa5fc0b4670 2024-03-25T18:27:24.084 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:27:24.082+0000 7fa601c9d640 1 -- --> v1:172.21.15.115:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa5fc0ab3c0 con 0x7fa5fc0b4ff0 2024-03-25T18:27:24.084 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:27:24.082+0000 7fa601c9d640 1 -- --> v1:172.21.15.1:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa5fc0a9f30 con 0x7fa5fc13c480 2024-03-25T18:27:24.084 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:27:24.082+0000 7fa601c9d640 10 monclient(hunting): _renew_subs 2024-03-25T18:27:24.084 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:27:24.082+0000 7fa601c9d640 10 monclient(hunting): authenticate will time out at 7524.552209s 2024-03-25T18:27:24.088 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:27:24.088+0000 7fa5fb7fe640 1 --1- >> v1:172.21.15.115:6789/0 conn(0x7fa5fc0b4ff0 0x7fa5fc13bd70 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.115:6789/0 says I am v1:172.21.15.1:50460/0 (socket says 172.21.15.1:50460) 2024-03-25T18:27:24.089 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:27:24.088+0000 7fa5fb7fe640 1 -- 172.21.15.1:0/80947010 learned_addr learned my addr 172.21.15.1:0/80947010 (peer_addr_for_me v1:172.21.15.1:0/0) 2024-03-25T18:27:24.102 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:27:24.101+0000 7fa5fa7fc640 1 -- 172.21.15.1:0/80947010 <== mon.1 v1:172.21.15.115:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2318860198 0 0) 0x7fa5fc0ab3c0 con 0x7fa5fc0b4ff0 2024-03-25T18:27:24.102 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:27:24.101+0000 7fa5fa7fc640 10 monclient(hunting): _negotiate 2024-03-25T18:27:24.102 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:27:24.101+0000 7fa5fa7fc640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:27:24.102 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:27:24.101+0000 7fa5fa7fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:27:24.102 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:27:24.101+0000 7fa5fa7fc640 10 monclient(hunting): my global_id is 4246 2024-03-25T18:27:24.102 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:27:24.101+0000 7fa5fa7fc640 1 -- 172.21.15.1:0/80947010 --> v1:172.21.15.115:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fa5e4003520 con 0x7fa5fc0b4ff0 2024-03-25T18:27:24.113 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:27:24.112+0000 7fa5fa7fc640 1 -- 172.21.15.1:0/80947010 <== mon.2 v1:172.21.15.1:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3343298407 0 0) 0x7fa5fc0a9f30 con 0x7fa5fc13c480 2024-03-25T18:27:24.113 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:27:24.112+0000 7fa5fa7fc640 10 monclient(hunting): _negotiate 2024-03-25T18:27:24.114 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:27:24.112+0000 7fa5fa7fc640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:27:24.114 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:27:24.112+0000 7fa5fa7fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:27:24.114 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:27:24.112+0000 7fa5fa7fc640 10 monclient(hunting): my global_id is 4253 2024-03-25T18:27:24.114 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:27:24.112+0000 7fa5fa7fc640 1 -- 172.21.15.1:0/80947010 --> v1:172.21.15.1:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fa5fc0ab3c0 con 0x7fa5fc13c480 2024-03-25T18:27:24.118 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:27:24.117+0000 7fa5fa7fc640 1 -- 172.21.15.1:0/80947010 <== mon.1 v1:172.21.15.115:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 865779349 0 0) 0x7fa5e4003520 con 0x7fa5fc0b4ff0 2024-03-25T18:27:24.118 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:27:24.117+0000 7fa5fa7fc640 1 -- 172.21.15.1:0/80947010 >> v1:172.21.15.1:6790/0 conn(0x7fa5fc13c480 legacy=0x7fa5fc14dc00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:27:24.118 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:27:24.117+0000 7fa5fa7fc640 1 -- 172.21.15.1:0/80947010 >> v1:172.21.15.1:6789/0 conn(0x7fa5fc0b4670 legacy=0x7fa5fc1425c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:27:24.118 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:27:24.117+0000 7fa5fa7fc640 10 monclient: _finish_hunting 0 2024-03-25T18:27:24.118 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:27:24.117+0000 7fa5fa7fc640 1 monclient: found mon.noname-c 2024-03-25T18:27:24.118 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:27:24.117+0000 7fa5fa7fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:27:24.118 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:27:24.117+0000 7fa5fa7fc640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.115:6789/0 2024-03-25T18:27:24.118 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:27:24.117+0000 7fa5fa7fc640 1 -- 172.21.15.1:0/80947010 --> v1:172.21.15.115:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa5fc0a9b70 con 0x7fa5fc0b4ff0 2024-03-25T18:27:24.118 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:27:24.117+0000 7fa5fa7fc640 10 monclient: _finish_auth 0 2024-03-25T18:27:24.119 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:27:24.117+0000 7fa5fa7fc640 10 monclient: _check_auth_tickets 2024-03-25T18:27:24.119 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:27:24.117+0000 7fa5fa7fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:27:24.119 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:27:24.117+0000 7fa5fa7fc640 1 -- 172.21.15.1:0/80947010 <== mon.1 v1:172.21.15.115:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1586965025 0 0) 0x7fa5ec001690 con 0x7fa5fc0b4ff0 2024-03-25T18:27:24.119 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:27:24.117+0000 7fa5fa7fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:27:24.119 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:27:24.117+0000 7fa5fa7fc640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-03-25T18:27:24.119 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:27:24.117+0000 7fa5fa7fc640 10 monclient: dump: 2024-03-25T18:27:24.119 INFO:teuthology.misc.health.smithi001.stderr:epoch 1 2024-03-25T18:27:24.119 INFO:teuthology.misc.health.smithi001.stderr:fsid c8ea72cf-9e99-4383-9b52-6c23d365fdfc 2024-03-25T18:27:24.119 INFO:teuthology.misc.health.smithi001.stderr:last_changed 2024-03-25T18:25:50.510519+0000 2024-03-25T18:27:24.119 INFO:teuthology.misc.health.smithi001.stderr:created 2024-03-25T18:25:50.510519+0000 2024-03-25T18:27:24.119 INFO:teuthology.misc.health.smithi001.stderr:min_mon_release 19 (squid) 2024-03-25T18:27:24.119 INFO:teuthology.misc.health.smithi001.stderr:election_strategy: 3 2024-03-25T18:27:24.119 INFO:teuthology.misc.health.smithi001.stderr:0: v1:172.21.15.1:6789/0 mon.a 2024-03-25T18:27:24.119 INFO:teuthology.misc.health.smithi001.stderr:1: v1:172.21.15.115:6789/0 mon.b 2024-03-25T18:27:24.120 INFO:teuthology.misc.health.smithi001.stderr:2: v1:172.21.15.1:6790/0 mon.c 2024-03-25T18:27:24.120 INFO:teuthology.misc.health.smithi001.stderr: 2024-03-25T18:27:24.120 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:27:24.117+0000 7fa601c9d640 5 monclient: authenticate success, global_id 4246 2024-03-25T18:27:24.120 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:27:24.117+0000 7fa601c9d640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T18:27:24.132 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:27:24.131+0000 7fa5fa7fc640 1 -- 172.21.15.1:0/80947010 <== mon.1 v1:172.21.15.115:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fa5ec0017f0 con 0x7fa5fc0b4ff0 2024-03-25T18:27:24.132 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:27:24.131+0000 7fa5fa7fc640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:27:24.132 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:27:24.131+0000 7fa5fa7fc640 1 -- 172.21.15.1:0/80947010 <== mon.1 v1:172.21.15.115:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1586965025 0 0) 0x7fa5ec001c00 con 0x7fa5fc0b4ff0 2024-03-25T18:27:24.132 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:27:24.131+0000 7fa601c9d640 10 monclient: get_monmap_and_config success 2024-03-25T18:27:24.132 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:27:24.131+0000 7fa601c9d640 4 set_mon_vals no callback set 2024-03-25T18:27:24.132 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:27:24.131+0000 7fa5fa7fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:27:24.132 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:27:24.131+0000 7fa5fa7fc640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-25T18:27:24.133 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:27:24.131+0000 7fa5fa7fc640 10 monclient: dump: 2024-03-25T18:27:24.133 INFO:teuthology.misc.health.smithi001.stderr:epoch 1 2024-03-25T18:27:24.133 INFO:teuthology.misc.health.smithi001.stderr:fsid c8ea72cf-9e99-4383-9b52-6c23d365fdfc 2024-03-25T18:27:24.133 INFO:teuthology.misc.health.smithi001.stderr:last_changed 2024-03-25T18:25:50.510519+0000 2024-03-25T18:27:24.133 INFO:teuthology.misc.health.smithi001.stderr:created 2024-03-25T18:25:50.510519+0000 2024-03-25T18:27:24.133 INFO:teuthology.misc.health.smithi001.stderr:min_mon_release 19 (squid) 2024-03-25T18:27:24.133 INFO:teuthology.misc.health.smithi001.stderr:election_strategy: 3 2024-03-25T18:27:24.133 INFO:teuthology.misc.health.smithi001.stderr:0: v1:172.21.15.1:6789/0 mon.a 2024-03-25T18:27:24.133 INFO:teuthology.misc.health.smithi001.stderr:1: v1:172.21.15.115:6789/0 mon.b 2024-03-25T18:27:24.133 INFO:teuthology.misc.health.smithi001.stderr:2: v1:172.21.15.1:6790/0 mon.c 2024-03-25T18:27:24.133 INFO:teuthology.misc.health.smithi001.stderr: 2024-03-25T18:27:24.135 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:27:24.132+0000 7fa601c9d640 10 monclient: shutdown 2024-03-25T18:27:24.135 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:27:24.132+0000 7fa601c9d640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:27:24.135 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:27:24.132+0000 7fa601c9d640 1 -- 172.21.15.1:0/80947010 >> v1:172.21.15.115:6789/0 conn(0x7fa5fc0b4ff0 legacy=0x7fa5fc13bd70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:27:24.135 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:27:24.132+0000 7fa601c9d640 1 -- 172.21.15.1:0/80947010 shutdown_connections 2024-03-25T18:27:24.135 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:27:24.132+0000 7fa601c9d640 1 -- 172.21.15.1:0/80947010 >> 172.21.15.1:0/80947010 conn(0x7fa5fc0617a0 msgr2=0x7fa5fc061bb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:27:24.135 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:27:24.132+0000 7fa601c9d640 1 -- 172.21.15.1:0/80947010 shutdown_connections 2024-03-25T18:27:24.136 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:27:24.132+0000 7fa601c9d640 1 -- 172.21.15.1:0/80947010 wait complete. 2024-03-25T18:27:24.136 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:27:24.133+0000 7fa601c9d640 10 monclient: build_initial_monmap 2024-03-25T18:27:24.136 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:27:24.133+0000 7fa601c9d640 10 monclient: monmap: 2024-03-25T18:27:24.136 INFO:teuthology.misc.health.smithi001.stderr:epoch 0 2024-03-25T18:27:24.136 INFO:teuthology.misc.health.smithi001.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T18:27:24.136 INFO:teuthology.misc.health.smithi001.stderr:last_changed 0.000000 2024-03-25T18:27:24.136 INFO:teuthology.misc.health.smithi001.stderr:created 0.000000 2024-03-25T18:27:24.136 INFO:teuthology.misc.health.smithi001.stderr:min_mon_release 0 (unknown) 2024-03-25T18:27:24.136 INFO:teuthology.misc.health.smithi001.stderr:election_strategy: 1 2024-03-25T18:27:24.136 INFO:teuthology.misc.health.smithi001.stderr:0: v1:172.21.15.1:6789/0 mon.noname-a 2024-03-25T18:27:24.136 INFO:teuthology.misc.health.smithi001.stderr:1: v1:172.21.15.115:6789/0 mon.noname-b 2024-03-25T18:27:24.137 INFO:teuthology.misc.health.smithi001.stderr:2: v1:172.21.15.1:6790/0 mon.noname-c 2024-03-25T18:27:24.137 INFO:teuthology.misc.health.smithi001.stderr: 2024-03-25T18:27:24.137 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:27:24.133+0000 7fa601c9d640 1 librados: starting msgr at 2024-03-25T18:27:24.137 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:27:24.133+0000 7fa601c9d640 1 librados: starting objecter 2024-03-25T18:27:24.137 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:27:24.133+0000 7fa601c9d640 1 Processor -- start 2024-03-25T18:27:24.137 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:27:24.133+0000 7fa601c9d640 1 -- start start 2024-03-25T18:27:24.137 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:27:24.133+0000 7fa601c9d640 1 librados: setting wanted keys 2024-03-25T18:27:24.137 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:27:24.133+0000 7fa601c9d640 1 librados: calling monclient init 2024-03-25T18:27:24.137 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:27:24.133+0000 7fa601c9d640 10 monclient: init 2024-03-25T18:27:24.137 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:27:24.134+0000 7fa601c9d640 10 monclient: _reopen_session rank -1 2024-03-25T18:27:24.137 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:27:24.134+0000 7fa601c9d640 10 monclient: _add_conns ranks=[0,1,2] 2024-03-25T18:27:24.137 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:27:24.134+0000 7fa601c9d640 10 monclient(hunting): picked mon.noname-a con 0x7fa5fc0b4670 addr v1:172.21.15.1:6789/0 2024-03-25T18:27:24.137 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:27:24.134+0000 7fa601c9d640 10 monclient(hunting): picked mon.noname-b con 0x7fa5fc0b4ff0 addr v1:172.21.15.115:6789/0 2024-03-25T18:27:24.137 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:27:24.134+0000 7fa601c9d640 10 monclient(hunting): picked mon.noname-c con 0x7fa5fc13c480 addr v1:172.21.15.1:6790/0 2024-03-25T18:27:24.138 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:27:24.134+0000 7fa601c9d640 1 -- --> v1:172.21.15.1:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa5fc0abc10 con 0x7fa5fc0b4670 2024-03-25T18:27:24.138 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:27:24.134+0000 7fa601c9d640 1 -- --> v1:172.21.15.115:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa5fc0ac380 con 0x7fa5fc0b4ff0 2024-03-25T18:27:24.138 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:27:24.134+0000 7fa601c9d640 1 -- --> v1:172.21.15.1:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa5fc0ac660 con 0x7fa5fc13c480 2024-03-25T18:27:24.138 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:27:24.134+0000 7fa601c9d640 10 monclient(hunting): _renew_subs 2024-03-25T18:27:24.138 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:27:24.134+0000 7fa601c9d640 10 monclient(hunting): authenticate will time out at 7524.604272s 2024-03-25T18:27:24.142 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:27:24.141+0000 7fa5fbfff640 1 --1- >> v1:172.21.15.1:6790/0 conn(0x7fa5fc13c480 0x7fa5fc1e3a20 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.1:6790/0 says I am v1:172.21.15.1:59760/0 (socket says 172.21.15.1:59760) 2024-03-25T18:27:24.142 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:27:24.141+0000 7fa5fbfff640 1 -- 172.21.15.1:0/3803895076 learned_addr learned my addr 172.21.15.1:0/3803895076 (peer_addr_for_me v1:172.21.15.1:0/0) 2024-03-25T18:27:24.146 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:27:24.145+0000 7fa5e3fff640 10 client.?.objecter ms_handle_connect 0x7fa5fc13c480 2024-03-25T18:27:24.146 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:27:24.145+0000 7fa5e3fff640 10 client.?.objecter resend_mon_ops 2024-03-25T18:27:24.157 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:27:24.156+0000 7fa5e3fff640 10 client.?.objecter ms_handle_connect 0x7fa5fc0b4670 2024-03-25T18:27:24.157 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:27:24.156+0000 7fa5e3fff640 10 client.?.objecter resend_mon_ops 2024-03-25T18:27:24.158 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:27:24.158+0000 7fa5e3fff640 1 -- 172.21.15.1:0/3803895076 <== mon.2 v1:172.21.15.1:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1514309650 0 0) 0x7fa5fc0ac660 con 0x7fa5fc13c480 2024-03-25T18:27:24.158 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:27:24.158+0000 7fa5e3fff640 10 monclient(hunting): _negotiate 2024-03-25T18:27:24.159 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:27:24.158+0000 7fa5e3fff640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:27:24.159 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:27:24.158+0000 7fa5e3fff640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:27:24.159 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:27:24.158+0000 7fa5e3fff640 10 monclient(hunting): my global_id is 4256 2024-03-25T18:27:24.159 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:27:24.158+0000 7fa5e3fff640 1 -- 172.21.15.1:0/3803895076 --> v1:172.21.15.1:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fa5e8004eb0 con 0x7fa5fc13c480 2024-03-25T18:27:24.164 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:27:24.164+0000 7fa5e3fff640 1 -- 172.21.15.1:0/3803895076 <== mon.0 v1:172.21.15.1:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 225452055 0 0) 0x7fa5fc0abc10 con 0x7fa5fc0b4670 2024-03-25T18:27:24.164 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:27:24.164+0000 7fa5e3fff640 10 monclient(hunting): _negotiate 2024-03-25T18:27:24.165 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:27:24.164+0000 7fa5e3fff640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:27:24.165 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:27:24.164+0000 7fa5e3fff640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:27:24.165 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:27:24.164+0000 7fa5e3fff640 10 monclient(hunting): my global_id is 4251 2024-03-25T18:27:24.165 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:27:24.164+0000 7fa5e3fff640 1 -- 172.21.15.1:0/3803895076 --> v1:172.21.15.1:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fa5fc0ac660 con 0x7fa5fc0b4670 2024-03-25T18:27:24.166 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:27:24.165+0000 7fa5e3fff640 10 client.?.objecter ms_handle_connect 0x7fa5fc0b4ff0 2024-03-25T18:27:24.166 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:27:24.165+0000 7fa5e3fff640 10 client.?.objecter resend_mon_ops 2024-03-25T18:27:24.170 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:27:24.169+0000 7fa5e3fff640 1 -- 172.21.15.1:0/3803895076 <== mon.1 v1:172.21.15.115:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4226505876 0 0) 0x7fa5fc0ac380 con 0x7fa5fc0b4ff0 2024-03-25T18:27:24.170 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:27:24.169+0000 7fa5e3fff640 10 monclient(hunting): _negotiate 2024-03-25T18:27:24.170 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:27:24.169+0000 7fa5e3fff640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:27:24.170 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:27:24.169+0000 7fa5e3fff640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:27:24.170 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:27:24.169+0000 7fa5e3fff640 10 monclient(hunting): my global_id is 4249 2024-03-25T18:27:24.170 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:27:24.169+0000 7fa5e3fff640 1 -- 172.21.15.1:0/3803895076 --> v1:172.21.15.115:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fa5fc0abc10 con 0x7fa5fc0b4ff0 2024-03-25T18:27:24.171 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:27:24.170+0000 7fa5e3fff640 1 -- 172.21.15.1:0/3803895076 <== mon.2 v1:172.21.15.1:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1210336330 0 0) 0x7fa5e8004eb0 con 0x7fa5fc13c480 2024-03-25T18:27:24.171 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:27:24.170+0000 7fa5e3fff640 1 -- 172.21.15.1:0/3803895076 --> v1:172.21.15.1:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fa5fc0ac380 con 0x7fa5fc13c480 2024-03-25T18:27:24.171 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:27:24.170+0000 7fa5e3fff640 1 -- 172.21.15.1:0/3803895076 <== mon.2 v1:172.21.15.1:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1586965025 0 0) 0x7fa5f4001c90 con 0x7fa5fc13c480 2024-03-25T18:27:24.171 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:27:24.170+0000 7fa5e3fff640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:27:24.171 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:27:24.170+0000 7fa5e3fff640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-03-25T18:27:24.171 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:27:24.170+0000 7fa5e3fff640 10 monclient(hunting): dump: 2024-03-25T18:27:24.171 INFO:teuthology.misc.health.smithi001.stderr:epoch 1 2024-03-25T18:27:24.172 INFO:teuthology.misc.health.smithi001.stderr:fsid c8ea72cf-9e99-4383-9b52-6c23d365fdfc 2024-03-25T18:27:24.172 INFO:teuthology.misc.health.smithi001.stderr:last_changed 2024-03-25T18:25:50.510519+0000 2024-03-25T18:27:24.172 INFO:teuthology.misc.health.smithi001.stderr:created 2024-03-25T18:25:50.510519+0000 2024-03-25T18:27:24.172 INFO:teuthology.misc.health.smithi001.stderr:min_mon_release 19 (squid) 2024-03-25T18:27:24.172 INFO:teuthology.misc.health.smithi001.stderr:election_strategy: 3 2024-03-25T18:27:24.172 INFO:teuthology.misc.health.smithi001.stderr:0: v1:172.21.15.1:6789/0 mon.a 2024-03-25T18:27:24.172 INFO:teuthology.misc.health.smithi001.stderr:1: v1:172.21.15.115:6789/0 mon.b 2024-03-25T18:27:24.172 INFO:teuthology.misc.health.smithi001.stderr:2: v1:172.21.15.1:6790/0 mon.c 2024-03-25T18:27:24.172 INFO:teuthology.misc.health.smithi001.stderr: 2024-03-25T18:27:24.172 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:27:24.170+0000 7fa5e3fff640 10 monclient(hunting): _finish_auth 0 2024-03-25T18:27:24.175 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:27:24.175+0000 7fa5e3fff640 1 -- 172.21.15.1:0/3803895076 <== mon.0 v1:172.21.15.1:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3175691591 0 0) 0x7fa5fc0ac660 con 0x7fa5fc0b4670 2024-03-25T18:27:24.175 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:27:24.175+0000 7fa5e3fff640 1 -- 172.21.15.1:0/3803895076 --> v1:172.21.15.1:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fa5e8004eb0 con 0x7fa5fc0b4670 2024-03-25T18:27:24.175 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:27:24.175+0000 7fa5e3fff640 1 -- 172.21.15.1:0/3803895076 <== mon.0 v1:172.21.15.1:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1586965025 0 0) 0x7fa5ec001b00 con 0x7fa5fc0b4670 2024-03-25T18:27:24.176 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:27:24.175+0000 7fa5e3fff640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:27:24.176 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:27:24.175+0000 7fa5e3fff640 10 monclient(hunting): got monmap 1 from mon.a (according to old e1) 2024-03-25T18:27:24.176 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:27:24.175+0000 7fa5e3fff640 10 monclient(hunting): dump: 2024-03-25T18:27:24.176 INFO:teuthology.misc.health.smithi001.stderr:epoch 1 2024-03-25T18:27:24.176 INFO:teuthology.misc.health.smithi001.stderr:fsid c8ea72cf-9e99-4383-9b52-6c23d365fdfc 2024-03-25T18:27:24.176 INFO:teuthology.misc.health.smithi001.stderr:last_changed 2024-03-25T18:25:50.510519+0000 2024-03-25T18:27:24.176 INFO:teuthology.misc.health.smithi001.stderr:created 2024-03-25T18:25:50.510519+0000 2024-03-25T18:27:24.176 INFO:teuthology.misc.health.smithi001.stderr:min_mon_release 19 (squid) 2024-03-25T18:27:24.176 INFO:teuthology.misc.health.smithi001.stderr:election_strategy: 3 2024-03-25T18:27:24.176 INFO:teuthology.misc.health.smithi001.stderr:0: v1:172.21.15.1:6789/0 mon.a 2024-03-25T18:27:24.176 INFO:teuthology.misc.health.smithi001.stderr:1: v1:172.21.15.115:6789/0 mon.b 2024-03-25T18:27:24.176 INFO:teuthology.misc.health.smithi001.stderr:2: v1:172.21.15.1:6790/0 mon.c 2024-03-25T18:27:24.176 INFO:teuthology.misc.health.smithi001.stderr: 2024-03-25T18:27:24.182 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:27:24.181+0000 7fa5e3fff640 1 -- 172.21.15.1:0/3803895076 <== mon.1 v1:172.21.15.115:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3584606202 0 0) 0x7fa5fc0abc10 con 0x7fa5fc0b4ff0 2024-03-25T18:27:24.182 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:27:24.181+0000 7fa5e3fff640 1 -- 172.21.15.1:0/3803895076 --> v1:172.21.15.115:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fa5fc0ac660 con 0x7fa5fc0b4ff0 2024-03-25T18:27:24.182 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:27:24.181+0000 7fa5e3fff640 1 -- 172.21.15.1:0/3803895076 <== mon.1 v1:172.21.15.115:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1586965025 0 0) 0x7fa5f00017f0 con 0x7fa5fc0b4ff0 2024-03-25T18:27:24.182 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:27:24.181+0000 7fa5e3fff640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:27:24.183 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:27:24.181+0000 7fa5e3fff640 10 monclient(hunting): got monmap 1 from mon.b (according to old e1) 2024-03-25T18:27:24.183 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:27:24.181+0000 7fa5e3fff640 10 monclient(hunting): dump: 2024-03-25T18:27:24.183 INFO:teuthology.misc.health.smithi001.stderr:epoch 1 2024-03-25T18:27:24.183 INFO:teuthology.misc.health.smithi001.stderr:fsid c8ea72cf-9e99-4383-9b52-6c23d365fdfc 2024-03-25T18:27:24.183 INFO:teuthology.misc.health.smithi001.stderr:last_changed 2024-03-25T18:25:50.510519+0000 2024-03-25T18:27:24.183 INFO:teuthology.misc.health.smithi001.stderr:created 2024-03-25T18:25:50.510519+0000 2024-03-25T18:27:24.183 INFO:teuthology.misc.health.smithi001.stderr:min_mon_release 19 (squid) 2024-03-25T18:27:24.183 INFO:teuthology.misc.health.smithi001.stderr:election_strategy: 3 2024-03-25T18:27:24.183 INFO:teuthology.misc.health.smithi001.stderr:0: v1:172.21.15.1:6789/0 mon.a 2024-03-25T18:27:24.183 INFO:teuthology.misc.health.smithi001.stderr:1: v1:172.21.15.115:6789/0 mon.b 2024-03-25T18:27:24.183 INFO:teuthology.misc.health.smithi001.stderr:2: v1:172.21.15.1:6790/0 mon.c 2024-03-25T18:27:24.183 INFO:teuthology.misc.health.smithi001.stderr: 2024-03-25T18:27:24.183 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:27:24.181+0000 7fa5e3fff640 1 -- 172.21.15.1:0/3803895076 <== mon.0 v1:172.21.15.1:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 3048366901 0 0) 0x7fa5e8004eb0 con 0x7fa5fc0b4670 2024-03-25T18:27:24.183 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:27:24.181+0000 7fa5e3fff640 1 -- 172.21.15.1:0/3803895076 >> v1:172.21.15.1:6790/0 conn(0x7fa5fc13c480 legacy=0x7fa5fc1e3a20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:27:24.183 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:27:24.181+0000 7fa5e3fff640 1 -- 172.21.15.1:0/3803895076 >> v1:172.21.15.115:6789/0 conn(0x7fa5fc0b4ff0 legacy=0x7fa5fc1d81e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:27:24.184 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:27:24.181+0000 7fa5e3fff640 10 monclient: _finish_hunting 0 2024-03-25T18:27:24.184 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:27:24.181+0000 7fa5e3fff640 1 monclient: found mon.a 2024-03-25T18:27:24.184 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:27:24.181+0000 7fa5e3fff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:27:24.184 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:27:24.181+0000 7fa5e3fff640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.1:6789/0 2024-03-25T18:27:24.184 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:27:24.181+0000 7fa5e3fff640 1 -- 172.21.15.1:0/3803895076 --> v1:172.21.15.1:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa5fc0ac940 con 0x7fa5fc0b4670 2024-03-25T18:27:24.184 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:27:24.181+0000 7fa5e3fff640 10 monclient: _finish_auth 0 2024-03-25T18:27:24.184 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:27:24.181+0000 7fa5e3fff640 10 monclient: _check_auth_tickets 2024-03-25T18:27:24.184 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:27:24.181+0000 7fa5e3fff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:27:24.184 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:27:24.181+0000 7fa601c9d640 5 monclient: authenticate success, global_id 4251 2024-03-25T18:27:24.184 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:27:24.181+0000 7fa601c9d640 10 monclient: _renew_subs 2024-03-25T18:27:24.184 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:27:24.181+0000 7fa601c9d640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.1:6789/0 2024-03-25T18:27:24.184 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:27:24.181+0000 7fa601c9d640 1 -- 172.21.15.1:0/3803895076 --> v1:172.21.15.1:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa5fc0ab8e0 con 0x7fa5fc0b4670 2024-03-25T18:27:24.184 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:27:24.181+0000 7fa601c9d640 10 client.4251.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T18:27:24.185 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:27:24.181+0000 7fa601c9d640 10 monclient: _renew_subs 2024-03-25T18:27:24.185 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:27:24.181+0000 7fa601c9d640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.1:6789/0 2024-03-25T18:27:24.185 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:27:24.181+0000 7fa601c9d640 1 -- 172.21.15.1:0/3803895076 --> v1:172.21.15.1:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7fa5fc0ac130 con 0x7fa5fc0b4670 2024-03-25T18:27:24.185 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:27:24.181+0000 7fa601c9d640 1 librados: init done 2024-03-25T18:27:24.185 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:27:24.182+0000 7fa601c9d640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T18:27:24.185 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:27:24.182+0000 7fa601c9d640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:27:24.186 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:27:24.182+0000 7fa601c9d640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.1:6789/0 2024-03-25T18:27:24.186 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:27:24.182+0000 7fa601c9d640 1 -- 172.21.15.1:0/3803895076 --> v1:172.21.15.1:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa5fc0b3fd0 con 0x7fa5fc0b4670 2024-03-25T18:27:24.206 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:27:24.205+0000 7fa5e3fff640 1 -- 172.21.15.1:0/3803895076 <== mon.0 v1:172.21.15.1:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fa5ec004a90 con 0x7fa5fc0b4670 2024-03-25T18:27:24.206 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:27:24.205+0000 7fa5e3fff640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:27:24.206 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:27:24.205+0000 7fa5e3fff640 1 -- 172.21.15.1:0/3803895076 <== mon.0 v1:172.21.15.1:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1586965025 0 0) 0x7fa5ec004da0 con 0x7fa5fc0b4670 2024-03-25T18:27:24.206 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:27:24.205+0000 7fa5e3fff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:27:24.206 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:27:24.205+0000 7fa5e3fff640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-25T18:27:24.206 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:27:24.205+0000 7fa5e3fff640 10 monclient: dump: 2024-03-25T18:27:24.206 INFO:teuthology.misc.health.smithi001.stderr:epoch 1 2024-03-25T18:27:24.207 INFO:teuthology.misc.health.smithi001.stderr:fsid c8ea72cf-9e99-4383-9b52-6c23d365fdfc 2024-03-25T18:27:24.207 INFO:teuthology.misc.health.smithi001.stderr:last_changed 2024-03-25T18:25:50.510519+0000 2024-03-25T18:27:24.207 INFO:teuthology.misc.health.smithi001.stderr:created 2024-03-25T18:25:50.510519+0000 2024-03-25T18:27:24.207 INFO:teuthology.misc.health.smithi001.stderr:min_mon_release 19 (squid) 2024-03-25T18:27:24.207 INFO:teuthology.misc.health.smithi001.stderr:election_strategy: 3 2024-03-25T18:27:24.207 INFO:teuthology.misc.health.smithi001.stderr:0: v1:172.21.15.1:6789/0 mon.a 2024-03-25T18:27:24.207 INFO:teuthology.misc.health.smithi001.stderr:1: v1:172.21.15.115:6789/0 mon.b 2024-03-25T18:27:24.207 INFO:teuthology.misc.health.smithi001.stderr:2: v1:172.21.15.1:6790/0 mon.c 2024-03-25T18:27:24.207 INFO:teuthology.misc.health.smithi001.stderr: 2024-03-25T18:27:24.207 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:27:24.205+0000 7fa5f9ffb640 4 set_mon_vals no callback set 2024-03-25T18:27:24.208 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:27:24.206+0000 7fa5e3fff640 1 -- 172.21.15.1:0/3803895076 <== mon.0 v1:172.21.15.1:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 21787975 0 0) 0x7fa5ec01ad70 con 0x7fa5fc0b4670 2024-03-25T18:27:24.208 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:27:24.207+0000 7fa5faffd640 10 monclient: get_auth_request con 0x7fa5e806e110 auth_method 0 2024-03-25T18:27:24.208 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:27:24.207+0000 7fa5e3fff640 10 client.4251.objecter ms_handle_connect 0x7fa5e806e110 2024-03-25T18:27:24.215 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:27:24.211+0000 7fa5e3fff640 1 -- 172.21.15.1:0/3803895076 <== mon.0 v1:172.21.15.1:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 996254230 0 0) 0x7fa5ec0547f0 con 0x7fa5fc0b4670 2024-03-25T18:27:24.215 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:27:24.211+0000 7fa5e3fff640 10 client.4251.objecter ms_dispatch 0x7fa5fc000b90 osd_map(10..10 src has 1..10) v4 2024-03-25T18:27:24.215 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:27:24.211+0000 7fa5e3fff640 3 client.4251.objecter handle_osd_map got epochs [10,10] > 0 2024-03-25T18:27:24.215 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:27:24.211+0000 7fa5e3fff640 3 client.4251.objecter handle_osd_map decoding full epoch 10 2024-03-25T18:27:24.216 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:27:24.211+0000 7fa5e3fff640 20 client.4251.objecter dump_active .. 0 homeless 2024-03-25T18:27:24.524 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:27:24.523+0000 7fa5e3fff640 1 -- 172.21.15.1:0/3803895076 <== mon.0 v1:172.21.15.1:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7fa5ec050130 con 0x7fa5fc0b4670 2024-03-25T18:27:24.524 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:27:24.523+0000 7fa5e3fff640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:27:24.524 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:27:24.523+0000 7fa5e3fff640 10 monclient: _finish_command 1 = system:0 2024-03-25T18:27:24.774 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:27:24.773+0000 7fa601c9d640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:27:24.774 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:27:24.773+0000 7fa601c9d640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:27:24.774 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:27:24.773+0000 7fa601c9d640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.1:6789/0 2024-03-25T18:27:24.774 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:27:24.773+0000 7fa601c9d640 1 -- 172.21.15.1:0/3803895076 --> v1:172.21.15.1:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fa5fc0aba70 con 0x7fa5fc0b4670 2024-03-25T18:27:24.862 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:27:24.861+0000 7fa5e3fff640 1 -- 172.21.15.1:0/3803895076 <== mon.0 v1:172.21.15.1:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 4092213348) 0x7fa5fc0aba70 con 0x7fa5fc0b4670 2024-03-25T18:27:24.862 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:27:24.861+0000 7fa5e3fff640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:27:24.862 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:27:24.861+0000 7fa5e3fff640 10 monclient: _finish_command 2 = system:0 2024-03-25T18:27:24.863 INFO:teuthology.misc.health.smithi001.stdout: 2024-03-25T18:27:24.863 INFO:teuthology.misc.health.smithi001.stdout:{"epoch":10,"fsid":"c8ea72cf-9e99-4383-9b52-6c23d365fdfc","created":"2024-03-25T18:26:38.856638+0000","modified":"2024-03-25T18:26:44.224208+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T18:26:44.224208+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"b9f34922-3a84-49be-8426-807c1eceaefc","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"42bf306f-55e3-4634-9874-8b4b73ac6644","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"292cb23a-18a9-4207-a7cc-4d5727feaec0","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"45e0e9ab-898d-4673-9590-b4117bc5e84a","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"831a5a09-73ab-4a61-9872-088e76bdbc66","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"88cd903d-3b55-4840-9396-6e3cfe005282","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"89db7355-bc9d-46f4-8ab0-e8d9e744e026","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"cac47bee-0914-4f08-896f-348838882866","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T18:27:24.865 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:27:24.864+0000 7fa601c9d640 10 librados: watch_flush enter 2024-03-25T18:27:24.865 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:27:24.864+0000 7fa601c9d640 10 librados: watch_flush exit 2024-03-25T18:27:24.865 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:27:24.864+0000 7fa601c9d640 20 client.4251.objecter shutdown clearing up homeless session... 2024-03-25T18:27:24.865 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:27:24.864+0000 7fa601c9d640 10 client.4251.objecter successfully canceled tick 2024-03-25T18:27:24.865 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:27:24.864+0000 7fa601c9d640 1 -- 172.21.15.1:0/3803895076 >> v1:172.21.15.115:6800/4237379176 conn(0x7fa5e806e110 legacy=0x7fa5e80705d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:27:24.865 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:27:24.864+0000 7fa601c9d640 10 monclient: shutdown 2024-03-25T18:27:24.865 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:27:24.864+0000 7fa601c9d640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:27:24.865 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:27:24.864+0000 7fa601c9d640 1 -- 172.21.15.1:0/3803895076 >> v1:172.21.15.1:6789/0 conn(0x7fa5fc0b4670 legacy=0x7fa5fc137070 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:27:24.865 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:27:24.864+0000 7fa601c9d640 1 -- 172.21.15.1:0/3803895076 shutdown_connections 2024-03-25T18:27:24.865 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:27:24.864+0000 7fa601c9d640 1 -- 172.21.15.1:0/3803895076 >> 172.21.15.1:0/3803895076 conn(0x7fa5fc0617a0 msgr2=0x7fa5fc145420 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:27:24.866 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:27:24.864+0000 7fa601c9d640 1 -- 172.21.15.1:0/3803895076 shutdown_connections 2024-03-25T18:27:24.866 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:27:24.864+0000 7fa601c9d640 1 -- 172.21.15.1:0/3803895076 wait complete. 2024-03-25T18:27:24.866 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:27:24.864+0000 7fa601c9d640 1 librados: shutdown 2024-03-25T18:27:24.876 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T18:27:31.683 DEBUG:teuthology.orchestra.run.smithi001:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T18:27:31.801 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:27:31.800+0000 7fd60e94b640 10 monclient: get_monmap_and_config 2024-03-25T18:27:31.801 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:27:31.800+0000 7fd60e94b640 10 monclient: build_initial_monmap 2024-03-25T18:27:31.801 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:27:31.800+0000 7fd60e94b640 10 monclient: monmap: 2024-03-25T18:27:31.801 INFO:teuthology.misc.health.smithi001.stderr:epoch 0 2024-03-25T18:27:31.801 INFO:teuthology.misc.health.smithi001.stderr:fsid c8ea72cf-9e99-4383-9b52-6c23d365fdfc 2024-03-25T18:27:31.801 INFO:teuthology.misc.health.smithi001.stderr:last_changed 2024-03-25T18:27:31.801208+0000 2024-03-25T18:27:31.801 INFO:teuthology.misc.health.smithi001.stderr:created 2024-03-25T18:27:31.801208+0000 2024-03-25T18:27:31.802 INFO:teuthology.misc.health.smithi001.stderr:min_mon_release 0 (unknown) 2024-03-25T18:27:31.802 INFO:teuthology.misc.health.smithi001.stderr:election_strategy: 3 2024-03-25T18:27:31.802 INFO:teuthology.misc.health.smithi001.stderr:0: v1:172.21.15.1:6789/0 mon.noname-a 2024-03-25T18:27:31.802 INFO:teuthology.misc.health.smithi001.stderr:1: v1:172.21.15.115:6789/0 mon.noname-c 2024-03-25T18:27:31.802 INFO:teuthology.misc.health.smithi001.stderr:2: v1:172.21.15.1:6790/0 mon.noname-b 2024-03-25T18:27:31.802 INFO:teuthology.misc.health.smithi001.stderr: 2024-03-25T18:27:31.802 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:27:31.801+0000 7fd60e94b640 1 Processor -- start 2024-03-25T18:27:31.802 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:27:31.802+0000 7fd60e94b640 1 -- start start 2024-03-25T18:27:31.803 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:27:31.802+0000 7fd60e94b640 10 monclient: init 2024-03-25T18:27:31.804 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:27:31.802+0000 7fd60e94b640 10 monclient: _reopen_session rank -1 2024-03-25T18:27:31.804 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:27:31.802+0000 7fd60e94b640 10 monclient: _add_conns ranks=[1,0,2] 2024-03-25T18:27:31.804 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:27:31.802+0000 7fd60e94b640 10 monclient(hunting): picked mon.noname-c con 0x7fd6080b4ff0 addr v1:172.21.15.115:6789/0 2024-03-25T18:27:31.804 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:27:31.802+0000 7fd60e94b640 10 monclient(hunting): picked mon.noname-a con 0x7fd6080b4670 addr v1:172.21.15.1:6789/0 2024-03-25T18:27:31.804 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:27:31.802+0000 7fd60e94b640 10 monclient(hunting): picked mon.noname-b con 0x7fd60813c480 addr v1:172.21.15.1:6790/0 2024-03-25T18:27:31.804 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:27:31.802+0000 7fd60e94b640 1 -- --> v1:172.21.15.1:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd6080ab0e0 con 0x7fd6080b4670 2024-03-25T18:27:31.804 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:27:31.802+0000 7fd60e94b640 1 -- --> v1:172.21.15.115:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd6080ab3c0 con 0x7fd6080b4ff0 2024-03-25T18:27:31.804 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:27:31.802+0000 7fd60e94b640 1 -- --> v1:172.21.15.1:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd6080a9f30 con 0x7fd60813c480 2024-03-25T18:27:31.804 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:27:31.802+0000 7fd60e94b640 10 monclient(hunting): _renew_subs 2024-03-25T18:27:31.804 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:27:31.802+0000 7fd60e94b640 10 monclient(hunting): authenticate will time out at 7532.272618s 2024-03-25T18:27:31.815 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:27:31.814+0000 7fd6077fe640 1 --1- >> v1:172.21.15.1:6789/0 conn(0x7fd6080b4670 0x7fd6081425c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.1:6789/0 says I am v1:172.21.15.1:47122/0 (socket says 172.21.15.1:47122) 2024-03-25T18:27:31.815 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:27:31.814+0000 7fd6077fe640 1 -- 172.21.15.1:0/520271149 learned_addr learned my addr 172.21.15.1:0/520271149 (peer_addr_for_me v1:172.21.15.1:0/0) 2024-03-25T18:27:31.827 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:27:31.827+0000 7fd606ffd640 1 -- 172.21.15.1:0/520271149 <== mon.0 v1:172.21.15.1:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 195690933 0 0) 0x7fd6080ab0e0 con 0x7fd6080b4670 2024-03-25T18:27:31.828 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:27:31.827+0000 7fd606ffd640 10 monclient(hunting): _negotiate 2024-03-25T18:27:31.828 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:27:31.827+0000 7fd606ffd640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:27:31.828 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:27:31.827+0000 7fd606ffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:27:31.828 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:27:31.827+0000 7fd606ffd640 10 monclient(hunting): my global_id is 4254 2024-03-25T18:27:31.828 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:27:31.827+0000 7fd606ffd640 1 -- 172.21.15.1:0/520271149 --> v1:172.21.15.1:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fd5f0003520 con 0x7fd6080b4670 2024-03-25T18:27:31.837 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:27:31.837+0000 7fd606ffd640 1 -- 172.21.15.1:0/520271149 <== mon.1 v1:172.21.15.115:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2775868901 0 0) 0x7fd6080ab3c0 con 0x7fd6080b4ff0 2024-03-25T18:27:31.837 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:27:31.837+0000 7fd606ffd640 10 monclient(hunting): _negotiate 2024-03-25T18:27:31.837 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:27:31.837+0000 7fd606ffd640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:27:31.837 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:27:31.837+0000 7fd606ffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:27:31.838 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:27:31.837+0000 7fd606ffd640 10 monclient(hunting): my global_id is 4252 2024-03-25T18:27:31.838 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:27:31.837+0000 7fd606ffd640 1 -- 172.21.15.1:0/520271149 --> v1:172.21.15.115:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fd6080ab0e0 con 0x7fd6080b4ff0 2024-03-25T18:27:31.843 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:27:31.842+0000 7fd606ffd640 1 -- 172.21.15.1:0/520271149 <== mon.0 v1:172.21.15.1:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1645883562 0 0) 0x7fd5f0003520 con 0x7fd6080b4670 2024-03-25T18:27:31.843 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:27:31.842+0000 7fd606ffd640 1 -- 172.21.15.1:0/520271149 >> v1:172.21.15.1:6790/0 conn(0x7fd60813c480 legacy=0x7fd60814dc00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:27:31.843 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:27:31.842+0000 7fd606ffd640 1 -- 172.21.15.1:0/520271149 >> v1:172.21.15.115:6789/0 conn(0x7fd6080b4ff0 legacy=0x7fd60813bd70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:27:31.843 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:27:31.842+0000 7fd606ffd640 10 monclient: _finish_hunting 0 2024-03-25T18:27:31.843 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:27:31.842+0000 7fd606ffd640 1 monclient: found mon.noname-a 2024-03-25T18:27:31.843 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:27:31.842+0000 7fd606ffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:27:31.844 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:27:31.842+0000 7fd606ffd640 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.1:6789/0 2024-03-25T18:27:31.844 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:27:31.842+0000 7fd606ffd640 1 -- 172.21.15.1:0/520271149 --> v1:172.21.15.1:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd6080a9b70 con 0x7fd6080b4670 2024-03-25T18:27:31.844 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:27:31.842+0000 7fd606ffd640 10 monclient: _finish_auth 0 2024-03-25T18:27:31.844 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:27:31.842+0000 7fd606ffd640 10 monclient: _check_auth_tickets 2024-03-25T18:27:31.844 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:27:31.842+0000 7fd606ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:27:31.844 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:27:31.842+0000 7fd606ffd640 1 -- 172.21.15.1:0/520271149 <== mon.0 v1:172.21.15.1:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1586965025 0 0) 0x7fd5fc001690 con 0x7fd6080b4670 2024-03-25T18:27:31.844 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:27:31.842+0000 7fd606ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:27:31.844 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:27:31.842+0000 7fd606ffd640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-03-25T18:27:31.844 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:27:31.842+0000 7fd606ffd640 10 monclient: dump: 2024-03-25T18:27:31.844 INFO:teuthology.misc.health.smithi001.stderr:epoch 1 2024-03-25T18:27:31.844 INFO:teuthology.misc.health.smithi001.stderr:fsid c8ea72cf-9e99-4383-9b52-6c23d365fdfc 2024-03-25T18:27:31.844 INFO:teuthology.misc.health.smithi001.stderr:last_changed 2024-03-25T18:25:50.510519+0000 2024-03-25T18:27:31.844 INFO:teuthology.misc.health.smithi001.stderr:created 2024-03-25T18:25:50.510519+0000 2024-03-25T18:27:31.844 INFO:teuthology.misc.health.smithi001.stderr:min_mon_release 19 (squid) 2024-03-25T18:27:31.845 INFO:teuthology.misc.health.smithi001.stderr:election_strategy: 3 2024-03-25T18:27:31.845 INFO:teuthology.misc.health.smithi001.stderr:0: v1:172.21.15.1:6789/0 mon.a 2024-03-25T18:27:31.845 INFO:teuthology.misc.health.smithi001.stderr:1: v1:172.21.15.115:6789/0 mon.b 2024-03-25T18:27:31.845 INFO:teuthology.misc.health.smithi001.stderr:2: v1:172.21.15.1:6790/0 mon.c 2024-03-25T18:27:31.845 INFO:teuthology.misc.health.smithi001.stderr: 2024-03-25T18:27:31.845 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:27:31.842+0000 7fd60e94b640 5 monclient: authenticate success, global_id 4254 2024-03-25T18:27:31.845 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:27:31.842+0000 7fd60e94b640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T18:27:31.862 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:27:31.861+0000 7fd606ffd640 1 -- 172.21.15.1:0/520271149 <== mon.0 v1:172.21.15.1:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fd5fc0017f0 con 0x7fd6080b4670 2024-03-25T18:27:31.862 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:27:31.861+0000 7fd606ffd640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:27:31.862 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:27:31.861+0000 7fd606ffd640 1 -- 172.21.15.1:0/520271149 <== mon.0 v1:172.21.15.1:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1586965025 0 0) 0x7fd5fc001c00 con 0x7fd6080b4670 2024-03-25T18:27:31.862 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:27:31.861+0000 7fd606ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:27:31.862 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:27:31.861+0000 7fd606ffd640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-25T18:27:31.863 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:27:31.861+0000 7fd606ffd640 10 monclient: dump: 2024-03-25T18:27:31.863 INFO:teuthology.misc.health.smithi001.stderr:epoch 1 2024-03-25T18:27:31.863 INFO:teuthology.misc.health.smithi001.stderr:fsid c8ea72cf-9e99-4383-9b52-6c23d365fdfc 2024-03-25T18:27:31.863 INFO:teuthology.misc.health.smithi001.stderr:last_changed 2024-03-25T18:25:50.510519+0000 2024-03-25T18:27:31.863 INFO:teuthology.misc.health.smithi001.stderr:created 2024-03-25T18:25:50.510519+0000 2024-03-25T18:27:31.863 INFO:teuthology.misc.health.smithi001.stderr:min_mon_release 19 (squid) 2024-03-25T18:27:31.863 INFO:teuthology.misc.health.smithi001.stderr:election_strategy: 3 2024-03-25T18:27:31.863 INFO:teuthology.misc.health.smithi001.stderr:0: v1:172.21.15.1:6789/0 mon.a 2024-03-25T18:27:31.863 INFO:teuthology.misc.health.smithi001.stderr:1: v1:172.21.15.115:6789/0 mon.b 2024-03-25T18:27:31.863 INFO:teuthology.misc.health.smithi001.stderr:2: v1:172.21.15.1:6790/0 mon.c 2024-03-25T18:27:31.863 INFO:teuthology.misc.health.smithi001.stderr: 2024-03-25T18:27:31.863 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:27:31.861+0000 7fd60e94b640 10 monclient: get_monmap_and_config success 2024-03-25T18:27:31.863 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:27:31.861+0000 7fd60e94b640 4 set_mon_vals no callback set 2024-03-25T18:27:31.865 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:27:31.862+0000 7fd60e94b640 10 monclient: shutdown 2024-03-25T18:27:31.865 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:27:31.862+0000 7fd60e94b640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:27:31.865 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:27:31.862+0000 7fd60e94b640 1 -- 172.21.15.1:0/520271149 >> v1:172.21.15.1:6789/0 conn(0x7fd6080b4670 legacy=0x7fd6081425c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:27:31.865 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:27:31.862+0000 7fd60e94b640 1 -- 172.21.15.1:0/520271149 shutdown_connections 2024-03-25T18:27:31.865 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:27:31.862+0000 7fd60e94b640 1 -- 172.21.15.1:0/520271149 >> 172.21.15.1:0/520271149 conn(0x7fd6080617a0 msgr2=0x7fd608061bb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:27:31.865 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:27:31.862+0000 7fd60e94b640 1 -- 172.21.15.1:0/520271149 shutdown_connections 2024-03-25T18:27:31.865 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:27:31.862+0000 7fd60e94b640 1 -- 172.21.15.1:0/520271149 wait complete. 2024-03-25T18:27:31.865 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:27:31.863+0000 7fd60e94b640 10 monclient: build_initial_monmap 2024-03-25T18:27:31.865 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:27:31.863+0000 7fd60e94b640 10 monclient: monmap: 2024-03-25T18:27:31.865 INFO:teuthology.misc.health.smithi001.stderr:epoch 0 2024-03-25T18:27:31.865 INFO:teuthology.misc.health.smithi001.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T18:27:31.865 INFO:teuthology.misc.health.smithi001.stderr:last_changed 0.000000 2024-03-25T18:27:31.866 INFO:teuthology.misc.health.smithi001.stderr:created 0.000000 2024-03-25T18:27:31.866 INFO:teuthology.misc.health.smithi001.stderr:min_mon_release 0 (unknown) 2024-03-25T18:27:31.866 INFO:teuthology.misc.health.smithi001.stderr:election_strategy: 1 2024-03-25T18:27:31.866 INFO:teuthology.misc.health.smithi001.stderr:0: v1:172.21.15.1:6789/0 mon.noname-a 2024-03-25T18:27:31.866 INFO:teuthology.misc.health.smithi001.stderr:1: v1:172.21.15.115:6789/0 mon.noname-b 2024-03-25T18:27:31.866 INFO:teuthology.misc.health.smithi001.stderr:2: v1:172.21.15.1:6790/0 mon.noname-c 2024-03-25T18:27:31.866 INFO:teuthology.misc.health.smithi001.stderr: 2024-03-25T18:27:31.866 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:27:31.864+0000 7fd60e94b640 1 librados: starting msgr at 2024-03-25T18:27:31.866 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:27:31.864+0000 7fd60e94b640 1 librados: starting objecter 2024-03-25T18:27:31.866 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:27:31.864+0000 7fd60e94b640 1 Processor -- start 2024-03-25T18:27:31.866 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:27:31.864+0000 7fd60e94b640 1 -- start start 2024-03-25T18:27:31.866 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:27:31.864+0000 7fd60e94b640 1 librados: setting wanted keys 2024-03-25T18:27:31.866 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:27:31.864+0000 7fd60e94b640 1 librados: calling monclient init 2024-03-25T18:27:31.866 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:27:31.864+0000 7fd60e94b640 10 monclient: init 2024-03-25T18:27:31.868 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:27:31.865+0000 7fd60e94b640 10 monclient: _reopen_session rank -1 2024-03-25T18:27:31.868 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:27:31.865+0000 7fd60e94b640 10 monclient: _add_conns ranks=[1,0,2] 2024-03-25T18:27:31.868 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:27:31.865+0000 7fd60e94b640 10 monclient(hunting): picked mon.noname-b con 0x7fd6080b4670 addr v1:172.21.15.115:6789/0 2024-03-25T18:27:31.868 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:27:31.865+0000 7fd60e94b640 10 monclient(hunting): picked mon.noname-a con 0x7fd6080b4ff0 addr v1:172.21.15.1:6789/0 2024-03-25T18:27:31.868 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:27:31.865+0000 7fd60e94b640 10 monclient(hunting): picked mon.noname-c con 0x7fd60813c480 addr v1:172.21.15.1:6790/0 2024-03-25T18:27:31.868 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:27:31.865+0000 7fd60e94b640 1 -- --> v1:172.21.15.1:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd608144740 con 0x7fd6080b4ff0 2024-03-25T18:27:31.868 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:27:31.865+0000 7fd60e94b640 1 -- --> v1:172.21.15.115:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd608130b60 con 0x7fd6080b4670 2024-03-25T18:27:31.869 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:27:31.865+0000 7fd60e94b640 1 -- --> v1:172.21.15.1:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd60813c930 con 0x7fd60813c480 2024-03-25T18:27:31.869 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:27:31.865+0000 7fd60e94b640 10 monclient(hunting): _renew_subs 2024-03-25T18:27:31.869 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:27:31.865+0000 7fd60e94b640 10 monclient(hunting): authenticate will time out at 7532.335124s 2024-03-25T18:27:31.869 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:27:31.865+0000 7fd607fff640 1 --1- >> v1:172.21.15.115:6789/0 conn(0x7fd6080b4670 0x7fd608145d20 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.115:6789/0 says I am v1:172.21.15.1:55424/0 (socket says 172.21.15.1:55424) 2024-03-25T18:27:31.869 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:27:31.865+0000 7fd607fff640 1 -- 172.21.15.1:0/1364281786 learned_addr learned my addr 172.21.15.1:0/1364281786 (peer_addr_for_me v1:172.21.15.1:0/0) 2024-03-25T18:27:31.870 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:27:31.866+0000 7fd604ff9640 10 client.?.objecter ms_handle_connect 0x7fd6080b4670 2024-03-25T18:27:31.870 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:27:31.866+0000 7fd604ff9640 10 client.?.objecter resend_mon_ops 2024-03-25T18:27:31.876 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:27:31.875+0000 7fd604ff9640 10 client.?.objecter ms_handle_connect 0x7fd60813c480 2024-03-25T18:27:31.876 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:27:31.875+0000 7fd604ff9640 10 client.?.objecter resend_mon_ops 2024-03-25T18:27:31.877 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:27:31.877+0000 7fd604ff9640 10 client.?.objecter ms_handle_connect 0x7fd6080b4ff0 2024-03-25T18:27:31.878 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:27:31.877+0000 7fd604ff9640 10 client.?.objecter resend_mon_ops 2024-03-25T18:27:31.881 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:27:31.880+0000 7fd604ff9640 1 -- 172.21.15.1:0/1364281786 <== mon.0 v1:172.21.15.1:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3482691271 0 0) 0x7fd608144740 con 0x7fd6080b4ff0 2024-03-25T18:27:31.881 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:27:31.880+0000 7fd604ff9640 10 monclient(hunting): _negotiate 2024-03-25T18:27:31.881 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:27:31.880+0000 7fd604ff9640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:27:31.881 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:27:31.880+0000 7fd604ff9640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:27:31.882 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:27:31.880+0000 7fd604ff9640 10 monclient(hunting): my global_id is 4257 2024-03-25T18:27:31.882 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:27:31.880+0000 7fd604ff9640 1 -- 172.21.15.1:0/1364281786 --> v1:172.21.15.1:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fd5f4004eb0 con 0x7fd6080b4ff0 2024-03-25T18:27:31.890 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:27:31.888+0000 7fd604ff9640 1 -- 172.21.15.1:0/1364281786 <== mon.0 v1:172.21.15.1:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2463048320 0 0) 0x7fd5f4004eb0 con 0x7fd6080b4ff0 2024-03-25T18:27:31.890 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:27:31.888+0000 7fd604ff9640 1 -- 172.21.15.1:0/1364281786 --> v1:172.21.15.1:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fd608144740 con 0x7fd6080b4ff0 2024-03-25T18:27:31.890 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:27:31.888+0000 7fd604ff9640 1 -- 172.21.15.1:0/1364281786 <== mon.0 v1:172.21.15.1:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1586965025 0 0) 0x7fd5fc001b00 con 0x7fd6080b4ff0 2024-03-25T18:27:31.890 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:27:31.888+0000 7fd604ff9640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:27:31.891 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:27:31.888+0000 7fd604ff9640 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2024-03-25T18:27:31.891 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:27:31.888+0000 7fd604ff9640 10 monclient(hunting): dump: 2024-03-25T18:27:31.891 INFO:teuthology.misc.health.smithi001.stderr:epoch 1 2024-03-25T18:27:31.891 INFO:teuthology.misc.health.smithi001.stderr:fsid c8ea72cf-9e99-4383-9b52-6c23d365fdfc 2024-03-25T18:27:31.891 INFO:teuthology.misc.health.smithi001.stderr:last_changed 2024-03-25T18:25:50.510519+0000 2024-03-25T18:27:31.891 INFO:teuthology.misc.health.smithi001.stderr:created 2024-03-25T18:25:50.510519+0000 2024-03-25T18:27:31.891 INFO:teuthology.misc.health.smithi001.stderr:min_mon_release 19 (squid) 2024-03-25T18:27:31.892 INFO:teuthology.misc.health.smithi001.stderr:election_strategy: 3 2024-03-25T18:27:31.892 INFO:teuthology.misc.health.smithi001.stderr:0: v1:172.21.15.1:6789/0 mon.a 2024-03-25T18:27:31.892 INFO:teuthology.misc.health.smithi001.stderr:1: v1:172.21.15.115:6789/0 mon.b 2024-03-25T18:27:31.892 INFO:teuthology.misc.health.smithi001.stderr:2: v1:172.21.15.1:6790/0 mon.c 2024-03-25T18:27:31.892 INFO:teuthology.misc.health.smithi001.stderr: 2024-03-25T18:27:31.892 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:27:31.889+0000 7fd604ff9640 10 monclient(hunting): _finish_auth 0 2024-03-25T18:27:31.903 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:27:31.902+0000 7fd604ff9640 1 -- 172.21.15.1:0/1364281786 <== mon.0 v1:172.21.15.1:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 991714092 0 0) 0x7fd608144740 con 0x7fd6080b4ff0 2024-03-25T18:27:31.903 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:27:31.902+0000 7fd604ff9640 1 -- 172.21.15.1:0/1364281786 >> v1:172.21.15.1:6790/0 conn(0x7fd60813c480 legacy=0x7fd608130450 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:27:31.903 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:27:31.902+0000 7fd604ff9640 1 -- 172.21.15.1:0/1364281786 >> v1:172.21.15.115:6789/0 conn(0x7fd6080b4670 legacy=0x7fd608145d20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:27:31.903 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:27:31.902+0000 7fd604ff9640 10 monclient: _finish_hunting 0 2024-03-25T18:27:31.904 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:27:31.902+0000 7fd604ff9640 1 monclient: found mon.a 2024-03-25T18:27:31.904 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:27:31.902+0000 7fd604ff9640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:27:31.904 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:27:31.902+0000 7fd604ff9640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.1:6789/0 2024-03-25T18:27:31.904 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:27:31.902+0000 7fd604ff9640 1 -- 172.21.15.1:0/1364281786 --> v1:172.21.15.1:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd608144970 con 0x7fd6080b4ff0 2024-03-25T18:27:31.904 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:27:31.902+0000 7fd604ff9640 10 monclient: _finish_auth 0 2024-03-25T18:27:31.904 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:27:31.902+0000 7fd604ff9640 10 monclient: _check_auth_tickets 2024-03-25T18:27:31.904 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:27:31.902+0000 7fd604ff9640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:27:31.904 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:27:31.902+0000 7fd60e94b640 5 monclient: authenticate success, global_id 4257 2024-03-25T18:27:31.904 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:27:31.902+0000 7fd60e94b640 10 monclient: _renew_subs 2024-03-25T18:27:31.904 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:27:31.902+0000 7fd60e94b640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.1:6789/0 2024-03-25T18:27:31.904 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:27:31.902+0000 7fd60e94b640 1 -- 172.21.15.1:0/1364281786 --> v1:172.21.15.1:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd608144c50 con 0x7fd6080b4ff0 2024-03-25T18:27:31.905 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:27:31.902+0000 7fd60e94b640 10 client.4257.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T18:27:31.905 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:27:31.902+0000 7fd60e94b640 10 monclient: _renew_subs 2024-03-25T18:27:31.905 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:27:31.902+0000 7fd60e94b640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.1:6789/0 2024-03-25T18:27:31.905 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:27:31.902+0000 7fd60e94b640 1 -- 172.21.15.1:0/1364281786 --> v1:172.21.15.1:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7fd60813f280 con 0x7fd6080b4ff0 2024-03-25T18:27:31.905 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:27:31.902+0000 7fd60e94b640 1 librados: init done 2024-03-25T18:27:31.905 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:27:31.903+0000 7fd60e94b640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T18:27:31.905 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:27:31.903+0000 7fd60e94b640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:27:31.905 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:27:31.903+0000 7fd60e94b640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.1:6789/0 2024-03-25T18:27:31.906 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:27:31.903+0000 7fd60e94b640 1 -- 172.21.15.1:0/1364281786 --> v1:172.21.15.1:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd60806dbd0 con 0x7fd6080b4ff0 2024-03-25T18:27:32.286 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:27:32.285+0000 7fd604ff9640 1 -- 172.21.15.1:0/1364281786 <== mon.0 v1:172.21.15.1:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fd5fc001220 con 0x7fd6080b4ff0 2024-03-25T18:27:32.286 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:27:32.285+0000 7fd604ff9640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:27:32.286 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:27:32.285+0000 7fd604ff9640 1 -- 172.21.15.1:0/1364281786 <== mon.0 v1:172.21.15.1:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1586965025 0 0) 0x7fd5fc004c30 con 0x7fd6080b4ff0 2024-03-25T18:27:32.286 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:27:32.285+0000 7fd604ff9640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:27:32.286 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:27:32.285+0000 7fd604ff9640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-25T18:27:32.287 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:27:32.285+0000 7fd604ff9640 10 monclient: dump: 2024-03-25T18:27:32.287 INFO:teuthology.misc.health.smithi001.stderr:epoch 1 2024-03-25T18:27:32.287 INFO:teuthology.misc.health.smithi001.stderr:fsid c8ea72cf-9e99-4383-9b52-6c23d365fdfc 2024-03-25T18:27:32.287 INFO:teuthology.misc.health.smithi001.stderr:last_changed 2024-03-25T18:25:50.510519+0000 2024-03-25T18:27:32.287 INFO:teuthology.misc.health.smithi001.stderr:created 2024-03-25T18:25:50.510519+0000 2024-03-25T18:27:32.287 INFO:teuthology.misc.health.smithi001.stderr:min_mon_release 19 (squid) 2024-03-25T18:27:32.287 INFO:teuthology.misc.health.smithi001.stderr:election_strategy: 3 2024-03-25T18:27:32.287 INFO:teuthology.misc.health.smithi001.stderr:0: v1:172.21.15.1:6789/0 mon.a 2024-03-25T18:27:32.287 INFO:teuthology.misc.health.smithi001.stderr:1: v1:172.21.15.115:6789/0 mon.b 2024-03-25T18:27:32.287 INFO:teuthology.misc.health.smithi001.stderr:2: v1:172.21.15.1:6790/0 mon.c 2024-03-25T18:27:32.287 INFO:teuthology.misc.health.smithi001.stderr: 2024-03-25T18:27:32.287 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:27:32.285+0000 7fd6067fc640 4 set_mon_vals no callback set 2024-03-25T18:27:32.288 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:27:32.286+0000 7fd604ff9640 1 -- 172.21.15.1:0/1364281786 <== mon.0 v1:172.21.15.1:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 21787975 0 0) 0x7fd5fc01ac00 con 0x7fd6080b4ff0 2024-03-25T18:27:32.288 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:27:32.287+0000 7fd607fff640 10 monclient: get_auth_request con 0x7fd5f406dce0 auth_method 0 2024-03-25T18:27:32.288 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:27:32.287+0000 7fd604ff9640 10 client.4257.objecter ms_handle_connect 0x7fd5f406dce0 2024-03-25T18:27:32.293 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:27:32.292+0000 7fd604ff9640 1 -- 172.21.15.1:0/1364281786 <== mon.0 v1:172.21.15.1:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 996254230 0 0) 0x7fd5fc054860 con 0x7fd6080b4ff0 2024-03-25T18:27:32.294 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:27:32.292+0000 7fd604ff9640 10 client.4257.objecter ms_dispatch 0x7fd608000b90 osd_map(10..10 src has 1..10) v4 2024-03-25T18:27:32.294 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:27:32.292+0000 7fd604ff9640 3 client.4257.objecter handle_osd_map got epochs [10,10] > 0 2024-03-25T18:27:32.294 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:27:32.292+0000 7fd604ff9640 3 client.4257.objecter handle_osd_map decoding full epoch 10 2024-03-25T18:27:32.294 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:27:32.292+0000 7fd604ff9640 20 client.4257.objecter dump_active .. 0 homeless 2024-03-25T18:27:32.294 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:27:32.292+0000 7fd604ff9640 1 -- 172.21.15.1:0/1364281786 <== mon.0 v1:172.21.15.1:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7fd5fc0501a0 con 0x7fd6080b4ff0 2024-03-25T18:27:32.294 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:27:32.292+0000 7fd604ff9640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:27:32.294 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:27:32.292+0000 7fd604ff9640 10 monclient: _finish_command 1 = system:0 2024-03-25T18:27:32.551 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:27:32.550+0000 7fd60e94b640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:27:32.551 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:27:32.550+0000 7fd60e94b640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:27:32.551 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:27:32.550+0000 7fd60e94b640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.1:6789/0 2024-03-25T18:27:32.551 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:27:32.550+0000 7fd60e94b640 1 -- 172.21.15.1:0/1364281786 --> v1:172.21.15.1:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fd6080688c0 con 0x7fd6080b4ff0 2024-03-25T18:27:32.577 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:27:32.576+0000 7fd604ff9640 1 -- 172.21.15.1:0/1364281786 <== mon.0 v1:172.21.15.1:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 4092213348) 0x7fd5fc0522a0 con 0x7fd6080b4ff0 2024-03-25T18:27:32.577 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:27:32.576+0000 7fd604ff9640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:27:32.577 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:27:32.576+0000 7fd604ff9640 10 monclient: _finish_command 2 = system:0 2024-03-25T18:27:32.577 INFO:teuthology.misc.health.smithi001.stdout: 2024-03-25T18:27:32.577 INFO:teuthology.misc.health.smithi001.stdout:{"epoch":10,"fsid":"c8ea72cf-9e99-4383-9b52-6c23d365fdfc","created":"2024-03-25T18:26:38.856638+0000","modified":"2024-03-25T18:26:44.224208+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T18:26:44.224208+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"b9f34922-3a84-49be-8426-807c1eceaefc","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"42bf306f-55e3-4634-9874-8b4b73ac6644","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"292cb23a-18a9-4207-a7cc-4d5727feaec0","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"45e0e9ab-898d-4673-9590-b4117bc5e84a","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"831a5a09-73ab-4a61-9872-088e76bdbc66","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"88cd903d-3b55-4840-9396-6e3cfe005282","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"89db7355-bc9d-46f4-8ab0-e8d9e744e026","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"cac47bee-0914-4f08-896f-348838882866","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T18:27:32.580 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:27:32.578+0000 7fd60e94b640 10 librados: watch_flush enter 2024-03-25T18:27:32.580 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:27:32.578+0000 7fd60e94b640 10 librados: watch_flush exit 2024-03-25T18:27:32.580 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:27:32.578+0000 7fd60e94b640 20 client.4257.objecter shutdown clearing up homeless session... 2024-03-25T18:27:32.581 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:27:32.578+0000 7fd60e94b640 10 client.4257.objecter successfully canceled tick 2024-03-25T18:27:32.581 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:27:32.578+0000 7fd60e94b640 1 -- 172.21.15.1:0/1364281786 >> v1:172.21.15.115:6800/4237379176 conn(0x7fd5f406dce0 legacy=0x7fd5f40701a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:27:32.581 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:27:32.578+0000 7fd60e94b640 10 monclient: shutdown 2024-03-25T18:27:32.581 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:27:32.578+0000 7fd60e94b640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:27:32.581 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:27:32.578+0000 7fd60e94b640 1 -- 172.21.15.1:0/1364281786 >> v1:172.21.15.1:6789/0 conn(0x7fd6080b4ff0 legacy=0x7fd60812fd40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:27:32.581 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:27:32.578+0000 7fd60e94b640 1 -- 172.21.15.1:0/1364281786 shutdown_connections 2024-03-25T18:27:32.581 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:27:32.578+0000 7fd60e94b640 1 -- 172.21.15.1:0/1364281786 >> 172.21.15.1:0/1364281786 conn(0x7fd6080617a0 msgr2=0x7fd6081407f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:27:32.581 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:27:32.579+0000 7fd60e94b640 1 -- 172.21.15.1:0/1364281786 shutdown_connections 2024-03-25T18:27:32.582 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:27:32.579+0000 7fd60e94b640 1 -- 172.21.15.1:0/1364281786 wait complete. 2024-03-25T18:27:32.582 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:27:32.579+0000 7fd60e94b640 1 librados: shutdown 2024-03-25T18:27:32.592 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T18:27:35.967 INFO:tasks.ceph.mon.a.smithi001.stderr:==00:00:01:14.526 34876== Warning: unimplemented fcntl command: 1036 2024-03-25T18:27:36.024 INFO:tasks.ceph.mon.a.smithi001.stderr:==00:00:01:14.583 34876== Warning: unimplemented fcntl command: 1036 2024-03-25T18:27:36.037 INFO:tasks.ceph.mon.b.smithi115.stderr:==00:00:01:14.520 34836== Warning: unimplemented fcntl command: 1036 2024-03-25T18:27:36.046 INFO:tasks.ceph.mon.c.smithi001.stderr:==00:00:01:14.605 34877== Warning: unimplemented fcntl command: 1036 2024-03-25T18:27:36.098 INFO:tasks.ceph.mon.b.smithi115.stderr:==00:00:01:14.582 34836== Warning: unimplemented fcntl command: 1036 2024-03-25T18:27:36.118 INFO:tasks.ceph.mon.c.smithi001.stderr:==00:00:01:14.677 34877== Warning: unimplemented fcntl command: 1036 2024-03-25T18:27:36.300 INFO:tasks.ceph.mon.a.smithi001.stderr:==00:00:01:14.859 34876== Warning: unimplemented fcntl command: 1036 2024-03-25T18:27:36.324 INFO:tasks.ceph.mon.b.smithi115.stderr:==00:00:01:14.807 34836== Warning: unimplemented fcntl command: 1036 2024-03-25T18:27:36.384 INFO:tasks.ceph.mon.c.smithi001.stderr:==00:00:01:14.943 34877== Warning: unimplemented fcntl command: 1036 2024-03-25T18:27:39.399 DEBUG:teuthology.orchestra.run.smithi001:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T18:27:39.514 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:27:39.513+0000 7f7ba1bcf640 10 monclient: get_monmap_and_config 2024-03-25T18:27:39.515 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:27:39.513+0000 7f7ba1bcf640 10 monclient: build_initial_monmap 2024-03-25T18:27:39.515 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:27:39.513+0000 7f7ba1bcf640 10 monclient: monmap: 2024-03-25T18:27:39.515 INFO:teuthology.misc.health.smithi001.stderr:epoch 0 2024-03-25T18:27:39.515 INFO:teuthology.misc.health.smithi001.stderr:fsid c8ea72cf-9e99-4383-9b52-6c23d365fdfc 2024-03-25T18:27:39.515 INFO:teuthology.misc.health.smithi001.stderr:last_changed 2024-03-25T18:27:39.514595+0000 2024-03-25T18:27:39.515 INFO:teuthology.misc.health.smithi001.stderr:created 2024-03-25T18:27:39.514595+0000 2024-03-25T18:27:39.515 INFO:teuthology.misc.health.smithi001.stderr:min_mon_release 0 (unknown) 2024-03-25T18:27:39.515 INFO:teuthology.misc.health.smithi001.stderr:election_strategy: 3 2024-03-25T18:27:39.515 INFO:teuthology.misc.health.smithi001.stderr:0: v1:172.21.15.1:6789/0 mon.noname-a 2024-03-25T18:27:39.515 INFO:teuthology.misc.health.smithi001.stderr:1: v1:172.21.15.115:6789/0 mon.noname-c 2024-03-25T18:27:39.515 INFO:teuthology.misc.health.smithi001.stderr:2: v1:172.21.15.1:6790/0 mon.noname-b 2024-03-25T18:27:39.516 INFO:teuthology.misc.health.smithi001.stderr: 2024-03-25T18:27:39.516 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:27:39.513+0000 7f7ba1bcf640 1 Processor -- start 2024-03-25T18:27:39.516 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:27:39.514+0000 7f7ba1bcf640 1 -- start start 2024-03-25T18:27:39.517 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:27:39.514+0000 7f7ba1bcf640 10 monclient: init 2024-03-25T18:27:39.517 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:27:39.514+0000 7f7ba1bcf640 10 monclient: _reopen_session rank -1 2024-03-25T18:27:39.517 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:27:39.514+0000 7f7ba1bcf640 10 monclient: _add_conns ranks=[1,0,2] 2024-03-25T18:27:39.517 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:27:39.514+0000 7f7ba1bcf640 10 monclient(hunting): picked mon.noname-c con 0x7f7b9c0b4ff0 addr v1:172.21.15.115:6789/0 2024-03-25T18:27:39.517 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:27:39.514+0000 7f7ba1bcf640 10 monclient(hunting): picked mon.noname-a con 0x7f7b9c0b4670 addr v1:172.21.15.1:6789/0 2024-03-25T18:27:39.517 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:27:39.514+0000 7f7ba1bcf640 10 monclient(hunting): picked mon.noname-b con 0x7f7b9c13c480 addr v1:172.21.15.1:6790/0 2024-03-25T18:27:39.517 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:27:39.514+0000 7f7ba1bcf640 1 -- --> v1:172.21.15.1:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7b9c0ab0e0 con 0x7f7b9c0b4670 2024-03-25T18:27:39.517 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:27:39.514+0000 7f7ba1bcf640 1 -- --> v1:172.21.15.115:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7b9c0ab3c0 con 0x7f7b9c0b4ff0 2024-03-25T18:27:39.517 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:27:39.514+0000 7f7ba1bcf640 1 -- --> v1:172.21.15.1:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7b9c0a9f30 con 0x7f7b9c13c480 2024-03-25T18:27:39.517 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:27:39.514+0000 7f7ba1bcf640 10 monclient(hunting): _renew_subs 2024-03-25T18:27:39.517 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:27:39.514+0000 7f7ba1bcf640 10 monclient(hunting): authenticate will time out at 7539.985617s 2024-03-25T18:27:39.533 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:27:39.531+0000 7f7b9bfff640 1 --1- >> v1:172.21.15.1:6790/0 conn(0x7f7b9c13c480 0x7f7b9c14dc00 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.1:6790/0 says I am v1:172.21.15.1:46088/0 (socket says 172.21.15.1:46088) 2024-03-25T18:27:39.533 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:27:39.531+0000 7f7b9bfff640 1 -- 172.21.15.1:0/2324046577 learned_addr learned my addr 172.21.15.1:0/2324046577 (peer_addr_for_me v1:172.21.15.1:0/0) 2024-03-25T18:27:39.548 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:27:39.546+0000 7f7b9a7fc640 1 -- 172.21.15.1:0/2324046577 <== mon.0 v1:172.21.15.1:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1665068322 0 0) 0x7f7b9c0ab0e0 con 0x7f7b9c0b4670 2024-03-25T18:27:39.548 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:27:39.546+0000 7f7b9a7fc640 10 monclient(hunting): _negotiate 2024-03-25T18:27:39.548 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:27:39.546+0000 7f7b9a7fc640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:27:39.548 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:27:39.546+0000 7f7b9a7fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:27:39.548 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:27:39.546+0000 7f7b9a7fc640 10 monclient(hunting): my global_id is 4260 2024-03-25T18:27:39.548 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:27:39.546+0000 7f7b9a7fc640 1 -- 172.21.15.1:0/2324046577 --> v1:172.21.15.1:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f7b84003520 con 0x7f7b9c0b4670 2024-03-25T18:27:39.551 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:27:39.549+0000 7f7b9a7fc640 1 -- 172.21.15.1:0/2324046577 <== mon.1 v1:172.21.15.115:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2340362169 0 0) 0x7f7b9c0ab3c0 con 0x7f7b9c0b4ff0 2024-03-25T18:27:39.551 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:27:39.549+0000 7f7b9a7fc640 10 monclient(hunting): _negotiate 2024-03-25T18:27:39.551 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:27:39.549+0000 7f7b9a7fc640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:27:39.551 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:27:39.549+0000 7f7b9a7fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:27:39.551 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:27:39.549+0000 7f7b9a7fc640 10 monclient(hunting): my global_id is 4255 2024-03-25T18:27:39.551 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:27:39.550+0000 7f7b9a7fc640 1 -- 172.21.15.1:0/2324046577 --> v1:172.21.15.115:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f7b9c0ab0e0 con 0x7f7b9c0b4ff0 2024-03-25T18:27:39.556 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:27:39.554+0000 7f7b9a7fc640 1 -- 172.21.15.1:0/2324046577 <== mon.2 v1:172.21.15.1:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1577094606 0 0) 0x7f7b9c0a9f30 con 0x7f7b9c13c480 2024-03-25T18:27:39.556 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:27:39.554+0000 7f7b9a7fc640 10 monclient(hunting): _negotiate 2024-03-25T18:27:39.556 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:27:39.554+0000 7f7b9a7fc640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:27:39.556 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:27:39.554+0000 7f7b9a7fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:27:39.556 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:27:39.554+0000 7f7b9a7fc640 10 monclient(hunting): my global_id is 4262 2024-03-25T18:27:39.557 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:27:39.554+0000 7f7b9a7fc640 1 -- 172.21.15.1:0/2324046577 --> v1:172.21.15.1:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f7b9c0ab3c0 con 0x7f7b9c13c480 2024-03-25T18:27:39.564 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:27:39.562+0000 7f7b9a7fc640 1 -- 172.21.15.1:0/2324046577 <== mon.0 v1:172.21.15.1:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2797186412 0 0) 0x7f7b84003520 con 0x7f7b9c0b4670 2024-03-25T18:27:39.564 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:27:39.562+0000 7f7b9a7fc640 1 -- 172.21.15.1:0/2324046577 >> v1:172.21.15.1:6790/0 conn(0x7f7b9c13c480 legacy=0x7f7b9c14dc00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:27:39.564 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:27:39.562+0000 7f7b9a7fc640 1 -- 172.21.15.1:0/2324046577 >> v1:172.21.15.115:6789/0 conn(0x7f7b9c0b4ff0 legacy=0x7f7b9c13bd70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:27:39.564 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:27:39.562+0000 7f7b9a7fc640 10 monclient: _finish_hunting 0 2024-03-25T18:27:39.565 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:27:39.562+0000 7f7b9a7fc640 1 monclient: found mon.noname-a 2024-03-25T18:27:39.565 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:27:39.562+0000 7f7b9a7fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:27:39.565 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:27:39.562+0000 7f7b9a7fc640 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.1:6789/0 2024-03-25T18:27:39.565 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:27:39.562+0000 7f7b9a7fc640 1 -- 172.21.15.1:0/2324046577 --> v1:172.21.15.1:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7b9c0a9b70 con 0x7f7b9c0b4670 2024-03-25T18:27:39.565 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:27:39.562+0000 7f7b9a7fc640 10 monclient: _finish_auth 0 2024-03-25T18:27:39.565 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:27:39.562+0000 7f7b9a7fc640 10 monclient: _check_auth_tickets 2024-03-25T18:27:39.565 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:27:39.562+0000 7f7b9a7fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:27:39.565 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:27:39.562+0000 7f7b9a7fc640 1 -- 172.21.15.1:0/2324046577 <== mon.0 v1:172.21.15.1:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1586965025 0 0) 0x7f7b8c001690 con 0x7f7b9c0b4670 2024-03-25T18:27:39.565 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:27:39.562+0000 7f7ba1bcf640 5 monclient: authenticate success, global_id 4260 2024-03-25T18:27:39.565 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:27:39.562+0000 7f7ba1bcf640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T18:27:39.565 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:27:39.562+0000 7f7b9a7fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:27:39.565 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:27:39.562+0000 7f7b9a7fc640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-03-25T18:27:39.565 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:27:39.562+0000 7f7b9a7fc640 10 monclient: dump: 2024-03-25T18:27:39.565 INFO:teuthology.misc.health.smithi001.stderr:epoch 1 2024-03-25T18:27:39.565 INFO:teuthology.misc.health.smithi001.stderr:fsid c8ea72cf-9e99-4383-9b52-6c23d365fdfc 2024-03-25T18:27:39.565 INFO:teuthology.misc.health.smithi001.stderr:last_changed 2024-03-25T18:25:50.510519+0000 2024-03-25T18:27:39.566 INFO:teuthology.misc.health.smithi001.stderr:created 2024-03-25T18:25:50.510519+0000 2024-03-25T18:27:39.566 INFO:teuthology.misc.health.smithi001.stderr:min_mon_release 19 (squid) 2024-03-25T18:27:39.566 INFO:teuthology.misc.health.smithi001.stderr:election_strategy: 3 2024-03-25T18:27:39.566 INFO:teuthology.misc.health.smithi001.stderr:0: v1:172.21.15.1:6789/0 mon.a 2024-03-25T18:27:39.566 INFO:teuthology.misc.health.smithi001.stderr:1: v1:172.21.15.115:6789/0 mon.b 2024-03-25T18:27:39.566 INFO:teuthology.misc.health.smithi001.stderr:2: v1:172.21.15.1:6790/0 mon.c 2024-03-25T18:27:39.566 INFO:teuthology.misc.health.smithi001.stderr: 2024-03-25T18:27:39.566 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:27:39.562+0000 7f7ba1bcf640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T18:27:39.570 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:27:39.569+0000 7f7b9a7fc640 1 -- 172.21.15.1:0/2324046577 <== mon.0 v1:172.21.15.1:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f7b8c0017f0 con 0x7f7b9c0b4670 2024-03-25T18:27:39.570 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:27:39.569+0000 7f7b9a7fc640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:27:39.571 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:27:39.569+0000 7f7b9a7fc640 1 -- 172.21.15.1:0/2324046577 <== mon.0 v1:172.21.15.1:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1586965025 0 0) 0x7f7b8c001c00 con 0x7f7b9c0b4670 2024-03-25T18:27:39.571 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:27:39.569+0000 7f7ba1bcf640 10 monclient: get_monmap_and_config success 2024-03-25T18:27:39.571 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:27:39.569+0000 7f7ba1bcf640 4 set_mon_vals no callback set 2024-03-25T18:27:39.571 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:27:39.569+0000 7f7b9a7fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:27:39.571 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:27:39.569+0000 7f7b9a7fc640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-25T18:27:39.571 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:27:39.569+0000 7f7b9a7fc640 10 monclient: dump: 2024-03-25T18:27:39.571 INFO:teuthology.misc.health.smithi001.stderr:epoch 1 2024-03-25T18:27:39.571 INFO:teuthology.misc.health.smithi001.stderr:fsid c8ea72cf-9e99-4383-9b52-6c23d365fdfc 2024-03-25T18:27:39.571 INFO:teuthology.misc.health.smithi001.stderr:last_changed 2024-03-25T18:25:50.510519+0000 2024-03-25T18:27:39.571 INFO:teuthology.misc.health.smithi001.stderr:created 2024-03-25T18:25:50.510519+0000 2024-03-25T18:27:39.571 INFO:teuthology.misc.health.smithi001.stderr:min_mon_release 19 (squid) 2024-03-25T18:27:39.571 INFO:teuthology.misc.health.smithi001.stderr:election_strategy: 3 2024-03-25T18:27:39.571 INFO:teuthology.misc.health.smithi001.stderr:0: v1:172.21.15.1:6789/0 mon.a 2024-03-25T18:27:39.571 INFO:teuthology.misc.health.smithi001.stderr:1: v1:172.21.15.115:6789/0 mon.b 2024-03-25T18:27:39.571 INFO:teuthology.misc.health.smithi001.stderr:2: v1:172.21.15.1:6790/0 mon.c 2024-03-25T18:27:39.572 INFO:teuthology.misc.health.smithi001.stderr: 2024-03-25T18:27:39.573 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:27:39.570+0000 7f7ba1bcf640 10 monclient: shutdown 2024-03-25T18:27:39.573 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:27:39.570+0000 7f7ba1bcf640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:27:39.573 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:27:39.570+0000 7f7ba1bcf640 1 -- 172.21.15.1:0/2324046577 >> v1:172.21.15.1:6789/0 conn(0x7f7b9c0b4670 legacy=0x7f7b9c1425c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:27:39.573 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:27:39.570+0000 7f7ba1bcf640 1 -- 172.21.15.1:0/2324046577 shutdown_connections 2024-03-25T18:27:39.573 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:27:39.570+0000 7f7ba1bcf640 1 -- 172.21.15.1:0/2324046577 >> 172.21.15.1:0/2324046577 conn(0x7f7b9c0617a0 msgr2=0x7f7b9c061bb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:27:39.573 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:27:39.570+0000 7f7ba1bcf640 1 -- 172.21.15.1:0/2324046577 shutdown_connections 2024-03-25T18:27:39.573 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:27:39.570+0000 7f7ba1bcf640 1 -- 172.21.15.1:0/2324046577 wait complete. 2024-03-25T18:27:39.573 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:27:39.571+0000 7f7ba1bcf640 10 monclient: build_initial_monmap 2024-03-25T18:27:39.573 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:27:39.571+0000 7f7ba1bcf640 10 monclient: monmap: 2024-03-25T18:27:39.573 INFO:teuthology.misc.health.smithi001.stderr:epoch 0 2024-03-25T18:27:39.574 INFO:teuthology.misc.health.smithi001.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T18:27:39.574 INFO:teuthology.misc.health.smithi001.stderr:last_changed 0.000000 2024-03-25T18:27:39.574 INFO:teuthology.misc.health.smithi001.stderr:created 0.000000 2024-03-25T18:27:39.574 INFO:teuthology.misc.health.smithi001.stderr:min_mon_release 0 (unknown) 2024-03-25T18:27:39.574 INFO:teuthology.misc.health.smithi001.stderr:election_strategy: 1 2024-03-25T18:27:39.574 INFO:teuthology.misc.health.smithi001.stderr:0: v1:172.21.15.1:6789/0 mon.noname-a 2024-03-25T18:27:39.574 INFO:teuthology.misc.health.smithi001.stderr:1: v1:172.21.15.115:6789/0 mon.noname-b 2024-03-25T18:27:39.574 INFO:teuthology.misc.health.smithi001.stderr:2: v1:172.21.15.1:6790/0 mon.noname-c 2024-03-25T18:27:39.574 INFO:teuthology.misc.health.smithi001.stderr: 2024-03-25T18:27:39.574 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:27:39.571+0000 7f7ba1bcf640 1 librados: starting msgr at 2024-03-25T18:27:39.574 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:27:39.571+0000 7f7ba1bcf640 1 librados: starting objecter 2024-03-25T18:27:39.574 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:27:39.571+0000 7f7ba1bcf640 1 Processor -- start 2024-03-25T18:27:39.574 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:27:39.572+0000 7f7ba1bcf640 1 -- start start 2024-03-25T18:27:39.574 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:27:39.572+0000 7f7ba1bcf640 1 librados: setting wanted keys 2024-03-25T18:27:39.575 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:27:39.572+0000 7f7ba1bcf640 1 librados: calling monclient init 2024-03-25T18:27:39.575 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:27:39.572+0000 7f7ba1bcf640 10 monclient: init 2024-03-25T18:27:39.576 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:27:39.572+0000 7f7ba1bcf640 10 monclient: _reopen_session rank -1 2024-03-25T18:27:39.576 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:27:39.572+0000 7f7ba1bcf640 10 monclient: _add_conns ranks=[1,0,2] 2024-03-25T18:27:39.576 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:27:39.572+0000 7f7ba1bcf640 10 monclient(hunting): picked mon.noname-b con 0x7f7b9c0b4670 addr v1:172.21.15.115:6789/0 2024-03-25T18:27:39.576 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:27:39.572+0000 7f7ba1bcf640 10 monclient(hunting): picked mon.noname-a con 0x7f7b9c0b4ff0 addr v1:172.21.15.1:6789/0 2024-03-25T18:27:39.576 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:27:39.572+0000 7f7ba1bcf640 10 monclient(hunting): picked mon.noname-c con 0x7f7b9c13c480 addr v1:172.21.15.1:6790/0 2024-03-25T18:27:39.576 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:27:39.572+0000 7f7ba1bcf640 1 -- --> v1:172.21.15.1:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7b9c130ad0 con 0x7f7b9c0b4ff0 2024-03-25T18:27:39.576 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:27:39.572+0000 7f7ba1bcf640 1 -- --> v1:172.21.15.115:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7b9c145010 con 0x7f7b9c0b4670 2024-03-25T18:27:39.576 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:27:39.572+0000 7f7ba1bcf640 1 -- --> v1:172.21.15.1:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7b9c1452f0 con 0x7f7b9c13c480 2024-03-25T18:27:39.577 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:27:39.572+0000 7f7ba1bcf640 10 monclient(hunting): _renew_subs 2024-03-25T18:27:39.577 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:27:39.572+0000 7f7ba1bcf640 10 monclient(hunting): authenticate will time out at 7540.043632s 2024-03-25T18:27:39.589 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:27:39.588+0000 7f7b9bfff640 1 --1- >> v1:172.21.15.1:6790/0 conn(0x7f7b9c13c480 0x7f7b9c1303c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.1:6790/0 says I am v1:172.21.15.1:46134/0 (socket says 172.21.15.1:46134) 2024-03-25T18:27:39.589 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:27:39.588+0000 7f7b9bfff640 1 -- 172.21.15.1:0/737731788 learned_addr learned my addr 172.21.15.1:0/737731788 (peer_addr_for_me v1:172.21.15.1:0/0) 2024-03-25T18:27:39.590 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:27:39.589+0000 7f7b83fff640 10 client.?.objecter ms_handle_connect 0x7f7b9c13c480 2024-03-25T18:27:39.590 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:27:39.589+0000 7f7b83fff640 10 client.?.objecter resend_mon_ops 2024-03-25T18:27:39.593 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:27:39.592+0000 7f7b83fff640 10 client.?.objecter ms_handle_connect 0x7f7b9c0b4670 2024-03-25T18:27:39.593 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:27:39.592+0000 7f7b83fff640 10 client.?.objecter resend_mon_ops 2024-03-25T18:27:39.601 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:27:39.600+0000 7f7b83fff640 10 client.?.objecter ms_handle_connect 0x7f7b9c0b4ff0 2024-03-25T18:27:39.601 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:27:39.600+0000 7f7b83fff640 10 client.?.objecter resend_mon_ops 2024-03-25T18:27:39.606 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:27:39.604+0000 7f7b83fff640 1 -- 172.21.15.1:0/737731788 <== mon.1 v1:172.21.15.115:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1192922035 0 0) 0x7f7b9c145010 con 0x7f7b9c0b4670 2024-03-25T18:27:39.606 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:27:39.605+0000 7f7b83fff640 10 monclient(hunting): _negotiate 2024-03-25T18:27:39.606 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:27:39.605+0000 7f7b83fff640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:27:39.606 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:27:39.605+0000 7f7b83fff640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:27:39.606 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:27:39.605+0000 7f7b83fff640 10 monclient(hunting): my global_id is 4258 2024-03-25T18:27:39.607 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:27:39.605+0000 7f7b83fff640 1 -- 172.21.15.1:0/737731788 --> v1:172.21.15.115:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f7b88004eb0 con 0x7f7b9c0b4670 2024-03-25T18:27:39.608 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:27:39.606+0000 7f7b83fff640 1 -- 172.21.15.1:0/737731788 <== mon.2 v1:172.21.15.1:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4592234 0 0) 0x7f7b9c1452f0 con 0x7f7b9c13c480 2024-03-25T18:27:39.608 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:27:39.606+0000 7f7b83fff640 10 monclient(hunting): _negotiate 2024-03-25T18:27:39.608 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:27:39.606+0000 7f7b83fff640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:27:39.608 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:27:39.606+0000 7f7b83fff640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:27:39.608 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:27:39.606+0000 7f7b83fff640 10 monclient(hunting): my global_id is 4265 2024-03-25T18:27:39.608 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:27:39.606+0000 7f7b83fff640 1 -- 172.21.15.1:0/737731788 --> v1:172.21.15.1:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f7b9c145010 con 0x7f7b9c13c480 2024-03-25T18:27:39.610 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:27:39.608+0000 7f7b83fff640 1 -- 172.21.15.1:0/737731788 <== mon.0 v1:172.21.15.1:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 358112182 0 0) 0x7f7b9c130ad0 con 0x7f7b9c0b4ff0 2024-03-25T18:27:39.610 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:27:39.608+0000 7f7b83fff640 10 monclient(hunting): _negotiate 2024-03-25T18:27:39.610 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:27:39.608+0000 7f7b83fff640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:27:39.610 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:27:39.608+0000 7f7b83fff640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:27:39.610 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:27:39.608+0000 7f7b83fff640 10 monclient(hunting): my global_id is 4263 2024-03-25T18:27:39.610 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:27:39.608+0000 7f7b83fff640 1 -- 172.21.15.1:0/737731788 --> v1:172.21.15.1:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f7b9c1452f0 con 0x7f7b9c0b4ff0 2024-03-25T18:27:39.617 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:27:39.616+0000 7f7b83fff640 1 -- 172.21.15.1:0/737731788 <== mon.1 v1:172.21.15.115:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3700540516 0 0) 0x7f7b88004eb0 con 0x7f7b9c0b4670 2024-03-25T18:27:39.618 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:27:39.616+0000 7f7b83fff640 1 -- 172.21.15.1:0/737731788 --> v1:172.21.15.115:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f7b9c130ad0 con 0x7f7b9c0b4670 2024-03-25T18:27:39.618 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:27:39.616+0000 7f7b83fff640 1 -- 172.21.15.1:0/737731788 <== mon.1 v1:172.21.15.115:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1586965025 0 0) 0x7f7b90001970 con 0x7f7b9c0b4670 2024-03-25T18:27:39.618 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:27:39.616+0000 7f7b83fff640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:27:39.618 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:27:39.616+0000 7f7b83fff640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-03-25T18:27:39.618 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:27:39.616+0000 7f7b83fff640 10 monclient(hunting): dump: 2024-03-25T18:27:39.618 INFO:teuthology.misc.health.smithi001.stderr:epoch 1 2024-03-25T18:27:39.618 INFO:teuthology.misc.health.smithi001.stderr:fsid c8ea72cf-9e99-4383-9b52-6c23d365fdfc 2024-03-25T18:27:39.618 INFO:teuthology.misc.health.smithi001.stderr:last_changed 2024-03-25T18:25:50.510519+0000 2024-03-25T18:27:39.618 INFO:teuthology.misc.health.smithi001.stderr:created 2024-03-25T18:25:50.510519+0000 2024-03-25T18:27:39.618 INFO:teuthology.misc.health.smithi001.stderr:min_mon_release 19 (squid) 2024-03-25T18:27:39.618 INFO:teuthology.misc.health.smithi001.stderr:election_strategy: 3 2024-03-25T18:27:39.618 INFO:teuthology.misc.health.smithi001.stderr:0: v1:172.21.15.1:6789/0 mon.a 2024-03-25T18:27:39.618 INFO:teuthology.misc.health.smithi001.stderr:1: v1:172.21.15.115:6789/0 mon.b 2024-03-25T18:27:39.619 INFO:teuthology.misc.health.smithi001.stderr:2: v1:172.21.15.1:6790/0 mon.c 2024-03-25T18:27:39.619 INFO:teuthology.misc.health.smithi001.stderr: 2024-03-25T18:27:39.619 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:27:39.616+0000 7f7b83fff640 10 monclient(hunting): _finish_auth 0 2024-03-25T18:27:39.623 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:27:39.621+0000 7f7b83fff640 1 -- 172.21.15.1:0/737731788 <== mon.0 v1:172.21.15.1:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2998678208 0 0) 0x7f7b9c1452f0 con 0x7f7b9c0b4ff0 2024-03-25T18:27:39.623 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:27:39.622+0000 7f7b83fff640 1 -- 172.21.15.1:0/737731788 --> v1:172.21.15.1:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f7b88004eb0 con 0x7f7b9c0b4ff0 2024-03-25T18:27:39.623 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:27:39.622+0000 7f7b83fff640 1 -- 172.21.15.1:0/737731788 <== mon.0 v1:172.21.15.1:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1586965025 0 0) 0x7f7b8c001b00 con 0x7f7b9c0b4ff0 2024-03-25T18:27:39.623 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:27:39.622+0000 7f7b83fff640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:27:39.623 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:27:39.622+0000 7f7b83fff640 10 monclient(hunting): got monmap 1 from mon.a (according to old e1) 2024-03-25T18:27:39.623 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:27:39.622+0000 7f7b83fff640 10 monclient(hunting): dump: 2024-03-25T18:27:39.624 INFO:teuthology.misc.health.smithi001.stderr:epoch 1 2024-03-25T18:27:39.624 INFO:teuthology.misc.health.smithi001.stderr:fsid c8ea72cf-9e99-4383-9b52-6c23d365fdfc 2024-03-25T18:27:39.624 INFO:teuthology.misc.health.smithi001.stderr:last_changed 2024-03-25T18:25:50.510519+0000 2024-03-25T18:27:39.624 INFO:teuthology.misc.health.smithi001.stderr:created 2024-03-25T18:25:50.510519+0000 2024-03-25T18:27:39.624 INFO:teuthology.misc.health.smithi001.stderr:min_mon_release 19 (squid) 2024-03-25T18:27:39.624 INFO:teuthology.misc.health.smithi001.stderr:election_strategy: 3 2024-03-25T18:27:39.624 INFO:teuthology.misc.health.smithi001.stderr:0: v1:172.21.15.1:6789/0 mon.a 2024-03-25T18:27:39.624 INFO:teuthology.misc.health.smithi001.stderr:1: v1:172.21.15.115:6789/0 mon.b 2024-03-25T18:27:39.624 INFO:teuthology.misc.health.smithi001.stderr:2: v1:172.21.15.1:6790/0 mon.c 2024-03-25T18:27:39.624 INFO:teuthology.misc.health.smithi001.stderr: 2024-03-25T18:27:39.626 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:27:39.624+0000 7f7b83fff640 1 -- 172.21.15.1:0/737731788 <== mon.2 v1:172.21.15.1:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3070888001 0 0) 0x7f7b9c145010 con 0x7f7b9c13c480 2024-03-25T18:27:39.626 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:27:39.625+0000 7f7b83fff640 1 -- 172.21.15.1:0/737731788 --> v1:172.21.15.1:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f7b9c1452f0 con 0x7f7b9c13c480 2024-03-25T18:27:39.627 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:27:39.625+0000 7f7b83fff640 1 -- 172.21.15.1:0/737731788 <== mon.2 v1:172.21.15.1:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1586965025 0 0) 0x7f7b94001b90 con 0x7f7b9c13c480 2024-03-25T18:27:39.627 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:27:39.625+0000 7f7b83fff640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:27:39.627 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:27:39.625+0000 7f7b83fff640 10 monclient(hunting): got monmap 1 from mon.c (according to old e1) 2024-03-25T18:27:39.627 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:27:39.625+0000 7f7b83fff640 10 monclient(hunting): dump: 2024-03-25T18:27:39.627 INFO:teuthology.misc.health.smithi001.stderr:epoch 1 2024-03-25T18:27:39.627 INFO:teuthology.misc.health.smithi001.stderr:fsid c8ea72cf-9e99-4383-9b52-6c23d365fdfc 2024-03-25T18:27:39.627 INFO:teuthology.misc.health.smithi001.stderr:last_changed 2024-03-25T18:25:50.510519+0000 2024-03-25T18:27:39.627 INFO:teuthology.misc.health.smithi001.stderr:created 2024-03-25T18:25:50.510519+0000 2024-03-25T18:27:39.627 INFO:teuthology.misc.health.smithi001.stderr:min_mon_release 19 (squid) 2024-03-25T18:27:39.627 INFO:teuthology.misc.health.smithi001.stderr:election_strategy: 3 2024-03-25T18:27:39.627 INFO:teuthology.misc.health.smithi001.stderr:0: v1:172.21.15.1:6789/0 mon.a 2024-03-25T18:27:39.627 INFO:teuthology.misc.health.smithi001.stderr:1: v1:172.21.15.115:6789/0 mon.b 2024-03-25T18:27:39.628 INFO:teuthology.misc.health.smithi001.stderr:2: v1:172.21.15.1:6790/0 mon.c 2024-03-25T18:27:39.628 INFO:teuthology.misc.health.smithi001.stderr: 2024-03-25T18:27:39.632 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:27:39.630+0000 7f7b83fff640 1 -- 172.21.15.1:0/737731788 <== mon.1 v1:172.21.15.115:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 3152111536 0 0) 0x7f7b9c130ad0 con 0x7f7b9c0b4670 2024-03-25T18:27:39.632 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:27:39.630+0000 7f7b83fff640 1 -- 172.21.15.1:0/737731788 >> v1:172.21.15.1:6790/0 conn(0x7f7b9c13c480 legacy=0x7f7b9c1303c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:27:39.632 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:27:39.630+0000 7f7b83fff640 1 -- 172.21.15.1:0/737731788 >> v1:172.21.15.1:6789/0 conn(0x7f7b9c0b4ff0 legacy=0x7f7b9c12fcb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:27:39.632 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:27:39.630+0000 7f7b83fff640 10 monclient: _finish_hunting 0 2024-03-25T18:27:39.632 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:27:39.630+0000 7f7b83fff640 1 monclient: found mon.b 2024-03-25T18:27:39.632 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:27:39.630+0000 7f7b83fff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:27:39.632 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:27:39.630+0000 7f7b83fff640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.115:6789/0 2024-03-25T18:27:39.632 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:27:39.630+0000 7f7b83fff640 1 -- 172.21.15.1:0/737731788 --> v1:172.21.15.115:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7b9c13f8a0 con 0x7f7b9c0b4670 2024-03-25T18:27:39.632 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:27:39.630+0000 7f7b83fff640 10 monclient: _finish_auth 0 2024-03-25T18:27:39.632 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:27:39.630+0000 7f7b83fff640 10 monclient: _check_auth_tickets 2024-03-25T18:27:39.632 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:27:39.630+0000 7f7b83fff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:27:39.632 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:27:39.630+0000 7f7ba1bcf640 5 monclient: authenticate success, global_id 4258 2024-03-25T18:27:39.632 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:27:39.630+0000 7f7ba1bcf640 10 monclient: _renew_subs 2024-03-25T18:27:39.633 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:27:39.630+0000 7f7ba1bcf640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.115:6789/0 2024-03-25T18:27:39.633 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:27:39.630+0000 7f7ba1bcf640 1 -- 172.21.15.1:0/737731788 --> v1:172.21.15.115:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7b9c13f530 con 0x7f7b9c0b4670 2024-03-25T18:27:39.633 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:27:39.630+0000 7f7ba1bcf640 10 client.4258.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T18:27:39.633 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:27:39.630+0000 7f7ba1bcf640 10 monclient: _renew_subs 2024-03-25T18:27:39.633 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:27:39.630+0000 7f7ba1bcf640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.115:6789/0 2024-03-25T18:27:39.633 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:27:39.630+0000 7f7ba1bcf640 1 -- 172.21.15.1:0/737731788 --> v1:172.21.15.115:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f7b9c13cd70 con 0x7f7b9c0b4670 2024-03-25T18:27:39.633 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:27:39.630+0000 7f7ba1bcf640 1 librados: init done 2024-03-25T18:27:39.633 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:27:39.631+0000 7f7ba1bcf640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T18:27:39.634 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:27:39.631+0000 7f7ba1bcf640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:27:39.634 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:27:39.631+0000 7f7ba1bcf640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.115:6789/0 2024-03-25T18:27:39.634 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:27:39.631+0000 7f7ba1bcf640 1 -- 172.21.15.1:0/737731788 --> v1:172.21.15.115:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7b9c0688c0 con 0x7f7b9c0b4670 2024-03-25T18:27:39.902 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:27:39.900+0000 7f7b83fff640 1 -- 172.21.15.1:0/737731788 <== mon.1 v1:172.21.15.115:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f7b900035f0 con 0x7f7b9c0b4670 2024-03-25T18:27:39.902 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:27:39.900+0000 7f7b83fff640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:27:39.902 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:27:39.900+0000 7f7b83fff640 1 -- 172.21.15.1:0/737731788 <== mon.1 v1:172.21.15.115:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1586965025 0 0) 0x7f7b90003760 con 0x7f7b9c0b4670 2024-03-25T18:27:39.902 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:27:39.900+0000 7f7b83fff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:27:39.903 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:27:39.900+0000 7f7b83fff640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-25T18:27:39.903 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:27:39.900+0000 7f7b83fff640 10 monclient: dump: 2024-03-25T18:27:39.903 INFO:teuthology.misc.health.smithi001.stderr:epoch 1 2024-03-25T18:27:39.903 INFO:teuthology.misc.health.smithi001.stderr:fsid c8ea72cf-9e99-4383-9b52-6c23d365fdfc 2024-03-25T18:27:39.903 INFO:teuthology.misc.health.smithi001.stderr:last_changed 2024-03-25T18:25:50.510519+0000 2024-03-25T18:27:39.903 INFO:teuthology.misc.health.smithi001.stderr:created 2024-03-25T18:25:50.510519+0000 2024-03-25T18:27:39.903 INFO:teuthology.misc.health.smithi001.stderr:min_mon_release 19 (squid) 2024-03-25T18:27:39.903 INFO:teuthology.misc.health.smithi001.stderr:election_strategy: 3 2024-03-25T18:27:39.903 INFO:teuthology.misc.health.smithi001.stderr:0: v1:172.21.15.1:6789/0 mon.a 2024-03-25T18:27:39.903 INFO:teuthology.misc.health.smithi001.stderr:1: v1:172.21.15.115:6789/0 mon.b 2024-03-25T18:27:39.903 INFO:teuthology.misc.health.smithi001.stderr:2: v1:172.21.15.1:6790/0 mon.c 2024-03-25T18:27:39.903 INFO:teuthology.misc.health.smithi001.stderr: 2024-03-25T18:27:39.903 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:27:39.900+0000 7f7b99ffb640 4 set_mon_vals no callback set 2024-03-25T18:27:39.904 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:27:39.901+0000 7f7b83fff640 1 -- 172.21.15.1:0/737731788 <== mon.1 v1:172.21.15.115:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 21787975 0 0) 0x7f7b900197b0 con 0x7f7b9c0b4670 2024-03-25T18:27:39.904 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:27:39.902+0000 7f7b9affd640 10 monclient: get_auth_request con 0x7f7b8806e020 auth_method 0 2024-03-25T18:27:39.904 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:27:39.902+0000 7f7b83fff640 10 client.4258.objecter ms_handle_connect 0x7f7b8806e020 2024-03-25T18:27:39.908 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:27:39.906+0000 7f7b83fff640 1 -- 172.21.15.1:0/737731788 <== mon.1 v1:172.21.15.115:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 996254230 0 0) 0x7f7b90053480 con 0x7f7b9c0b4670 2024-03-25T18:27:39.908 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:27:39.906+0000 7f7b83fff640 10 client.4258.objecter ms_dispatch 0x7f7b9c000b90 osd_map(10..10 src has 1..10) v4 2024-03-25T18:27:39.908 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:27:39.906+0000 7f7b83fff640 3 client.4258.objecter handle_osd_map got epochs [10,10] > 0 2024-03-25T18:27:39.908 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:27:39.906+0000 7f7b83fff640 3 client.4258.objecter handle_osd_map decoding full epoch 10 2024-03-25T18:27:39.908 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:27:39.906+0000 7f7b83fff640 20 client.4258.objecter dump_active .. 0 homeless 2024-03-25T18:27:39.909 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:27:39.907+0000 7f7b83fff640 1 -- 172.21.15.1:0/737731788 <== mon.1 v1:172.21.15.115:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f7b9004edc0 con 0x7f7b9c0b4670 2024-03-25T18:27:39.909 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:27:39.907+0000 7f7b83fff640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:27:39.909 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:27:39.907+0000 7f7b83fff640 10 monclient: _finish_command 1 = system:0 2024-03-25T18:27:40.143 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:27:40.141+0000 7f7ba1bcf640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:27:40.143 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:27:40.141+0000 7f7ba1bcf640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:27:40.143 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:27:40.141+0000 7f7ba1bcf640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.115:6789/0 2024-03-25T18:27:40.144 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:27:40.141+0000 7f7ba1bcf640 1 -- 172.21.15.1:0/737731788 --> v1:172.21.15.115:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f7b9c13f6c0 con 0x7f7b9c0b4670 2024-03-25T18:27:40.171 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:27:40.169+0000 7f7b83fff640 1 -- 172.21.15.1:0/737731788 <== mon.1 v1:172.21.15.115:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 4092213348) 0x7f7b90050ec0 con 0x7f7b9c0b4670 2024-03-25T18:27:40.171 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:27:40.169+0000 7f7b83fff640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:27:40.171 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:27:40.169+0000 7f7b83fff640 10 monclient: _finish_command 2 = system:0 2024-03-25T18:27:40.171 INFO:teuthology.misc.health.smithi001.stdout: 2024-03-25T18:27:40.172 INFO:teuthology.misc.health.smithi001.stdout:{"epoch":10,"fsid":"c8ea72cf-9e99-4383-9b52-6c23d365fdfc","created":"2024-03-25T18:26:38.856638+0000","modified":"2024-03-25T18:26:44.224208+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T18:26:44.224208+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"b9f34922-3a84-49be-8426-807c1eceaefc","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"42bf306f-55e3-4634-9874-8b4b73ac6644","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"292cb23a-18a9-4207-a7cc-4d5727feaec0","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"45e0e9ab-898d-4673-9590-b4117bc5e84a","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"831a5a09-73ab-4a61-9872-088e76bdbc66","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"88cd903d-3b55-4840-9396-6e3cfe005282","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"89db7355-bc9d-46f4-8ab0-e8d9e744e026","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"cac47bee-0914-4f08-896f-348838882866","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T18:27:40.173 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:27:40.171+0000 7f7ba1bcf640 10 librados: watch_flush enter 2024-03-25T18:27:40.173 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:27:40.171+0000 7f7ba1bcf640 10 librados: watch_flush exit 2024-03-25T18:27:40.173 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:27:40.171+0000 7f7ba1bcf640 20 client.4258.objecter shutdown clearing up homeless session... 2024-03-25T18:27:40.174 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:27:40.171+0000 7f7ba1bcf640 10 client.4258.objecter successfully canceled tick 2024-03-25T18:27:40.174 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:27:40.171+0000 7f7ba1bcf640 1 -- 172.21.15.1:0/737731788 >> v1:172.21.15.115:6800/4237379176 conn(0x7f7b8806e020 legacy=0x7f7b880704e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:27:40.174 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:27:40.171+0000 7f7ba1bcf640 10 monclient: shutdown 2024-03-25T18:27:40.174 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:27:40.172+0000 7f7ba1bcf640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:27:40.174 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:27:40.172+0000 7f7ba1bcf640 1 -- 172.21.15.1:0/737731788 >> v1:172.21.15.115:6789/0 conn(0x7f7b9c0b4670 legacy=0x7f7b9c145d60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:27:40.174 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:27:40.172+0000 7f7ba1bcf640 1 -- 172.21.15.1:0/737731788 shutdown_connections 2024-03-25T18:27:40.174 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:27:40.172+0000 7f7ba1bcf640 1 -- 172.21.15.1:0/737731788 >> 172.21.15.1:0/737731788 conn(0x7f7b9c0617a0 msgr2=0x7f7b9c1407a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:27:40.174 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:27:40.172+0000 7f7ba1bcf640 1 -- 172.21.15.1:0/737731788 shutdown_connections 2024-03-25T18:27:40.174 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:27:40.172+0000 7f7ba1bcf640 1 -- 172.21.15.1:0/737731788 wait complete. 2024-03-25T18:27:40.174 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:27:40.172+0000 7f7ba1bcf640 1 librados: shutdown 2024-03-25T18:27:40.186 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T18:27:46.993 DEBUG:teuthology.orchestra.run.smithi001:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T18:27:47.089 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:27:47.087+0000 7fb8269e2640 10 monclient: get_monmap_and_config 2024-03-25T18:27:47.089 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:27:47.087+0000 7fb8269e2640 10 monclient: build_initial_monmap 2024-03-25T18:27:47.089 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:27:47.087+0000 7fb8269e2640 10 monclient: monmap: 2024-03-25T18:27:47.089 INFO:teuthology.misc.health.smithi001.stderr:epoch 0 2024-03-25T18:27:47.089 INFO:teuthology.misc.health.smithi001.stderr:fsid c8ea72cf-9e99-4383-9b52-6c23d365fdfc 2024-03-25T18:27:47.089 INFO:teuthology.misc.health.smithi001.stderr:last_changed 2024-03-25T18:27:47.089184+0000 2024-03-25T18:27:47.089 INFO:teuthology.misc.health.smithi001.stderr:created 2024-03-25T18:27:47.089184+0000 2024-03-25T18:27:47.090 INFO:teuthology.misc.health.smithi001.stderr:min_mon_release 0 (unknown) 2024-03-25T18:27:47.090 INFO:teuthology.misc.health.smithi001.stderr:election_strategy: 3 2024-03-25T18:27:47.090 INFO:teuthology.misc.health.smithi001.stderr:0: v1:172.21.15.1:6789/0 mon.noname-a 2024-03-25T18:27:47.090 INFO:teuthology.misc.health.smithi001.stderr:1: v1:172.21.15.115:6789/0 mon.noname-c 2024-03-25T18:27:47.090 INFO:teuthology.misc.health.smithi001.stderr:2: v1:172.21.15.1:6790/0 mon.noname-b 2024-03-25T18:27:47.090 INFO:teuthology.misc.health.smithi001.stderr: 2024-03-25T18:27:47.091 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:27:47.088+0000 7fb8269e2640 1 Processor -- start 2024-03-25T18:27:47.091 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:27:47.088+0000 7fb8269e2640 1 -- start start 2024-03-25T18:27:47.091 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:27:47.088+0000 7fb8269e2640 10 monclient: init 2024-03-25T18:27:47.091 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:27:47.088+0000 7fb8269e2640 10 monclient: _reopen_session rank -1 2024-03-25T18:27:47.091 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:27:47.088+0000 7fb8269e2640 10 monclient: _add_conns ranks=[0,2,1] 2024-03-25T18:27:47.091 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:27:47.088+0000 7fb8269e2640 10 monclient(hunting): picked mon.noname-a con 0x7fb8200b4e70 addr v1:172.21.15.1:6789/0 2024-03-25T18:27:47.092 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:27:47.088+0000 7fb8269e2640 10 monclient(hunting): picked mon.noname-b con 0x7fb8200b44b0 addr v1:172.21.15.1:6790/0 2024-03-25T18:27:47.092 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:27:47.088+0000 7fb8269e2640 10 monclient(hunting): picked mon.noname-c con 0x7fb8200b1b00 addr v1:172.21.15.115:6789/0 2024-03-25T18:27:47.092 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:27:47.088+0000 7fb8269e2640 1 -- --> v1:172.21.15.1:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb8200a96f0 con 0x7fb8200b4e70 2024-03-25T18:27:47.092 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:27:47.089+0000 7fb8269e2640 1 -- --> v1:172.21.15.115:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb8200ab0e0 con 0x7fb8200b1b00 2024-03-25T18:27:47.092 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:27:47.089+0000 7fb8269e2640 1 -- --> v1:172.21.15.1:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb820145f70 con 0x7fb8200b44b0 2024-03-25T18:27:47.092 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:27:47.089+0000 7fb8269e2640 10 monclient(hunting): _renew_subs 2024-03-25T18:27:47.092 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:27:47.089+0000 7fb8269e2640 10 monclient(hunting): authenticate will time out at 7547.559679s 2024-03-25T18:27:47.092 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:27:47.089+0000 7fb81f7fe640 1 --1- >> v1:172.21.15.1:6790/0 conn(0x7fb8200b44b0 0x7fb820145860 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.1:6790/0 says I am v1:172.21.15.1:35568/0 (socket says 172.21.15.1:35568) 2024-03-25T18:27:47.092 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:27:47.089+0000 7fb81f7fe640 1 -- 172.21.15.1:0/118735663 learned_addr learned my addr 172.21.15.1:0/118735663 (peer_addr_for_me v1:172.21.15.1:0/0) 2024-03-25T18:27:47.115 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:27:47.114+0000 7fb81effd640 1 -- 172.21.15.1:0/118735663 <== mon.1 v1:172.21.15.115:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2185072917 0 0) 0x7fb8200ab0e0 con 0x7fb8200b1b00 2024-03-25T18:27:47.115 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:27:47.114+0000 7fb81effd640 10 monclient(hunting): _negotiate 2024-03-25T18:27:47.115 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:27:47.114+0000 7fb81effd640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:27:47.115 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:27:47.114+0000 7fb81effd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:27:47.115 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:27:47.114+0000 7fb81effd640 10 monclient(hunting): my global_id is 4261 2024-03-25T18:27:47.116 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:27:47.114+0000 7fb81effd640 1 -- 172.21.15.1:0/118735663 --> v1:172.21.15.115:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fb80c003520 con 0x7fb8200b1b00 2024-03-25T18:27:47.119 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:27:47.117+0000 7fb81effd640 1 -- 172.21.15.1:0/118735663 <== mon.2 v1:172.21.15.1:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 952532829 0 0) 0x7fb820145f70 con 0x7fb8200b44b0 2024-03-25T18:27:47.119 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:27:47.117+0000 7fb81effd640 10 monclient(hunting): _negotiate 2024-03-25T18:27:47.119 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:27:47.117+0000 7fb81effd640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:27:47.119 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:27:47.117+0000 7fb81effd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:27:47.119 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:27:47.117+0000 7fb81effd640 10 monclient(hunting): my global_id is 4268 2024-03-25T18:27:47.119 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:27:47.117+0000 7fb81effd640 1 -- 172.21.15.1:0/118735663 --> v1:172.21.15.1:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fb8200ab0e0 con 0x7fb8200b44b0 2024-03-25T18:27:47.129 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:27:47.127+0000 7fb81effd640 1 -- 172.21.15.1:0/118735663 <== mon.1 v1:172.21.15.115:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2209399636 0 0) 0x7fb80c003520 con 0x7fb8200b1b00 2024-03-25T18:27:47.129 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:27:47.127+0000 7fb81effd640 1 -- 172.21.15.1:0/118735663 >> v1:172.21.15.1:6790/0 conn(0x7fb8200b44b0 legacy=0x7fb820145860 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:27:47.129 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:27:47.127+0000 7fb81effd640 1 -- 172.21.15.1:0/118735663 >> v1:172.21.15.1:6789/0 conn(0x7fb8200b4e70 legacy=0x7fb82013f940 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:27:47.129 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:27:47.127+0000 7fb81effd640 10 monclient: _finish_hunting 0 2024-03-25T18:27:47.130 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:27:47.127+0000 7fb81effd640 1 monclient: found mon.noname-c 2024-03-25T18:27:47.130 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:27:47.127+0000 7fb81effd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:27:47.130 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:27:47.127+0000 7fb81effd640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.115:6789/0 2024-03-25T18:27:47.130 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:27:47.127+0000 7fb81effd640 1 -- 172.21.15.1:0/118735663 --> v1:172.21.15.115:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb8200a9b80 con 0x7fb8200b1b00 2024-03-25T18:27:47.130 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:27:47.127+0000 7fb81effd640 10 monclient: _finish_auth 0 2024-03-25T18:27:47.130 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:27:47.127+0000 7fb81effd640 10 monclient: _check_auth_tickets 2024-03-25T18:27:47.130 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:27:47.127+0000 7fb81effd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:27:47.130 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:27:47.127+0000 7fb81effd640 1 -- 172.21.15.1:0/118735663 <== mon.1 v1:172.21.15.115:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1586965025 0 0) 0x7fb818001a10 con 0x7fb8200b1b00 2024-03-25T18:27:47.130 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:27:47.127+0000 7fb81effd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:27:47.130 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:27:47.127+0000 7fb81effd640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-03-25T18:27:47.131 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:27:47.127+0000 7fb81effd640 10 monclient: dump: 2024-03-25T18:27:47.131 INFO:teuthology.misc.health.smithi001.stderr:epoch 1 2024-03-25T18:27:47.131 INFO:teuthology.misc.health.smithi001.stderr:fsid c8ea72cf-9e99-4383-9b52-6c23d365fdfc 2024-03-25T18:27:47.131 INFO:teuthology.misc.health.smithi001.stderr:last_changed 2024-03-25T18:25:50.510519+0000 2024-03-25T18:27:47.131 INFO:teuthology.misc.health.smithi001.stderr:created 2024-03-25T18:25:50.510519+0000 2024-03-25T18:27:47.131 INFO:teuthology.misc.health.smithi001.stderr:min_mon_release 19 (squid) 2024-03-25T18:27:47.131 INFO:teuthology.misc.health.smithi001.stderr:election_strategy: 3 2024-03-25T18:27:47.131 INFO:teuthology.misc.health.smithi001.stderr:0: v1:172.21.15.1:6789/0 mon.a 2024-03-25T18:27:47.131 INFO:teuthology.misc.health.smithi001.stderr:1: v1:172.21.15.115:6789/0 mon.b 2024-03-25T18:27:47.131 INFO:teuthology.misc.health.smithi001.stderr:2: v1:172.21.15.1:6790/0 mon.c 2024-03-25T18:27:47.131 INFO:teuthology.misc.health.smithi001.stderr: 2024-03-25T18:27:47.131 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:27:47.127+0000 7fb8269e2640 5 monclient: authenticate success, global_id 4261 2024-03-25T18:27:47.131 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:27:47.127+0000 7fb8269e2640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T18:27:47.132 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:27:47.130+0000 7fb81effd640 1 -- 172.21.15.1:0/118735663 <== mon.1 v1:172.21.15.115:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fb818001b70 con 0x7fb8200b1b00 2024-03-25T18:27:47.132 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:27:47.130+0000 7fb81effd640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:27:47.132 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:27:47.130+0000 7fb81effd640 1 -- 172.21.15.1:0/118735663 <== mon.1 v1:172.21.15.115:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1586965025 0 0) 0x7fb818001f80 con 0x7fb8200b1b00 2024-03-25T18:27:47.132 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:27:47.130+0000 7fb8269e2640 10 monclient: get_monmap_and_config success 2024-03-25T18:27:47.132 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:27:47.130+0000 7fb8269e2640 4 set_mon_vals no callback set 2024-03-25T18:27:47.133 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:27:47.130+0000 7fb81effd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:27:47.133 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:27:47.130+0000 7fb81effd640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-25T18:27:47.133 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:27:47.130+0000 7fb81effd640 10 monclient: dump: 2024-03-25T18:27:47.133 INFO:teuthology.misc.health.smithi001.stderr:epoch 1 2024-03-25T18:27:47.133 INFO:teuthology.misc.health.smithi001.stderr:fsid c8ea72cf-9e99-4383-9b52-6c23d365fdfc 2024-03-25T18:27:47.133 INFO:teuthology.misc.health.smithi001.stderr:last_changed 2024-03-25T18:25:50.510519+0000 2024-03-25T18:27:47.133 INFO:teuthology.misc.health.smithi001.stderr:created 2024-03-25T18:25:50.510519+0000 2024-03-25T18:27:47.133 INFO:teuthology.misc.health.smithi001.stderr:min_mon_release 19 (squid) 2024-03-25T18:27:47.133 INFO:teuthology.misc.health.smithi001.stderr:election_strategy: 3 2024-03-25T18:27:47.133 INFO:teuthology.misc.health.smithi001.stderr:0: v1:172.21.15.1:6789/0 mon.a 2024-03-25T18:27:47.133 INFO:teuthology.misc.health.smithi001.stderr:1: v1:172.21.15.115:6789/0 mon.b 2024-03-25T18:27:47.133 INFO:teuthology.misc.health.smithi001.stderr:2: v1:172.21.15.1:6790/0 mon.c 2024-03-25T18:27:47.133 INFO:teuthology.misc.health.smithi001.stderr: 2024-03-25T18:27:47.135 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:27:47.131+0000 7fb8269e2640 10 monclient: shutdown 2024-03-25T18:27:47.135 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:27:47.131+0000 7fb8269e2640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:27:47.136 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:27:47.131+0000 7fb8269e2640 1 -- 172.21.15.1:0/118735663 >> v1:172.21.15.115:6789/0 conn(0x7fb8200b1b00 legacy=0x7fb82014dc00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:27:47.136 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:27:47.131+0000 7fb8269e2640 1 -- 172.21.15.1:0/118735663 shutdown_connections 2024-03-25T18:27:47.136 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:27:47.131+0000 7fb8269e2640 1 -- 172.21.15.1:0/118735663 >> 172.21.15.1:0/118735663 conn(0x7fb8200617a0 msgr2=0x7fb820061bb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:27:47.136 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:27:47.131+0000 7fb8269e2640 1 -- 172.21.15.1:0/118735663 shutdown_connections 2024-03-25T18:27:47.136 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:27:47.131+0000 7fb8269e2640 1 -- 172.21.15.1:0/118735663 wait complete. 2024-03-25T18:27:47.136 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:27:47.132+0000 7fb8269e2640 10 monclient: build_initial_monmap 2024-03-25T18:27:47.136 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:27:47.132+0000 7fb8269e2640 10 monclient: monmap: 2024-03-25T18:27:47.136 INFO:teuthology.misc.health.smithi001.stderr:epoch 0 2024-03-25T18:27:47.136 INFO:teuthology.misc.health.smithi001.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T18:27:47.136 INFO:teuthology.misc.health.smithi001.stderr:last_changed 0.000000 2024-03-25T18:27:47.136 INFO:teuthology.misc.health.smithi001.stderr:created 0.000000 2024-03-25T18:27:47.136 INFO:teuthology.misc.health.smithi001.stderr:min_mon_release 0 (unknown) 2024-03-25T18:27:47.136 INFO:teuthology.misc.health.smithi001.stderr:election_strategy: 1 2024-03-25T18:27:47.136 INFO:teuthology.misc.health.smithi001.stderr:0: v1:172.21.15.1:6789/0 mon.noname-a 2024-03-25T18:27:47.137 INFO:teuthology.misc.health.smithi001.stderr:1: v1:172.21.15.115:6789/0 mon.noname-b 2024-03-25T18:27:47.137 INFO:teuthology.misc.health.smithi001.stderr:2: v1:172.21.15.1:6790/0 mon.noname-c 2024-03-25T18:27:47.137 INFO:teuthology.misc.health.smithi001.stderr: 2024-03-25T18:27:47.137 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:27:47.132+0000 7fb8269e2640 1 librados: starting msgr at 2024-03-25T18:27:47.137 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:27:47.132+0000 7fb8269e2640 1 librados: starting objecter 2024-03-25T18:27:47.137 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:27:47.132+0000 7fb8269e2640 1 Processor -- start 2024-03-25T18:27:47.137 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:27:47.133+0000 7fb8269e2640 1 -- start start 2024-03-25T18:27:47.137 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:27:47.133+0000 7fb8269e2640 1 librados: setting wanted keys 2024-03-25T18:27:47.137 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:27:47.133+0000 7fb8269e2640 1 librados: calling monclient init 2024-03-25T18:27:47.137 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:27:47.133+0000 7fb8269e2640 10 monclient: init 2024-03-25T18:27:47.137 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:27:47.133+0000 7fb8269e2640 10 monclient: _reopen_session rank -1 2024-03-25T18:27:47.137 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:27:47.133+0000 7fb8269e2640 10 monclient: _add_conns ranks=[1,0,2] 2024-03-25T18:27:47.137 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:27:47.133+0000 7fb8269e2640 10 monclient(hunting): picked mon.noname-b con 0x7fb8200b1b00 addr v1:172.21.15.115:6789/0 2024-03-25T18:27:47.137 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:27:47.133+0000 7fb8269e2640 10 monclient(hunting): picked mon.noname-a con 0x7fb8200b44b0 addr v1:172.21.15.1:6789/0 2024-03-25T18:27:47.137 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:27:47.133+0000 7fb8269e2640 10 monclient(hunting): picked mon.noname-c con 0x7fb8200b4e70 addr v1:172.21.15.1:6790/0 2024-03-25T18:27:47.138 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:27:47.133+0000 7fb8269e2640 1 -- --> v1:172.21.15.1:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb82013c450 con 0x7fb8200b44b0 2024-03-25T18:27:47.138 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:27:47.133+0000 7fb8269e2640 1 -- --> v1:172.21.15.115:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb82013c6a0 con 0x7fb8200b1b00 2024-03-25T18:27:47.138 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:27:47.133+0000 7fb8269e2640 1 -- --> v1:172.21.15.1:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb82013c980 con 0x7fb8200b4e70 2024-03-25T18:27:47.138 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:27:47.133+0000 7fb8269e2640 10 monclient(hunting): _renew_subs 2024-03-25T18:27:47.138 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:27:47.133+0000 7fb8269e2640 10 monclient(hunting): authenticate will time out at 7547.604426s 2024-03-25T18:27:47.140 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:27:47.139+0000 7fb81ffff640 1 --1- >> v1:172.21.15.115:6789/0 conn(0x7fb8200b1b00 0x7fb82013a520 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.115:6789/0 says I am v1:172.21.15.1:43752/0 (socket says 172.21.15.1:43752) 2024-03-25T18:27:47.140 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:27:47.139+0000 7fb81ffff640 1 -- 172.21.15.1:0/780661744 learned_addr learned my addr 172.21.15.1:0/780661744 (peer_addr_for_me v1:172.21.15.1:0/0) 2024-03-25T18:27:47.144 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:27:47.143+0000 7fb81cff9640 10 client.?.objecter ms_handle_connect 0x7fb8200b44b0 2024-03-25T18:27:47.145 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:27:47.143+0000 7fb81cff9640 10 client.?.objecter resend_mon_ops 2024-03-25T18:27:47.146 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:27:47.144+0000 7fb81cff9640 10 client.?.objecter ms_handle_connect 0x7fb8200b1b00 2024-03-25T18:27:47.146 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:27:47.144+0000 7fb81cff9640 10 client.?.objecter resend_mon_ops 2024-03-25T18:27:47.149 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:27:47.148+0000 7fb81cff9640 10 client.?.objecter ms_handle_connect 0x7fb8200b4e70 2024-03-25T18:27:47.149 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:27:47.148+0000 7fb81cff9640 10 client.?.objecter resend_mon_ops 2024-03-25T18:27:47.165 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:27:47.163+0000 7fb81cff9640 1 -- 172.21.15.1:0/780661744 <== mon.0 v1:172.21.15.1:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1662161807 0 0) 0x7fb82013c450 con 0x7fb8200b44b0 2024-03-25T18:27:47.165 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:27:47.163+0000 7fb81cff9640 10 monclient(hunting): _negotiate 2024-03-25T18:27:47.165 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:27:47.163+0000 7fb81cff9640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:27:47.165 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:27:47.163+0000 7fb81cff9640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:27:47.165 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:27:47.163+0000 7fb81cff9640 10 monclient(hunting): my global_id is 4269 2024-03-25T18:27:47.165 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:27:47.163+0000 7fb81cff9640 1 -- 172.21.15.1:0/780661744 --> v1:172.21.15.1:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fb800004eb0 con 0x7fb8200b44b0 2024-03-25T18:27:47.167 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:27:47.165+0000 7fb81cff9640 1 -- 172.21.15.1:0/780661744 <== mon.1 v1:172.21.15.115:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3184178780 0 0) 0x7fb82013c6a0 con 0x7fb8200b1b00 2024-03-25T18:27:47.167 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:27:47.165+0000 7fb81cff9640 10 monclient(hunting): _negotiate 2024-03-25T18:27:47.167 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:27:47.165+0000 7fb81cff9640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:27:47.167 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:27:47.165+0000 7fb81cff9640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:27:47.167 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:27:47.165+0000 7fb81cff9640 10 monclient(hunting): my global_id is 4264 2024-03-25T18:27:47.168 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:27:47.165+0000 7fb81cff9640 1 -- 172.21.15.1:0/780661744 --> v1:172.21.15.115:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fb82013c450 con 0x7fb8200b1b00 2024-03-25T18:27:47.177 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:27:47.176+0000 7fb81cff9640 1 -- 172.21.15.1:0/780661744 <== mon.2 v1:172.21.15.1:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3147702423 0 0) 0x7fb82013c980 con 0x7fb8200b4e70 2024-03-25T18:27:47.178 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:27:47.176+0000 7fb81cff9640 10 monclient(hunting): _negotiate 2024-03-25T18:27:47.178 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:27:47.176+0000 7fb81cff9640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:27:47.178 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:27:47.176+0000 7fb81cff9640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:27:47.178 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:27:47.176+0000 7fb81cff9640 10 monclient(hunting): my global_id is 4271 2024-03-25T18:27:47.178 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:27:47.176+0000 7fb81cff9640 1 -- 172.21.15.1:0/780661744 --> v1:172.21.15.1:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fb82013c6a0 con 0x7fb8200b4e70 2024-03-25T18:27:47.180 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:27:47.178+0000 7fb81cff9640 1 -- 172.21.15.1:0/780661744 <== mon.1 v1:172.21.15.115:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 143214327 0 0) 0x7fb82013c450 con 0x7fb8200b1b00 2024-03-25T18:27:47.180 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:27:47.178+0000 7fb81cff9640 1 -- 172.21.15.1:0/780661744 --> v1:172.21.15.115:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fb82013c980 con 0x7fb8200b1b00 2024-03-25T18:27:47.180 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:27:47.178+0000 7fb81cff9640 1 -- 172.21.15.1:0/780661744 <== mon.1 v1:172.21.15.115:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1586965025 0 0) 0x7fb810001910 con 0x7fb8200b1b00 2024-03-25T18:27:47.180 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:27:47.178+0000 7fb81cff9640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:27:47.180 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:27:47.178+0000 7fb81cff9640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-03-25T18:27:47.180 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:27:47.178+0000 7fb81cff9640 10 monclient(hunting): dump: 2024-03-25T18:27:47.180 INFO:teuthology.misc.health.smithi001.stderr:epoch 1 2024-03-25T18:27:47.181 INFO:teuthology.misc.health.smithi001.stderr:fsid c8ea72cf-9e99-4383-9b52-6c23d365fdfc 2024-03-25T18:27:47.181 INFO:teuthology.misc.health.smithi001.stderr:last_changed 2024-03-25T18:25:50.510519+0000 2024-03-25T18:27:47.181 INFO:teuthology.misc.health.smithi001.stderr:created 2024-03-25T18:25:50.510519+0000 2024-03-25T18:27:47.181 INFO:teuthology.misc.health.smithi001.stderr:min_mon_release 19 (squid) 2024-03-25T18:27:47.181 INFO:teuthology.misc.health.smithi001.stderr:election_strategy: 3 2024-03-25T18:27:47.181 INFO:teuthology.misc.health.smithi001.stderr:0: v1:172.21.15.1:6789/0 mon.a 2024-03-25T18:27:47.181 INFO:teuthology.misc.health.smithi001.stderr:1: v1:172.21.15.115:6789/0 mon.b 2024-03-25T18:27:47.181 INFO:teuthology.misc.health.smithi001.stderr:2: v1:172.21.15.1:6790/0 mon.c 2024-03-25T18:27:47.181 INFO:teuthology.misc.health.smithi001.stderr: 2024-03-25T18:27:47.181 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:27:47.178+0000 7fb81cff9640 10 monclient(hunting): _finish_auth 0 2024-03-25T18:27:47.184 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:27:47.182+0000 7fb81cff9640 1 -- 172.21.15.1:0/780661744 <== mon.0 v1:172.21.15.1:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 983424431 0 0) 0x7fb800004eb0 con 0x7fb8200b44b0 2024-03-25T18:27:47.184 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:27:47.182+0000 7fb81cff9640 1 -- 172.21.15.1:0/780661744 --> v1:172.21.15.1:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fb82013c450 con 0x7fb8200b44b0 2024-03-25T18:27:47.184 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:27:47.182+0000 7fb81cff9640 1 -- 172.21.15.1:0/780661744 <== mon.0 v1:172.21.15.1:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1586965025 0 0) 0x7fb808001930 con 0x7fb8200b44b0 2024-03-25T18:27:47.184 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:27:47.182+0000 7fb81cff9640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:27:47.184 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:27:47.182+0000 7fb81cff9640 10 monclient(hunting): got monmap 1 from mon.a (according to old e1) 2024-03-25T18:27:47.184 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:27:47.182+0000 7fb81cff9640 10 monclient(hunting): dump: 2024-03-25T18:27:47.184 INFO:teuthology.misc.health.smithi001.stderr:epoch 1 2024-03-25T18:27:47.184 INFO:teuthology.misc.health.smithi001.stderr:fsid c8ea72cf-9e99-4383-9b52-6c23d365fdfc 2024-03-25T18:27:47.184 INFO:teuthology.misc.health.smithi001.stderr:last_changed 2024-03-25T18:25:50.510519+0000 2024-03-25T18:27:47.184 INFO:teuthology.misc.health.smithi001.stderr:created 2024-03-25T18:25:50.510519+0000 2024-03-25T18:27:47.184 INFO:teuthology.misc.health.smithi001.stderr:min_mon_release 19 (squid) 2024-03-25T18:27:47.184 INFO:teuthology.misc.health.smithi001.stderr:election_strategy: 3 2024-03-25T18:27:47.184 INFO:teuthology.misc.health.smithi001.stderr:0: v1:172.21.15.1:6789/0 mon.a 2024-03-25T18:27:47.184 INFO:teuthology.misc.health.smithi001.stderr:1: v1:172.21.15.115:6789/0 mon.b 2024-03-25T18:27:47.185 INFO:teuthology.misc.health.smithi001.stderr:2: v1:172.21.15.1:6790/0 mon.c 2024-03-25T18:27:47.185 INFO:teuthology.misc.health.smithi001.stderr: 2024-03-25T18:27:47.188 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:27:47.187+0000 7fb81cff9640 1 -- 172.21.15.1:0/780661744 <== mon.2 v1:172.21.15.1:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1370244479 0 0) 0x7fb82013c6a0 con 0x7fb8200b4e70 2024-03-25T18:27:47.189 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:27:47.187+0000 7fb81cff9640 1 -- 172.21.15.1:0/780661744 --> v1:172.21.15.1:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fb800004eb0 con 0x7fb8200b4e70 2024-03-25T18:27:47.189 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:27:47.187+0000 7fb81cff9640 1 -- 172.21.15.1:0/780661744 <== mon.2 v1:172.21.15.1:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1586965025 0 0) 0x7fb818001e80 con 0x7fb8200b4e70 2024-03-25T18:27:47.189 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:27:47.187+0000 7fb81cff9640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:27:47.189 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:27:47.187+0000 7fb81cff9640 10 monclient(hunting): got monmap 1 from mon.c (according to old e1) 2024-03-25T18:27:47.189 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:27:47.187+0000 7fb81cff9640 10 monclient(hunting): dump: 2024-03-25T18:27:47.189 INFO:teuthology.misc.health.smithi001.stderr:epoch 1 2024-03-25T18:27:47.189 INFO:teuthology.misc.health.smithi001.stderr:fsid c8ea72cf-9e99-4383-9b52-6c23d365fdfc 2024-03-25T18:27:47.189 INFO:teuthology.misc.health.smithi001.stderr:last_changed 2024-03-25T18:25:50.510519+0000 2024-03-25T18:27:47.189 INFO:teuthology.misc.health.smithi001.stderr:created 2024-03-25T18:25:50.510519+0000 2024-03-25T18:27:47.189 INFO:teuthology.misc.health.smithi001.stderr:min_mon_release 19 (squid) 2024-03-25T18:27:47.189 INFO:teuthology.misc.health.smithi001.stderr:election_strategy: 3 2024-03-25T18:27:47.189 INFO:teuthology.misc.health.smithi001.stderr:0: v1:172.21.15.1:6789/0 mon.a 2024-03-25T18:27:47.189 INFO:teuthology.misc.health.smithi001.stderr:1: v1:172.21.15.115:6789/0 mon.b 2024-03-25T18:27:47.189 INFO:teuthology.misc.health.smithi001.stderr:2: v1:172.21.15.1:6790/0 mon.c 2024-03-25T18:27:47.189 INFO:teuthology.misc.health.smithi001.stderr: 2024-03-25T18:27:47.199 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:27:47.197+0000 7fb81cff9640 1 -- 172.21.15.1:0/780661744 <== mon.2 v1:172.21.15.1:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 1385693132 0 0) 0x7fb800004eb0 con 0x7fb8200b4e70 2024-03-25T18:27:47.199 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:27:47.197+0000 7fb81cff9640 1 -- 172.21.15.1:0/780661744 >> v1:172.21.15.115:6789/0 conn(0x7fb8200b1b00 legacy=0x7fb82013a520 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:27:47.199 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:27:47.197+0000 7fb81cff9640 1 -- 172.21.15.1:0/780661744 >> v1:172.21.15.1:6789/0 conn(0x7fb8200b44b0 legacy=0x7fb82013af10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:27:47.199 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:27:47.197+0000 7fb81cff9640 10 monclient: _finish_hunting 0 2024-03-25T18:27:47.199 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:27:47.197+0000 7fb81cff9640 1 monclient: found mon.c 2024-03-25T18:27:47.199 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:27:47.197+0000 7fb81cff9640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:27:47.199 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:27:47.197+0000 7fb81cff9640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.1:6790/0 2024-03-25T18:27:47.199 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:27:47.197+0000 7fb81cff9640 1 -- 172.21.15.1:0/780661744 --> v1:172.21.15.1:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb8201454c0 con 0x7fb8200b4e70 2024-03-25T18:27:47.199 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:27:47.197+0000 7fb81cff9640 10 monclient: _finish_auth 0 2024-03-25T18:27:47.199 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:27:47.197+0000 7fb81cff9640 10 monclient: _check_auth_tickets 2024-03-25T18:27:47.200 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:27:47.197+0000 7fb81cff9640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:27:47.200 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:27:47.197+0000 7fb8269e2640 5 monclient: authenticate success, global_id 4271 2024-03-25T18:27:47.200 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:27:47.197+0000 7fb8269e2640 10 monclient: _renew_subs 2024-03-25T18:27:47.200 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:27:47.197+0000 7fb8269e2640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.1:6790/0 2024-03-25T18:27:47.200 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:27:47.197+0000 7fb8269e2640 1 -- 172.21.15.1:0/780661744 --> v1:172.21.15.1:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb820139980 con 0x7fb8200b4e70 2024-03-25T18:27:47.200 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:27:47.197+0000 7fb8269e2640 10 client.4271.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T18:27:47.200 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:27:47.197+0000 7fb8269e2640 10 monclient: _renew_subs 2024-03-25T18:27:47.200 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:27:47.197+0000 7fb8269e2640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.1:6790/0 2024-03-25T18:27:47.200 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:27:47.197+0000 7fb8269e2640 1 -- 172.21.15.1:0/780661744 --> v1:172.21.15.1:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7fb820140160 con 0x7fb8200b4e70 2024-03-25T18:27:47.200 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:27:47.197+0000 7fb8269e2640 1 librados: init done 2024-03-25T18:27:47.201 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:27:47.198+0000 7fb8269e2640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T18:27:47.201 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:27:47.198+0000 7fb8269e2640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:27:47.201 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:27:47.198+0000 7fb8269e2640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.1:6790/0 2024-03-25T18:27:47.201 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:27:47.198+0000 7fb8269e2640 1 -- 172.21.15.1:0/780661744 --> v1:172.21.15.1:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb820140410 con 0x7fb8200b4e70 2024-03-25T18:27:47.515 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:27:47.513+0000 7fb81cff9640 1 -- 172.21.15.1:0/780661744 <== mon.2 v1:172.21.15.1:6790/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fb818001fe0 con 0x7fb8200b4e70 2024-03-25T18:27:47.515 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:27:47.513+0000 7fb81cff9640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:27:47.515 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:27:47.513+0000 7fb81cff9640 1 -- 172.21.15.1:0/780661744 <== mon.2 v1:172.21.15.1:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1586965025 0 0) 0x7fb818001560 con 0x7fb8200b4e70 2024-03-25T18:27:47.515 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:27:47.513+0000 7fb81cff9640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:27:47.515 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:27:47.513+0000 7fb81cff9640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-25T18:27:47.515 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:27:47.513+0000 7fb81cff9640 10 monclient: dump: 2024-03-25T18:27:47.515 INFO:teuthology.misc.health.smithi001.stderr:epoch 1 2024-03-25T18:27:47.515 INFO:teuthology.misc.health.smithi001.stderr:fsid c8ea72cf-9e99-4383-9b52-6c23d365fdfc 2024-03-25T18:27:47.515 INFO:teuthology.misc.health.smithi001.stderr:last_changed 2024-03-25T18:25:50.510519+0000 2024-03-25T18:27:47.515 INFO:teuthology.misc.health.smithi001.stderr:created 2024-03-25T18:25:50.510519+0000 2024-03-25T18:27:47.516 INFO:teuthology.misc.health.smithi001.stderr:min_mon_release 19 (squid) 2024-03-25T18:27:47.516 INFO:teuthology.misc.health.smithi001.stderr:election_strategy: 3 2024-03-25T18:27:47.516 INFO:teuthology.misc.health.smithi001.stderr:0: v1:172.21.15.1:6789/0 mon.a 2024-03-25T18:27:47.516 INFO:teuthology.misc.health.smithi001.stderr:1: v1:172.21.15.115:6789/0 mon.b 2024-03-25T18:27:47.516 INFO:teuthology.misc.health.smithi001.stderr:2: v1:172.21.15.1:6790/0 mon.c 2024-03-25T18:27:47.516 INFO:teuthology.misc.health.smithi001.stderr: 2024-03-25T18:27:47.516 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:27:47.513+0000 7fb81e7fc640 4 set_mon_vals no callback set 2024-03-25T18:27:47.517 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:27:47.514+0000 7fb81cff9640 1 -- 172.21.15.1:0/780661744 <== mon.2 v1:172.21.15.1:6790/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 21787975 0 0) 0x7fb818019ad0 con 0x7fb8200b4e70 2024-03-25T18:27:47.517 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:27:47.515+0000 7fb81ffff640 10 monclient: get_auth_request con 0x7fb80006e110 auth_method 0 2024-03-25T18:27:47.517 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:27:47.515+0000 7fb81cff9640 10 client.4271.objecter ms_handle_connect 0x7fb80006e110 2024-03-25T18:27:47.524 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:27:47.522+0000 7fb81cff9640 1 -- 172.21.15.1:0/780661744 <== mon.2 v1:172.21.15.1:6790/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 996254230 0 0) 0x7fb8180536b0 con 0x7fb8200b4e70 2024-03-25T18:27:47.524 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:27:47.522+0000 7fb81cff9640 10 client.4271.objecter ms_dispatch 0x7fb820000b90 osd_map(10..10 src has 1..10) v4 2024-03-25T18:27:47.524 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:27:47.522+0000 7fb81cff9640 3 client.4271.objecter handle_osd_map got epochs [10,10] > 0 2024-03-25T18:27:47.524 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:27:47.522+0000 7fb81cff9640 3 client.4271.objecter handle_osd_map decoding full epoch 10 2024-03-25T18:27:47.524 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:27:47.522+0000 7fb81cff9640 20 client.4271.objecter dump_active .. 0 homeless 2024-03-25T18:27:47.524 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:27:47.522+0000 7fb81cff9640 1 -- 172.21.15.1:0/780661744 <== mon.2 v1:172.21.15.1:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7fb81804eff0 con 0x7fb8200b4e70 2024-03-25T18:27:47.525 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:27:47.522+0000 7fb81cff9640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:27:47.525 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:27:47.522+0000 7fb81cff9640 10 monclient: _finish_command 1 = system:0 2024-03-25T18:27:47.776 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:27:47.774+0000 7fb8269e2640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:27:47.776 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:27:47.774+0000 7fb8269e2640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:27:47.776 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:27:47.774+0000 7fb8269e2640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.1:6790/0 2024-03-25T18:27:47.776 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:27:47.774+0000 7fb8269e2640 1 -- 172.21.15.1:0/780661744 --> v1:172.21.15.1:6790/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fb82013ac30 con 0x7fb8200b4e70 2024-03-25T18:27:47.800 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:27:47.798+0000 7fb81cff9640 1 -- 172.21.15.1:0/780661744 <== mon.2 v1:172.21.15.1:6790/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 4092213348) 0x7fb82013ac30 con 0x7fb8200b4e70 2024-03-25T18:27:47.800 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:27:47.798+0000 7fb81cff9640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:27:47.800 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:27:47.798+0000 7fb81cff9640 10 monclient: _finish_command 2 = system:0 2024-03-25T18:27:47.800 INFO:teuthology.misc.health.smithi001.stdout: 2024-03-25T18:27:47.800 INFO:teuthology.misc.health.smithi001.stdout:{"epoch":10,"fsid":"c8ea72cf-9e99-4383-9b52-6c23d365fdfc","created":"2024-03-25T18:26:38.856638+0000","modified":"2024-03-25T18:26:44.224208+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T18:26:44.224208+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"b9f34922-3a84-49be-8426-807c1eceaefc","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"42bf306f-55e3-4634-9874-8b4b73ac6644","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"292cb23a-18a9-4207-a7cc-4d5727feaec0","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"45e0e9ab-898d-4673-9590-b4117bc5e84a","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"831a5a09-73ab-4a61-9872-088e76bdbc66","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"88cd903d-3b55-4840-9396-6e3cfe005282","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"89db7355-bc9d-46f4-8ab0-e8d9e744e026","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"cac47bee-0914-4f08-896f-348838882866","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T18:27:47.803 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:27:47.801+0000 7fb8269e2640 10 librados: watch_flush enter 2024-03-25T18:27:47.804 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:27:47.801+0000 7fb8269e2640 10 librados: watch_flush exit 2024-03-25T18:27:47.804 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:27:47.801+0000 7fb8269e2640 20 client.4271.objecter shutdown clearing up homeless session... 2024-03-25T18:27:47.804 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:27:47.801+0000 7fb8269e2640 10 client.4271.objecter successfully canceled tick 2024-03-25T18:27:47.804 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:27:47.801+0000 7fb8269e2640 1 -- 172.21.15.1:0/780661744 >> v1:172.21.15.115:6800/4237379176 conn(0x7fb80006e110 legacy=0x7fb8000705d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:27:47.804 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:27:47.801+0000 7fb8269e2640 10 monclient: shutdown 2024-03-25T18:27:47.804 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:27:47.801+0000 7fb8269e2640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:27:47.804 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:27:47.801+0000 7fb8269e2640 1 -- 172.21.15.1:0/780661744 >> v1:172.21.15.1:6790/0 conn(0x7fb8200b4e70 legacy=0x7fb8201e25f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:27:47.804 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:27:47.801+0000 7fb8269e2640 1 -- 172.21.15.1:0/780661744 shutdown_connections 2024-03-25T18:27:47.804 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:27:47.801+0000 7fb8269e2640 1 -- 172.21.15.1:0/780661744 >> 172.21.15.1:0/780661744 conn(0x7fb8200617a0 msgr2=0x7fb8200b2710 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:27:47.804 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:27:47.801+0000 7fb8269e2640 1 -- 172.21.15.1:0/780661744 shutdown_connections 2024-03-25T18:27:47.804 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:27:47.801+0000 7fb8269e2640 1 -- 172.21.15.1:0/780661744 wait complete. 2024-03-25T18:27:47.804 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:27:47.801+0000 7fb8269e2640 1 librados: shutdown 2024-03-25T18:27:47.816 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T18:27:54.622 DEBUG:teuthology.orchestra.run.smithi001:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T18:27:54.735 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:27:54.733+0000 7f0f7f7f6640 10 monclient: get_monmap_and_config 2024-03-25T18:27:54.735 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:27:54.733+0000 7f0f7f7f6640 10 monclient: build_initial_monmap 2024-03-25T18:27:54.735 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:27:54.733+0000 7f0f7f7f6640 10 monclient: monmap: 2024-03-25T18:27:54.736 INFO:teuthology.misc.health.smithi001.stderr:epoch 0 2024-03-25T18:27:54.736 INFO:teuthology.misc.health.smithi001.stderr:fsid c8ea72cf-9e99-4383-9b52-6c23d365fdfc 2024-03-25T18:27:54.736 INFO:teuthology.misc.health.smithi001.stderr:last_changed 2024-03-25T18:27:54.735518+0000 2024-03-25T18:27:54.736 INFO:teuthology.misc.health.smithi001.stderr:created 2024-03-25T18:27:54.735518+0000 2024-03-25T18:27:54.736 INFO:teuthology.misc.health.smithi001.stderr:min_mon_release 0 (unknown) 2024-03-25T18:27:54.736 INFO:teuthology.misc.health.smithi001.stderr:election_strategy: 3 2024-03-25T18:27:54.736 INFO:teuthology.misc.health.smithi001.stderr:0: v1:172.21.15.1:6789/0 mon.noname-a 2024-03-25T18:27:54.736 INFO:teuthology.misc.health.smithi001.stderr:1: v1:172.21.15.115:6789/0 mon.noname-c 2024-03-25T18:27:54.736 INFO:teuthology.misc.health.smithi001.stderr:2: v1:172.21.15.1:6790/0 mon.noname-b 2024-03-25T18:27:54.736 INFO:teuthology.misc.health.smithi001.stderr: 2024-03-25T18:27:54.737 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:27:54.734+0000 7f0f7f7f6640 1 Processor -- start 2024-03-25T18:27:54.737 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:27:54.734+0000 7f0f7f7f6640 1 -- start start 2024-03-25T18:27:54.737 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:27:54.734+0000 7f0f7f7f6640 10 monclient: init 2024-03-25T18:27:54.737 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:27:54.735+0000 7f0f7f7f6640 10 monclient: _reopen_session rank -1 2024-03-25T18:27:54.737 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:27:54.735+0000 7f0f7f7f6640 10 monclient: _add_conns ranks=[2,1,0] 2024-03-25T18:27:54.737 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:27:54.735+0000 7f0f7f7f6640 10 monclient(hunting): picked mon.noname-b con 0x7f0f780b4e70 addr v1:172.21.15.1:6790/0 2024-03-25T18:27:54.737 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:27:54.735+0000 7f0f7f7f6640 10 monclient(hunting): picked mon.noname-c con 0x7f0f780b44b0 addr v1:172.21.15.115:6789/0 2024-03-25T18:27:54.738 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:27:54.735+0000 7f0f7f7f6640 10 monclient(hunting): picked mon.noname-a con 0x7f0f780b1b00 addr v1:172.21.15.1:6789/0 2024-03-25T18:27:54.738 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:27:54.735+0000 7f0f7f7f6640 1 -- --> v1:172.21.15.1:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0f780a96f0 con 0x7f0f780b1b00 2024-03-25T18:27:54.738 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:27:54.735+0000 7f0f7f7f6640 1 -- --> v1:172.21.15.115:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0f780ab0e0 con 0x7f0f780b44b0 2024-03-25T18:27:54.738 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:27:54.735+0000 7f0f7f7f6640 1 -- --> v1:172.21.15.1:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0f78145f70 con 0x7f0f780b4e70 2024-03-25T18:27:54.738 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:27:54.735+0000 7f0f7f7f6640 10 monclient(hunting): _renew_subs 2024-03-25T18:27:54.738 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:27:54.735+0000 7f0f7f7f6640 10 monclient(hunting): authenticate will time out at 7555.206386s 2024-03-25T18:27:54.749 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:27:54.748+0000 7f0f7cd91640 1 --1- >> v1:172.21.15.115:6789/0 conn(0x7f0f780b44b0 0x7f0f78145860 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.115:6789/0 says I am v1:172.21.15.1:35656/0 (socket says 172.21.15.1:35656) 2024-03-25T18:27:54.749 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:27:54.748+0000 7f0f7cd91640 1 -- 172.21.15.1:0/313612283 learned_addr learned my addr 172.21.15.1:0/313612283 (peer_addr_for_me v1:172.21.15.1:0/0) 2024-03-25T18:27:54.765 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:27:54.764+0000 7f0f6ffff640 1 -- 172.21.15.1:0/313612283 <== mon.0 v1:172.21.15.1:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3788177515 0 0) 0x7f0f780a96f0 con 0x7f0f780b1b00 2024-03-25T18:27:54.765 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:27:54.764+0000 7f0f6ffff640 10 monclient(hunting): _negotiate 2024-03-25T18:27:54.765 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:27:54.764+0000 7f0f6ffff640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:27:54.765 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:27:54.764+0000 7f0f6ffff640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:27:54.766 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:27:54.764+0000 7f0f6ffff640 10 monclient(hunting): my global_id is 4272 2024-03-25T18:27:54.766 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:27:54.764+0000 7f0f6ffff640 1 -- 172.21.15.1:0/313612283 --> v1:172.21.15.1:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f0f64003520 con 0x7f0f780b1b00 2024-03-25T18:27:54.767 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:27:54.765+0000 7f0f6ffff640 1 -- 172.21.15.1:0/313612283 <== mon.1 v1:172.21.15.115:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2163111317 0 0) 0x7f0f780ab0e0 con 0x7f0f780b44b0 2024-03-25T18:27:54.767 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:27:54.765+0000 7f0f6ffff640 10 monclient(hunting): _negotiate 2024-03-25T18:27:54.767 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:27:54.765+0000 7f0f6ffff640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:27:54.767 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:27:54.765+0000 7f0f6ffff640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:27:54.767 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:27:54.765+0000 7f0f6ffff640 10 monclient(hunting): my global_id is 4267 2024-03-25T18:27:54.767 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:27:54.765+0000 7f0f6ffff640 1 -- 172.21.15.1:0/313612283 --> v1:172.21.15.115:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f0f780a96f0 con 0x7f0f780b44b0 2024-03-25T18:27:54.773 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:27:54.771+0000 7f0f6ffff640 1 -- 172.21.15.1:0/313612283 <== mon.0 v1:172.21.15.1:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1278894277 0 0) 0x7f0f64003520 con 0x7f0f780b1b00 2024-03-25T18:27:54.773 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:27:54.771+0000 7f0f6ffff640 1 -- 172.21.15.1:0/313612283 >> v1:172.21.15.1:6790/0 conn(0x7f0f780b4e70 legacy=0x7f0f7813f940 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:27:54.773 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:27:54.771+0000 7f0f6ffff640 1 -- 172.21.15.1:0/313612283 >> v1:172.21.15.115:6789/0 conn(0x7f0f780b44b0 legacy=0x7f0f78145860 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:27:54.773 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:27:54.771+0000 7f0f6ffff640 10 monclient: _finish_hunting 0 2024-03-25T18:27:54.773 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:27:54.771+0000 7f0f6ffff640 1 monclient: found mon.noname-a 2024-03-25T18:27:54.773 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:27:54.771+0000 7f0f6ffff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:27:54.773 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:27:54.771+0000 7f0f6ffff640 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.1:6789/0 2024-03-25T18:27:54.773 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:27:54.771+0000 7f0f6ffff640 1 -- 172.21.15.1:0/313612283 --> v1:172.21.15.1:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0f780a9b80 con 0x7f0f780b1b00 2024-03-25T18:27:54.773 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:27:54.771+0000 7f0f6ffff640 10 monclient: _finish_auth 0 2024-03-25T18:27:54.774 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:27:54.771+0000 7f0f6ffff640 10 monclient: _check_auth_tickets 2024-03-25T18:27:54.774 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:27:54.771+0000 7f0f6ffff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:27:54.774 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:27:54.771+0000 7f0f7f7f6640 5 monclient: authenticate success, global_id 4272 2024-03-25T18:27:54.774 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:27:54.771+0000 7f0f7f7f6640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T18:27:54.774 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:27:54.771+0000 7f0f6ffff640 1 -- 172.21.15.1:0/313612283 <== mon.0 v1:172.21.15.1:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1586965025 0 0) 0x7f0f700019d0 con 0x7f0f780b1b00 2024-03-25T18:27:54.774 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:27:54.771+0000 7f0f6ffff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:27:54.774 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:27:54.771+0000 7f0f6ffff640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-03-25T18:27:54.774 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:27:54.771+0000 7f0f6ffff640 10 monclient: dump: 2024-03-25T18:27:54.774 INFO:teuthology.misc.health.smithi001.stderr:epoch 1 2024-03-25T18:27:54.774 INFO:teuthology.misc.health.smithi001.stderr:fsid c8ea72cf-9e99-4383-9b52-6c23d365fdfc 2024-03-25T18:27:54.774 INFO:teuthology.misc.health.smithi001.stderr:last_changed 2024-03-25T18:25:50.510519+0000 2024-03-25T18:27:54.774 INFO:teuthology.misc.health.smithi001.stderr:created 2024-03-25T18:25:50.510519+0000 2024-03-25T18:27:54.774 INFO:teuthology.misc.health.smithi001.stderr:min_mon_release 19 (squid) 2024-03-25T18:27:54.774 INFO:teuthology.misc.health.smithi001.stderr:election_strategy: 3 2024-03-25T18:27:54.775 INFO:teuthology.misc.health.smithi001.stderr:0: v1:172.21.15.1:6789/0 mon.a 2024-03-25T18:27:54.775 INFO:teuthology.misc.health.smithi001.stderr:1: v1:172.21.15.115:6789/0 mon.b 2024-03-25T18:27:54.775 INFO:teuthology.misc.health.smithi001.stderr:2: v1:172.21.15.1:6790/0 mon.c 2024-03-25T18:27:54.775 INFO:teuthology.misc.health.smithi001.stderr: 2024-03-25T18:27:54.775 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:27:54.771+0000 7f0f7f7f6640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T18:27:54.777 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:27:54.776+0000 7f0f6ffff640 1 -- 172.21.15.1:0/313612283 <== mon.0 v1:172.21.15.1:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f0f70001c20 con 0x7f0f780b1b00 2024-03-25T18:27:54.777 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:27:54.776+0000 7f0f6ffff640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:27:54.777 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:27:54.776+0000 7f0f6ffff640 1 -- 172.21.15.1:0/313612283 <== mon.0 v1:172.21.15.1:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1586965025 0 0) 0x7f0f70001fb0 con 0x7f0f780b1b00 2024-03-25T18:27:54.778 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:27:54.776+0000 7f0f7f7f6640 10 monclient: get_monmap_and_config success 2024-03-25T18:27:54.778 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:27:54.776+0000 7f0f7f7f6640 4 set_mon_vals no callback set 2024-03-25T18:27:54.778 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:27:54.776+0000 7f0f6ffff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:27:54.778 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:27:54.776+0000 7f0f6ffff640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-25T18:27:54.778 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:27:54.776+0000 7f0f6ffff640 10 monclient: dump: 2024-03-25T18:27:54.778 INFO:teuthology.misc.health.smithi001.stderr:epoch 1 2024-03-25T18:27:54.778 INFO:teuthology.misc.health.smithi001.stderr:fsid c8ea72cf-9e99-4383-9b52-6c23d365fdfc 2024-03-25T18:27:54.778 INFO:teuthology.misc.health.smithi001.stderr:last_changed 2024-03-25T18:25:50.510519+0000 2024-03-25T18:27:54.778 INFO:teuthology.misc.health.smithi001.stderr:created 2024-03-25T18:25:50.510519+0000 2024-03-25T18:27:54.778 INFO:teuthology.misc.health.smithi001.stderr:min_mon_release 19 (squid) 2024-03-25T18:27:54.778 INFO:teuthology.misc.health.smithi001.stderr:election_strategy: 3 2024-03-25T18:27:54.778 INFO:teuthology.misc.health.smithi001.stderr:0: v1:172.21.15.1:6789/0 mon.a 2024-03-25T18:27:54.778 INFO:teuthology.misc.health.smithi001.stderr:1: v1:172.21.15.115:6789/0 mon.b 2024-03-25T18:27:54.779 INFO:teuthology.misc.health.smithi001.stderr:2: v1:172.21.15.1:6790/0 mon.c 2024-03-25T18:27:54.779 INFO:teuthology.misc.health.smithi001.stderr: 2024-03-25T18:27:54.781 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:27:54.776+0000 7f0f7f7f6640 10 monclient: shutdown 2024-03-25T18:27:54.781 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:27:54.776+0000 7f0f7f7f6640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:27:54.781 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:27:54.776+0000 7f0f7f7f6640 1 -- 172.21.15.1:0/313612283 >> v1:172.21.15.1:6789/0 conn(0x7f0f780b1b00 legacy=0x7f0f7814dc00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:27:54.781 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:27:54.776+0000 7f0f7f7f6640 1 -- 172.21.15.1:0/313612283 shutdown_connections 2024-03-25T18:27:54.781 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:27:54.777+0000 7f0f7f7f6640 1 -- 172.21.15.1:0/313612283 >> 172.21.15.1:0/313612283 conn(0x7f0f780617a0 msgr2=0x7f0f78061bb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:27:54.781 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:27:54.777+0000 7f0f7f7f6640 1 -- 172.21.15.1:0/313612283 shutdown_connections 2024-03-25T18:27:54.781 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:27:54.777+0000 7f0f7f7f6640 1 -- 172.21.15.1:0/313612283 wait complete. 2024-03-25T18:27:54.781 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:27:54.777+0000 7f0f7f7f6640 10 monclient: build_initial_monmap 2024-03-25T18:27:54.781 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:27:54.777+0000 7f0f7f7f6640 10 monclient: monmap: 2024-03-25T18:27:54.781 INFO:teuthology.misc.health.smithi001.stderr:epoch 0 2024-03-25T18:27:54.781 INFO:teuthology.misc.health.smithi001.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T18:27:54.781 INFO:teuthology.misc.health.smithi001.stderr:last_changed 0.000000 2024-03-25T18:27:54.782 INFO:teuthology.misc.health.smithi001.stderr:created 0.000000 2024-03-25T18:27:54.782 INFO:teuthology.misc.health.smithi001.stderr:min_mon_release 0 (unknown) 2024-03-25T18:27:54.782 INFO:teuthology.misc.health.smithi001.stderr:election_strategy: 1 2024-03-25T18:27:54.782 INFO:teuthology.misc.health.smithi001.stderr:0: v1:172.21.15.1:6789/0 mon.noname-a 2024-03-25T18:27:54.782 INFO:teuthology.misc.health.smithi001.stderr:1: v1:172.21.15.115:6789/0 mon.noname-b 2024-03-25T18:27:54.782 INFO:teuthology.misc.health.smithi001.stderr:2: v1:172.21.15.1:6790/0 mon.noname-c 2024-03-25T18:27:54.782 INFO:teuthology.misc.health.smithi001.stderr: 2024-03-25T18:27:54.782 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:27:54.778+0000 7f0f7f7f6640 1 librados: starting msgr at 2024-03-25T18:27:54.782 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:27:54.778+0000 7f0f7f7f6640 1 librados: starting objecter 2024-03-25T18:27:54.782 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:27:54.778+0000 7f0f7f7f6640 1 Processor -- start 2024-03-25T18:27:54.782 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:27:54.778+0000 7f0f7f7f6640 1 -- start start 2024-03-25T18:27:54.782 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:27:54.778+0000 7f0f7f7f6640 1 librados: setting wanted keys 2024-03-25T18:27:54.782 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:27:54.778+0000 7f0f7f7f6640 1 librados: calling monclient init 2024-03-25T18:27:54.783 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:27:54.778+0000 7f0f7f7f6640 10 monclient: init 2024-03-25T18:27:54.783 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:27:54.778+0000 7f0f7f7f6640 10 monclient: _reopen_session rank -1 2024-03-25T18:27:54.783 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:27:54.778+0000 7f0f7f7f6640 10 monclient: _add_conns ranks=[0,1,2] 2024-03-25T18:27:54.783 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:27:54.778+0000 7f0f7f7f6640 10 monclient(hunting): picked mon.noname-a con 0x7f0f780b1b00 addr v1:172.21.15.1:6789/0 2024-03-25T18:27:54.783 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:27:54.778+0000 7f0f7f7f6640 10 monclient(hunting): picked mon.noname-b con 0x7f0f780b44b0 addr v1:172.21.15.115:6789/0 2024-03-25T18:27:54.783 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:27:54.778+0000 7f0f7f7f6640 10 monclient(hunting): picked mon.noname-c con 0x7f0f780b4e70 addr v1:172.21.15.1:6790/0 2024-03-25T18:27:54.783 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:27:54.778+0000 7f0f7f7f6640 1 -- --> v1:172.21.15.1:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0f7813c8b0 con 0x7f0f780b1b00 2024-03-25T18:27:54.783 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:27:54.778+0000 7f0f7f7f6640 1 -- --> v1:172.21.15.115:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0f78140160 con 0x7f0f780b44b0 2024-03-25T18:27:54.783 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:27:54.778+0000 7f0f7f7f6640 1 -- --> v1:172.21.15.1:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0f781d87a0 con 0x7f0f780b4e70 2024-03-25T18:27:54.783 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:27:54.778+0000 7f0f7f7f6640 10 monclient(hunting): _renew_subs 2024-03-25T18:27:54.783 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:27:54.778+0000 7f0f7f7f6640 10 monclient(hunting): authenticate will time out at 7555.249629s 2024-03-25T18:27:54.799 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:27:54.797+0000 7f0f7dd93640 1 --1- >> v1:172.21.15.1:6790/0 conn(0x7f0f780b4e70 0x7f0f781d4b80 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.1:6790/0 says I am v1:172.21.15.1:56166/0 (socket says 172.21.15.1:56166) 2024-03-25T18:27:54.799 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:27:54.797+0000 7f0f7dd93640 1 -- 172.21.15.1:0/1798004735 learned_addr learned my addr 172.21.15.1:0/1798004735 (peer_addr_for_me v1:172.21.15.1:0/0) 2024-03-25T18:27:54.803 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:27:54.802+0000 7f0f6dffb640 10 client.?.objecter ms_handle_connect 0x7f0f780b1b00 2024-03-25T18:27:54.803 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:27:54.802+0000 7f0f6dffb640 10 client.?.objecter resend_mon_ops 2024-03-25T18:27:54.804 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:27:54.803+0000 7f0f6dffb640 10 client.?.objecter ms_handle_connect 0x7f0f780b4e70 2024-03-25T18:27:54.805 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:27:54.803+0000 7f0f6dffb640 10 client.?.objecter resend_mon_ops 2024-03-25T18:27:54.806 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:27:54.805+0000 7f0f6dffb640 10 client.?.objecter ms_handle_connect 0x7f0f780b44b0 2024-03-25T18:27:54.806 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:27:54.805+0000 7f0f6dffb640 10 client.?.objecter resend_mon_ops 2024-03-25T18:27:54.812 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:27:54.811+0000 7f0f6dffb640 1 -- 172.21.15.1:0/1798004735 <== mon.1 v1:172.21.15.115:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3498366666 0 0) 0x7f0f78140160 con 0x7f0f780b44b0 2024-03-25T18:27:54.812 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:27:54.811+0000 7f0f6dffb640 10 monclient(hunting): _negotiate 2024-03-25T18:27:54.812 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:27:54.811+0000 7f0f6dffb640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:27:54.812 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:27:54.811+0000 7f0f6dffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:27:54.813 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:27:54.811+0000 7f0f6dffb640 10 monclient(hunting): my global_id is 4270 2024-03-25T18:27:54.813 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:27:54.811+0000 7f0f6dffb640 1 -- 172.21.15.1:0/1798004735 --> v1:172.21.15.115:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f0f5c004eb0 con 0x7f0f780b44b0 2024-03-25T18:27:54.816 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:27:54.814+0000 7f0f6dffb640 1 -- 172.21.15.1:0/1798004735 <== mon.2 v1:172.21.15.1:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1908630362 0 0) 0x7f0f781d87a0 con 0x7f0f780b4e70 2024-03-25T18:27:54.816 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:27:54.814+0000 7f0f6dffb640 10 monclient(hunting): _negotiate 2024-03-25T18:27:54.816 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:27:54.814+0000 7f0f6dffb640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:27:54.816 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:27:54.814+0000 7f0f6dffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:27:54.816 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:27:54.814+0000 7f0f6dffb640 10 monclient(hunting): my global_id is 4277 2024-03-25T18:27:54.816 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:27:54.814+0000 7f0f6dffb640 1 -- 172.21.15.1:0/1798004735 --> v1:172.21.15.1:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f0f78140160 con 0x7f0f780b4e70 2024-03-25T18:27:54.824 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:27:54.822+0000 7f0f6dffb640 1 -- 172.21.15.1:0/1798004735 <== mon.0 v1:172.21.15.1:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1969647854 0 0) 0x7f0f7813c8b0 con 0x7f0f780b1b00 2024-03-25T18:27:54.824 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:27:54.822+0000 7f0f6dffb640 10 monclient(hunting): _negotiate 2024-03-25T18:27:54.825 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:27:54.822+0000 7f0f6dffb640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:27:54.825 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:27:54.822+0000 7f0f6dffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:27:54.825 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:27:54.822+0000 7f0f6dffb640 10 monclient(hunting): my global_id is 4275 2024-03-25T18:27:54.825 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:27:54.822+0000 7f0f6dffb640 1 -- 172.21.15.1:0/1798004735 --> v1:172.21.15.1:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f0f781d87a0 con 0x7f0f780b1b00 2024-03-25T18:27:54.829 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:27:54.827+0000 7f0f6dffb640 1 -- 172.21.15.1:0/1798004735 <== mon.2 v1:172.21.15.1:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2790408915 0 0) 0x7f0f78140160 con 0x7f0f780b4e70 2024-03-25T18:27:54.829 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:27:54.827+0000 7f0f6dffb640 1 -- 172.21.15.1:0/1798004735 --> v1:172.21.15.1:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f0f7813c8b0 con 0x7f0f780b4e70 2024-03-25T18:27:54.829 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:27:54.827+0000 7f0f6dffb640 1 -- 172.21.15.1:0/1798004735 <== mon.2 v1:172.21.15.1:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1586965025 0 0) 0x7f0f70001f30 con 0x7f0f780b4e70 2024-03-25T18:27:54.829 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:27:54.827+0000 7f0f6dffb640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:27:54.830 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:27:54.827+0000 7f0f6dffb640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-03-25T18:27:54.830 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:27:54.827+0000 7f0f6dffb640 10 monclient(hunting): dump: 2024-03-25T18:27:54.830 INFO:teuthology.misc.health.smithi001.stderr:epoch 1 2024-03-25T18:27:54.830 INFO:teuthology.misc.health.smithi001.stderr:fsid c8ea72cf-9e99-4383-9b52-6c23d365fdfc 2024-03-25T18:27:54.830 INFO:teuthology.misc.health.smithi001.stderr:last_changed 2024-03-25T18:25:50.510519+0000 2024-03-25T18:27:54.830 INFO:teuthology.misc.health.smithi001.stderr:created 2024-03-25T18:25:50.510519+0000 2024-03-25T18:27:54.830 INFO:teuthology.misc.health.smithi001.stderr:min_mon_release 19 (squid) 2024-03-25T18:27:54.830 INFO:teuthology.misc.health.smithi001.stderr:election_strategy: 3 2024-03-25T18:27:54.830 INFO:teuthology.misc.health.smithi001.stderr:0: v1:172.21.15.1:6789/0 mon.a 2024-03-25T18:27:54.830 INFO:teuthology.misc.health.smithi001.stderr:1: v1:172.21.15.115:6789/0 mon.b 2024-03-25T18:27:54.830 INFO:teuthology.misc.health.smithi001.stderr:2: v1:172.21.15.1:6790/0 mon.c 2024-03-25T18:27:54.830 INFO:teuthology.misc.health.smithi001.stderr: 2024-03-25T18:27:54.830 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:27:54.827+0000 7f0f6dffb640 10 monclient(hunting): _finish_auth 0 2024-03-25T18:27:54.835 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:27:54.834+0000 7f0f6dffb640 1 -- 172.21.15.1:0/1798004735 <== mon.0 v1:172.21.15.1:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3912056299 0 0) 0x7f0f781d87a0 con 0x7f0f780b1b00 2024-03-25T18:27:54.835 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:27:54.834+0000 7f0f6dffb640 1 -- 172.21.15.1:0/1798004735 --> v1:172.21.15.1:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f0f78140160 con 0x7f0f780b1b00 2024-03-25T18:27:54.836 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:27:54.834+0000 7f0f6dffb640 1 -- 172.21.15.1:0/1798004735 <== mon.0 v1:172.21.15.1:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1586965025 0 0) 0x7f0f740017d0 con 0x7f0f780b1b00 2024-03-25T18:27:54.836 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:27:54.834+0000 7f0f6dffb640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:27:54.836 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:27:54.834+0000 7f0f6dffb640 10 monclient(hunting): got monmap 1 from mon.a (according to old e1) 2024-03-25T18:27:54.836 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:27:54.834+0000 7f0f6dffb640 10 monclient(hunting): dump: 2024-03-25T18:27:54.836 INFO:teuthology.misc.health.smithi001.stderr:epoch 1 2024-03-25T18:27:54.836 INFO:teuthology.misc.health.smithi001.stderr:fsid c8ea72cf-9e99-4383-9b52-6c23d365fdfc 2024-03-25T18:27:54.836 INFO:teuthology.misc.health.smithi001.stderr:last_changed 2024-03-25T18:25:50.510519+0000 2024-03-25T18:27:54.836 INFO:teuthology.misc.health.smithi001.stderr:created 2024-03-25T18:25:50.510519+0000 2024-03-25T18:27:54.836 INFO:teuthology.misc.health.smithi001.stderr:min_mon_release 19 (squid) 2024-03-25T18:27:54.836 INFO:teuthology.misc.health.smithi001.stderr:election_strategy: 3 2024-03-25T18:27:54.836 INFO:teuthology.misc.health.smithi001.stderr:0: v1:172.21.15.1:6789/0 mon.a 2024-03-25T18:27:54.836 INFO:teuthology.misc.health.smithi001.stderr:1: v1:172.21.15.115:6789/0 mon.b 2024-03-25T18:27:54.836 INFO:teuthology.misc.health.smithi001.stderr:2: v1:172.21.15.1:6790/0 mon.c 2024-03-25T18:27:54.836 INFO:teuthology.misc.health.smithi001.stderr: 2024-03-25T18:27:54.837 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:27:54.834+0000 7f0f6dffb640 1 -- 172.21.15.1:0/1798004735 <== mon.1 v1:172.21.15.115:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2243898269 0 0) 0x7f0f5c004eb0 con 0x7f0f780b44b0 2024-03-25T18:27:54.837 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:27:54.834+0000 7f0f6dffb640 1 -- 172.21.15.1:0/1798004735 --> v1:172.21.15.115:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f0f781d87a0 con 0x7f0f780b44b0 2024-03-25T18:27:54.837 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:27:54.834+0000 7f0f6dffb640 1 -- 172.21.15.1:0/1798004735 <== mon.1 v1:172.21.15.115:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1586965025 0 0) 0x7f0f68001930 con 0x7f0f780b44b0 2024-03-25T18:27:54.837 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:27:54.834+0000 7f0f6dffb640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:27:54.837 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:27:54.834+0000 7f0f6dffb640 10 monclient(hunting): got monmap 1 from mon.b (according to old e1) 2024-03-25T18:27:54.837 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:27:54.834+0000 7f0f6dffb640 10 monclient(hunting): dump: 2024-03-25T18:27:54.837 INFO:teuthology.misc.health.smithi001.stderr:epoch 1 2024-03-25T18:27:54.837 INFO:teuthology.misc.health.smithi001.stderr:fsid c8ea72cf-9e99-4383-9b52-6c23d365fdfc 2024-03-25T18:27:54.837 INFO:teuthology.misc.health.smithi001.stderr:last_changed 2024-03-25T18:25:50.510519+0000 2024-03-25T18:27:54.837 INFO:teuthology.misc.health.smithi001.stderr:created 2024-03-25T18:25:50.510519+0000 2024-03-25T18:27:54.837 INFO:teuthology.misc.health.smithi001.stderr:min_mon_release 19 (squid) 2024-03-25T18:27:54.837 INFO:teuthology.misc.health.smithi001.stderr:election_strategy: 3 2024-03-25T18:27:54.837 INFO:teuthology.misc.health.smithi001.stderr:0: v1:172.21.15.1:6789/0 mon.a 2024-03-25T18:27:54.837 INFO:teuthology.misc.health.smithi001.stderr:1: v1:172.21.15.115:6789/0 mon.b 2024-03-25T18:27:54.837 INFO:teuthology.misc.health.smithi001.stderr:2: v1:172.21.15.1:6790/0 mon.c 2024-03-25T18:27:54.837 INFO:teuthology.misc.health.smithi001.stderr: 2024-03-25T18:27:54.851 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:27:54.849+0000 7f0f6dffb640 1 -- 172.21.15.1:0/1798004735 <== mon.2 v1:172.21.15.1:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 1191351680 0 0) 0x7f0f7813c8b0 con 0x7f0f780b4e70 2024-03-25T18:27:54.852 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:27:54.849+0000 7f0f6dffb640 1 -- 172.21.15.1:0/1798004735 >> v1:172.21.15.115:6789/0 conn(0x7f0f780b44b0 legacy=0x7f0f78141710 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:27:54.852 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:27:54.849+0000 7f0f6dffb640 1 -- 172.21.15.1:0/1798004735 >> v1:172.21.15.1:6789/0 conn(0x7f0f780b1b00 legacy=0x7f0f781d8090 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:27:54.852 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:27:54.849+0000 7f0f6dffb640 10 monclient: _finish_hunting 0 2024-03-25T18:27:54.852 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:27:54.849+0000 7f0f6dffb640 1 monclient: found mon.c 2024-03-25T18:27:54.852 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:27:54.849+0000 7f0f6dffb640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:27:54.852 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:27:54.849+0000 7f0f6dffb640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.1:6790/0 2024-03-25T18:27:54.852 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:27:54.849+0000 7f0f6dffb640 1 -- 172.21.15.1:0/1798004735 --> v1:172.21.15.1:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0f781d8930 con 0x7f0f780b4e70 2024-03-25T18:27:54.852 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:27:54.849+0000 7f0f6dffb640 10 monclient: _finish_auth 0 2024-03-25T18:27:54.852 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:27:54.849+0000 7f0f6dffb640 10 monclient: _check_auth_tickets 2024-03-25T18:27:54.852 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:27:54.849+0000 7f0f6dffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:27:54.852 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:27:54.849+0000 7f0f7f7f6640 5 monclient: authenticate success, global_id 4277 2024-03-25T18:27:54.852 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:27:54.849+0000 7f0f7f7f6640 10 monclient: _renew_subs 2024-03-25T18:27:54.852 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:27:54.849+0000 7f0f7f7f6640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.1:6790/0 2024-03-25T18:27:54.853 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:27:54.849+0000 7f0f7f7f6640 1 -- 172.21.15.1:0/1798004735 --> v1:172.21.15.1:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0f781d4540 con 0x7f0f780b4e70 2024-03-25T18:27:54.853 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:27:54.850+0000 7f0f7f7f6640 10 client.4277.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T18:27:54.853 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:27:54.850+0000 7f0f7f7f6640 10 monclient: _renew_subs 2024-03-25T18:27:54.853 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:27:54.850+0000 7f0f7f7f6640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.1:6790/0 2024-03-25T18:27:54.853 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:27:54.850+0000 7f0f7f7f6640 1 -- 172.21.15.1:0/1798004735 --> v1:172.21.15.1:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f0f78142000 con 0x7f0f780b4e70 2024-03-25T18:27:54.853 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:27:54.850+0000 7f0f7f7f6640 1 librados: init done 2024-03-25T18:27:54.853 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:27:54.851+0000 7f0f7f7f6640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T18:27:54.853 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:27:54.851+0000 7f0f7f7f6640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:27:54.854 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:27:54.851+0000 7f0f7f7f6640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.1:6790/0 2024-03-25T18:27:54.854 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:27:54.851+0000 7f0f7f7f6640 1 -- 172.21.15.1:0/1798004735 --> v1:172.21.15.1:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0f781d7ee0 con 0x7f0f780b4e70 2024-03-25T18:27:55.169 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:27:55.167+0000 7f0f6dffb640 1 -- 172.21.15.1:0/1798004735 <== mon.2 v1:172.21.15.1:6790/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f0f700015b0 con 0x7f0f780b4e70 2024-03-25T18:27:55.169 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:27:55.167+0000 7f0f6dffb640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:27:55.169 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:27:55.167+0000 7f0f6dffb640 1 -- 172.21.15.1:0/1798004735 <== mon.2 v1:172.21.15.1:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1586965025 0 0) 0x7f0f70003e80 con 0x7f0f780b4e70 2024-03-25T18:27:55.169 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:27:55.167+0000 7f0f6dffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:27:55.169 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:27:55.167+0000 7f0f6dffb640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-25T18:27:55.169 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:27:55.167+0000 7f0f6dffb640 10 monclient: dump: 2024-03-25T18:27:55.170 INFO:teuthology.misc.health.smithi001.stderr:epoch 1 2024-03-25T18:27:55.170 INFO:teuthology.misc.health.smithi001.stderr:fsid c8ea72cf-9e99-4383-9b52-6c23d365fdfc 2024-03-25T18:27:55.170 INFO:teuthology.misc.health.smithi001.stderr:last_changed 2024-03-25T18:25:50.510519+0000 2024-03-25T18:27:55.170 INFO:teuthology.misc.health.smithi001.stderr:created 2024-03-25T18:25:50.510519+0000 2024-03-25T18:27:55.170 INFO:teuthology.misc.health.smithi001.stderr:min_mon_release 19 (squid) 2024-03-25T18:27:55.170 INFO:teuthology.misc.health.smithi001.stderr:election_strategy: 3 2024-03-25T18:27:55.170 INFO:teuthology.misc.health.smithi001.stderr:0: v1:172.21.15.1:6789/0 mon.a 2024-03-25T18:27:55.170 INFO:teuthology.misc.health.smithi001.stderr:1: v1:172.21.15.115:6789/0 mon.b 2024-03-25T18:27:55.170 INFO:teuthology.misc.health.smithi001.stderr:2: v1:172.21.15.1:6790/0 mon.c 2024-03-25T18:27:55.170 INFO:teuthology.misc.health.smithi001.stderr: 2024-03-25T18:27:55.170 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:27:55.167+0000 7f0f6f7fe640 4 set_mon_vals no callback set 2024-03-25T18:27:55.171 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:27:55.168+0000 7f0f6dffb640 1 -- 172.21.15.1:0/1798004735 <== mon.2 v1:172.21.15.1:6790/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 21787975 0 0) 0x7f0f700040a0 con 0x7f0f780b4e70 2024-03-25T18:27:55.171 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:27:55.169+0000 7f0f7d592640 10 monclient: get_auth_request con 0x7f0f5c06e110 auth_method 0 2024-03-25T18:27:55.171 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:27:55.169+0000 7f0f6dffb640 10 client.4277.objecter ms_handle_connect 0x7f0f5c06e110 2024-03-25T18:27:55.176 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:27:55.174+0000 7f0f6dffb640 1 -- 172.21.15.1:0/1798004735 <== mon.2 v1:172.21.15.1:6790/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 996254230 0 0) 0x7f0f70054b30 con 0x7f0f780b4e70 2024-03-25T18:27:55.176 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:27:55.174+0000 7f0f6dffb640 10 client.4277.objecter ms_dispatch 0x7f0f78000b90 osd_map(10..10 src has 1..10) v4 2024-03-25T18:27:55.176 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:27:55.174+0000 7f0f6dffb640 3 client.4277.objecter handle_osd_map got epochs [10,10] > 0 2024-03-25T18:27:55.176 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:27:55.174+0000 7f0f6dffb640 3 client.4277.objecter handle_osd_map decoding full epoch 10 2024-03-25T18:27:55.177 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:27:55.175+0000 7f0f6dffb640 20 client.4277.objecter dump_active .. 0 homeless 2024-03-25T18:27:55.177 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:27:55.175+0000 7f0f6dffb640 1 -- 172.21.15.1:0/1798004735 <== mon.2 v1:172.21.15.1:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f0f70050640 con 0x7f0f780b4e70 2024-03-25T18:27:55.177 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:27:55.175+0000 7f0f6dffb640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:27:55.177 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:27:55.175+0000 7f0f6dffb640 10 monclient: _finish_command 1 = system:0 2024-03-25T18:27:55.409 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:27:55.407+0000 7f0f7f7f6640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:27:55.409 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:27:55.407+0000 7f0f7f7f6640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:27:55.409 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:27:55.407+0000 7f0f7f7f6640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.1:6790/0 2024-03-25T18:27:55.410 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:27:55.407+0000 7f0f7f7f6640 1 -- 172.21.15.1:0/1798004735 --> v1:172.21.15.1:6790/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f0f78143220 con 0x7f0f780b4e70 2024-03-25T18:27:55.561 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:27:55.559+0000 7f0f6dffb640 1 -- 172.21.15.1:0/1798004735 <== mon.2 v1:172.21.15.1:6790/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 4092213348) 0x7f0f70052570 con 0x7f0f780b4e70 2024-03-25T18:27:55.561 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:27:55.559+0000 7f0f6dffb640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:27:55.561 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:27:55.559+0000 7f0f6dffb640 10 monclient: _finish_command 2 = system:0 2024-03-25T18:27:55.561 INFO:teuthology.misc.health.smithi001.stdout: 2024-03-25T18:27:55.562 INFO:teuthology.misc.health.smithi001.stdout:{"epoch":10,"fsid":"c8ea72cf-9e99-4383-9b52-6c23d365fdfc","created":"2024-03-25T18:26:38.856638+0000","modified":"2024-03-25T18:26:44.224208+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T18:26:44.224208+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"b9f34922-3a84-49be-8426-807c1eceaefc","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"42bf306f-55e3-4634-9874-8b4b73ac6644","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"292cb23a-18a9-4207-a7cc-4d5727feaec0","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"45e0e9ab-898d-4673-9590-b4117bc5e84a","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"831a5a09-73ab-4a61-9872-088e76bdbc66","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"88cd903d-3b55-4840-9396-6e3cfe005282","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"89db7355-bc9d-46f4-8ab0-e8d9e744e026","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"cac47bee-0914-4f08-896f-348838882866","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T18:27:55.564 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:27:55.562+0000 7f0f7f7f6640 10 librados: watch_flush enter 2024-03-25T18:27:55.564 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:27:55.562+0000 7f0f7f7f6640 10 librados: watch_flush exit 2024-03-25T18:27:55.564 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:27:55.562+0000 7f0f7f7f6640 20 client.4277.objecter shutdown clearing up homeless session... 2024-03-25T18:27:55.564 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:27:55.562+0000 7f0f7f7f6640 10 client.4277.objecter successfully canceled tick 2024-03-25T18:27:55.564 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:27:55.562+0000 7f0f7f7f6640 1 -- 172.21.15.1:0/1798004735 >> v1:172.21.15.115:6800/4237379176 conn(0x7f0f5c06e110 legacy=0x7f0f5c0705d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:27:55.565 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:27:55.562+0000 7f0f7f7f6640 10 monclient: shutdown 2024-03-25T18:27:55.565 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:27:55.562+0000 7f0f7f7f6640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:27:55.565 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:27:55.562+0000 7f0f7f7f6640 1 -- 172.21.15.1:0/1798004735 >> v1:172.21.15.1:6790/0 conn(0x7f0f780b4e70 legacy=0x7f0f781d4b80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:27:55.565 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:27:55.562+0000 7f0f7f7f6640 1 -- 172.21.15.1:0/1798004735 shutdown_connections 2024-03-25T18:27:55.565 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:27:55.562+0000 7f0f7f7f6640 1 -- 172.21.15.1:0/1798004735 >> 172.21.15.1:0/1798004735 conn(0x7f0f780617a0 msgr2=0x7f0f7813eda0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:27:55.565 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:27:55.562+0000 7f0f7f7f6640 1 -- 172.21.15.1:0/1798004735 shutdown_connections 2024-03-25T18:27:55.565 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:27:55.562+0000 7f0f7f7f6640 1 -- 172.21.15.1:0/1798004735 wait complete. 2024-03-25T18:27:55.565 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:27:55.562+0000 7f0f7f7f6640 1 librados: shutdown 2024-03-25T18:27:55.575 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T18:27:56.014 INFO:tasks.ceph.mon.a.smithi001.stderr:==00:00:01:34.573 34876== Warning: unimplemented fcntl command: 1036 2024-03-25T18:27:56.021 INFO:tasks.ceph.mon.a.smithi001.stderr:==00:00:01:34.579 34876== Warning: unimplemented fcntl command: 1036 2024-03-25T18:27:56.064 INFO:tasks.ceph.mon.a.smithi001.stderr:==00:00:01:34.622 34876== Warning: unimplemented fcntl command: 1036 2024-03-25T18:27:56.073 INFO:tasks.ceph.mon.b.smithi115.stderr:==00:00:01:34.556 34836== Warning: unimplemented fcntl command: 1036 2024-03-25T18:27:56.077 INFO:tasks.ceph.mon.b.smithi115.stderr:==00:00:01:34.560 34836== Warning: unimplemented fcntl command: 1036 2024-03-25T18:27:56.085 INFO:tasks.ceph.mon.c.smithi001.stderr:==00:00:01:34.644 34877== Warning: unimplemented fcntl command: 1036 2024-03-25T18:27:56.091 INFO:tasks.ceph.mon.c.smithi001.stderr:==00:00:01:34.649 34877== Warning: unimplemented fcntl command: 1036 2024-03-25T18:27:56.129 INFO:tasks.ceph.mon.b.smithi115.stderr:==00:00:01:34.612 34836== Warning: unimplemented fcntl command: 1036 2024-03-25T18:27:56.148 INFO:tasks.ceph.mon.c.smithi001.stderr:==00:00:01:34.707 34877== Warning: unimplemented fcntl command: 1036 2024-03-25T18:28:02.382 DEBUG:teuthology.orchestra.run.smithi001:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T18:28:02.496 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:28:02.494+0000 7f73683a8640 10 monclient: get_monmap_and_config 2024-03-25T18:28:02.496 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:28:02.494+0000 7f73683a8640 10 monclient: build_initial_monmap 2024-03-25T18:28:02.496 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:28:02.494+0000 7f73683a8640 10 monclient: monmap: 2024-03-25T18:28:02.496 INFO:teuthology.misc.health.smithi001.stderr:epoch 0 2024-03-25T18:28:02.496 INFO:teuthology.misc.health.smithi001.stderr:fsid c8ea72cf-9e99-4383-9b52-6c23d365fdfc 2024-03-25T18:28:02.496 INFO:teuthology.misc.health.smithi001.stderr:last_changed 2024-03-25T18:28:02.495924+0000 2024-03-25T18:28:02.496 INFO:teuthology.misc.health.smithi001.stderr:created 2024-03-25T18:28:02.495924+0000 2024-03-25T18:28:02.496 INFO:teuthology.misc.health.smithi001.stderr:min_mon_release 0 (unknown) 2024-03-25T18:28:02.496 INFO:teuthology.misc.health.smithi001.stderr:election_strategy: 3 2024-03-25T18:28:02.496 INFO:teuthology.misc.health.smithi001.stderr:0: v1:172.21.15.1:6789/0 mon.noname-a 2024-03-25T18:28:02.497 INFO:teuthology.misc.health.smithi001.stderr:1: v1:172.21.15.115:6789/0 mon.noname-c 2024-03-25T18:28:02.497 INFO:teuthology.misc.health.smithi001.stderr:2: v1:172.21.15.1:6790/0 mon.noname-b 2024-03-25T18:28:02.497 INFO:teuthology.misc.health.smithi001.stderr: 2024-03-25T18:28:02.498 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:28:02.495+0000 7f73683a8640 1 Processor -- start 2024-03-25T18:28:02.498 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:28:02.495+0000 7f73683a8640 1 -- start start 2024-03-25T18:28:02.498 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:28:02.495+0000 7f73683a8640 10 monclient: init 2024-03-25T18:28:02.498 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:28:02.496+0000 7f73683a8640 10 monclient: _reopen_session rank -1 2024-03-25T18:28:02.498 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:28:02.496+0000 7f73683a8640 10 monclient: _add_conns ranks=[2,1,0] 2024-03-25T18:28:02.498 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:28:02.496+0000 7f73683a8640 10 monclient(hunting): picked mon.noname-b con 0x7f73600b4ff0 addr v1:172.21.15.1:6790/0 2024-03-25T18:28:02.499 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:28:02.496+0000 7f73683a8640 10 monclient(hunting): picked mon.noname-c con 0x7f73600b4670 addr v1:172.21.15.115:6789/0 2024-03-25T18:28:02.499 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:28:02.496+0000 7f73683a8640 10 monclient(hunting): picked mon.noname-a con 0x7f736013c480 addr v1:172.21.15.1:6789/0 2024-03-25T18:28:02.499 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:28:02.496+0000 7f73683a8640 1 -- --> v1:172.21.15.1:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f73600ab0e0 con 0x7f736013c480 2024-03-25T18:28:02.499 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:28:02.496+0000 7f73683a8640 1 -- --> v1:172.21.15.115:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f73600ab3c0 con 0x7f73600b4670 2024-03-25T18:28:02.499 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:28:02.496+0000 7f73683a8640 1 -- --> v1:172.21.15.1:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f73600a9f30 con 0x7f73600b4ff0 2024-03-25T18:28:02.499 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:28:02.496+0000 7f73683a8640 10 monclient(hunting): _renew_subs 2024-03-25T18:28:02.499 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:28:02.496+0000 7f73683a8640 10 monclient(hunting): authenticate will time out at 7562.967086s 2024-03-25T18:28:02.517 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:28:02.516+0000 7f7366144640 1 --1- >> v1:172.21.15.1:6790/0 conn(0x7f73600b4ff0 0x7f736013bd70 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.1:6790/0 says I am v1:172.21.15.1:55882/0 (socket says 172.21.15.1:55882) 2024-03-25T18:28:02.517 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:28:02.516+0000 7f7366144640 1 -- 172.21.15.1:0/3539775258 learned_addr learned my addr 172.21.15.1:0/3539775258 (peer_addr_for_me v1:172.21.15.1:0/0) 2024-03-25T18:28:02.530 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:28:02.527+0000 7f7365142640 1 -- 172.21.15.1:0/3539775258 <== mon.1 v1:172.21.15.115:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3019160830 0 0) 0x7f73600ab3c0 con 0x7f73600b4670 2024-03-25T18:28:02.530 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:28:02.527+0000 7f7365142640 10 monclient(hunting): _negotiate 2024-03-25T18:28:02.530 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:28:02.527+0000 7f7365142640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:28:02.531 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:28:02.527+0000 7f7365142640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:28:02.531 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:28:02.527+0000 7f7365142640 10 monclient(hunting): my global_id is 4273 2024-03-25T18:28:02.531 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:28:02.527+0000 7f7365142640 1 -- 172.21.15.1:0/3539775258 --> v1:172.21.15.115:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f734c003520 con 0x7f73600b4670 2024-03-25T18:28:02.531 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:28:02.529+0000 7f7365142640 1 -- 172.21.15.1:0/3539775258 <== mon.2 v1:172.21.15.1:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3153276813 0 0) 0x7f73600a9f30 con 0x7f73600b4ff0 2024-03-25T18:28:02.531 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:28:02.529+0000 7f7365142640 10 monclient(hunting): _negotiate 2024-03-25T18:28:02.531 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:28:02.529+0000 7f7365142640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:28:02.531 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:28:02.529+0000 7f7365142640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:28:02.531 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:28:02.529+0000 7f7365142640 10 monclient(hunting): my global_id is 4280 2024-03-25T18:28:02.531 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:28:02.529+0000 7f7365142640 1 -- 172.21.15.1:0/3539775258 --> v1:172.21.15.1:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f73600ab3c0 con 0x7f73600b4ff0 2024-03-25T18:28:02.533 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:28:02.531+0000 7f7365142640 1 -- 172.21.15.1:0/3539775258 <== mon.0 v1:172.21.15.1:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 152400143 0 0) 0x7f73600ab0e0 con 0x7f736013c480 2024-03-25T18:28:02.533 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:28:02.531+0000 7f7365142640 10 monclient(hunting): _negotiate 2024-03-25T18:28:02.533 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:28:02.531+0000 7f7365142640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:28:02.533 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:28:02.531+0000 7f7365142640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:28:02.533 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:28:02.531+0000 7f7365142640 10 monclient(hunting): my global_id is 4278 2024-03-25T18:28:02.533 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:28:02.531+0000 7f7365142640 1 -- 172.21.15.1:0/3539775258 --> v1:172.21.15.1:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f73600a9f30 con 0x7f736013c480 2024-03-25T18:28:02.536 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:28:02.534+0000 7f7365142640 1 -- 172.21.15.1:0/3539775258 <== mon.1 v1:172.21.15.115:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1836710011 0 0) 0x7f734c003520 con 0x7f73600b4670 2024-03-25T18:28:02.536 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:28:02.534+0000 7f7365142640 1 -- 172.21.15.1:0/3539775258 >> v1:172.21.15.1:6790/0 conn(0x7f73600b4ff0 legacy=0x7f736013bd70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:28:02.537 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:28:02.534+0000 7f7365142640 1 -- 172.21.15.1:0/3539775258 >> v1:172.21.15.1:6789/0 conn(0x7f736013c480 legacy=0x7f736014dc00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:28:02.537 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:28:02.534+0000 7f7365142640 10 monclient: _finish_hunting 0 2024-03-25T18:28:02.537 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:28:02.534+0000 7f7365142640 1 monclient: found mon.noname-c 2024-03-25T18:28:02.537 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:28:02.534+0000 7f7365142640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:28:02.537 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:28:02.534+0000 7f7365142640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.115:6789/0 2024-03-25T18:28:02.537 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:28:02.534+0000 7f7365142640 1 -- 172.21.15.1:0/3539775258 --> v1:172.21.15.115:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f73600a9b70 con 0x7f73600b4670 2024-03-25T18:28:02.537 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:28:02.534+0000 7f7365142640 10 monclient: _finish_auth 0 2024-03-25T18:28:02.537 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:28:02.534+0000 7f7365142640 10 monclient: _check_auth_tickets 2024-03-25T18:28:02.537 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:28:02.534+0000 7f7365142640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:28:02.537 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:28:02.534+0000 7f7365142640 1 -- 172.21.15.1:0/3539775258 <== mon.1 v1:172.21.15.115:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1586965025 0 0) 0x7f7350001690 con 0x7f73600b4670 2024-03-25T18:28:02.537 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:28:02.535+0000 7f7365142640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:28:02.537 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:28:02.535+0000 7f7365142640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-03-25T18:28:02.537 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:28:02.535+0000 7f7365142640 10 monclient: dump: 2024-03-25T18:28:02.537 INFO:teuthology.misc.health.smithi001.stderr:epoch 1 2024-03-25T18:28:02.537 INFO:teuthology.misc.health.smithi001.stderr:fsid c8ea72cf-9e99-4383-9b52-6c23d365fdfc 2024-03-25T18:28:02.538 INFO:teuthology.misc.health.smithi001.stderr:last_changed 2024-03-25T18:25:50.510519+0000 2024-03-25T18:28:02.538 INFO:teuthology.misc.health.smithi001.stderr:created 2024-03-25T18:25:50.510519+0000 2024-03-25T18:28:02.538 INFO:teuthology.misc.health.smithi001.stderr:min_mon_release 19 (squid) 2024-03-25T18:28:02.538 INFO:teuthology.misc.health.smithi001.stderr:election_strategy: 3 2024-03-25T18:28:02.538 INFO:teuthology.misc.health.smithi001.stderr:0: v1:172.21.15.1:6789/0 mon.a 2024-03-25T18:28:02.538 INFO:teuthology.misc.health.smithi001.stderr:1: v1:172.21.15.115:6789/0 mon.b 2024-03-25T18:28:02.538 INFO:teuthology.misc.health.smithi001.stderr:2: v1:172.21.15.1:6790/0 mon.c 2024-03-25T18:28:02.538 INFO:teuthology.misc.health.smithi001.stderr: 2024-03-25T18:28:02.538 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:28:02.535+0000 7f73683a8640 5 monclient: authenticate success, global_id 4273 2024-03-25T18:28:02.538 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:28:02.535+0000 7f73683a8640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T18:28:02.558 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:28:02.556+0000 7f7365142640 1 -- 172.21.15.1:0/3539775258 <== mon.1 v1:172.21.15.115:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f73500017f0 con 0x7f73600b4670 2024-03-25T18:28:02.558 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:28:02.556+0000 7f7365142640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:28:02.558 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:28:02.556+0000 7f7365142640 1 -- 172.21.15.1:0/3539775258 <== mon.1 v1:172.21.15.115:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1586965025 0 0) 0x7f7350001c00 con 0x7f73600b4670 2024-03-25T18:28:02.558 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:28:02.556+0000 7f73683a8640 10 monclient: get_monmap_and_config success 2024-03-25T18:28:02.558 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:28:02.556+0000 7f73683a8640 4 set_mon_vals no callback set 2024-03-25T18:28:02.558 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:28:02.556+0000 7f7365142640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:28:02.558 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:28:02.556+0000 7f7365142640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-25T18:28:02.558 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:28:02.556+0000 7f7365142640 10 monclient: dump: 2024-03-25T18:28:02.558 INFO:teuthology.misc.health.smithi001.stderr:epoch 1 2024-03-25T18:28:02.559 INFO:teuthology.misc.health.smithi001.stderr:fsid c8ea72cf-9e99-4383-9b52-6c23d365fdfc 2024-03-25T18:28:02.559 INFO:teuthology.misc.health.smithi001.stderr:last_changed 2024-03-25T18:25:50.510519+0000 2024-03-25T18:28:02.559 INFO:teuthology.misc.health.smithi001.stderr:created 2024-03-25T18:25:50.510519+0000 2024-03-25T18:28:02.559 INFO:teuthology.misc.health.smithi001.stderr:min_mon_release 19 (squid) 2024-03-25T18:28:02.559 INFO:teuthology.misc.health.smithi001.stderr:election_strategy: 3 2024-03-25T18:28:02.559 INFO:teuthology.misc.health.smithi001.stderr:0: v1:172.21.15.1:6789/0 mon.a 2024-03-25T18:28:02.559 INFO:teuthology.misc.health.smithi001.stderr:1: v1:172.21.15.115:6789/0 mon.b 2024-03-25T18:28:02.559 INFO:teuthology.misc.health.smithi001.stderr:2: v1:172.21.15.1:6790/0 mon.c 2024-03-25T18:28:02.559 INFO:teuthology.misc.health.smithi001.stderr: 2024-03-25T18:28:02.561 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:28:02.557+0000 7f73683a8640 10 monclient: shutdown 2024-03-25T18:28:02.561 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:28:02.557+0000 7f73683a8640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:28:02.561 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:28:02.557+0000 7f73683a8640 1 -- 172.21.15.1:0/3539775258 >> v1:172.21.15.115:6789/0 conn(0x7f73600b4670 legacy=0x7f73601425c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:28:02.561 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:28:02.557+0000 7f73683a8640 1 -- 172.21.15.1:0/3539775258 shutdown_connections 2024-03-25T18:28:02.561 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:28:02.557+0000 7f73683a8640 1 -- 172.21.15.1:0/3539775258 >> 172.21.15.1:0/3539775258 conn(0x7f73600617a0 msgr2=0x7f7360061bb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:28:02.561 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:28:02.557+0000 7f73683a8640 1 -- 172.21.15.1:0/3539775258 shutdown_connections 2024-03-25T18:28:02.562 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:28:02.557+0000 7f73683a8640 1 -- 172.21.15.1:0/3539775258 wait complete. 2024-03-25T18:28:02.562 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:28:02.558+0000 7f73683a8640 10 monclient: build_initial_monmap 2024-03-25T18:28:02.562 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:28:02.558+0000 7f73683a8640 10 monclient: monmap: 2024-03-25T18:28:02.562 INFO:teuthology.misc.health.smithi001.stderr:epoch 0 2024-03-25T18:28:02.562 INFO:teuthology.misc.health.smithi001.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T18:28:02.562 INFO:teuthology.misc.health.smithi001.stderr:last_changed 0.000000 2024-03-25T18:28:02.562 INFO:teuthology.misc.health.smithi001.stderr:created 0.000000 2024-03-25T18:28:02.562 INFO:teuthology.misc.health.smithi001.stderr:min_mon_release 0 (unknown) 2024-03-25T18:28:02.562 INFO:teuthology.misc.health.smithi001.stderr:election_strategy: 1 2024-03-25T18:28:02.562 INFO:teuthology.misc.health.smithi001.stderr:0: v1:172.21.15.1:6789/0 mon.noname-a 2024-03-25T18:28:02.562 INFO:teuthology.misc.health.smithi001.stderr:1: v1:172.21.15.115:6789/0 mon.noname-b 2024-03-25T18:28:02.562 INFO:teuthology.misc.health.smithi001.stderr:2: v1:172.21.15.1:6790/0 mon.noname-c 2024-03-25T18:28:02.562 INFO:teuthology.misc.health.smithi001.stderr: 2024-03-25T18:28:02.562 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:28:02.558+0000 7f73683a8640 1 librados: starting msgr at 2024-03-25T18:28:02.562 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:28:02.558+0000 7f73683a8640 1 librados: starting objecter 2024-03-25T18:28:02.563 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:28:02.558+0000 7f73683a8640 1 Processor -- start 2024-03-25T18:28:02.563 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:28:02.558+0000 7f73683a8640 1 -- start start 2024-03-25T18:28:02.563 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:28:02.558+0000 7f73683a8640 1 librados: setting wanted keys 2024-03-25T18:28:02.563 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:28:02.558+0000 7f73683a8640 1 librados: calling monclient init 2024-03-25T18:28:02.563 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:28:02.558+0000 7f73683a8640 10 monclient: init 2024-03-25T18:28:02.563 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:28:02.559+0000 7f73683a8640 10 monclient: _reopen_session rank -1 2024-03-25T18:28:02.563 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:28:02.559+0000 7f73683a8640 10 monclient: _add_conns ranks=[0,1,2] 2024-03-25T18:28:02.563 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:28:02.559+0000 7f73683a8640 10 monclient(hunting): picked mon.noname-a con 0x7f73600b4670 addr v1:172.21.15.1:6789/0 2024-03-25T18:28:02.563 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:28:02.559+0000 7f73683a8640 10 monclient(hunting): picked mon.noname-b con 0x7f73600b4ff0 addr v1:172.21.15.115:6789/0 2024-03-25T18:28:02.563 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:28:02.559+0000 7f73683a8640 10 monclient(hunting): picked mon.noname-c con 0x7f736013c480 addr v1:172.21.15.1:6790/0 2024-03-25T18:28:02.563 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:28:02.559+0000 7f73683a8640 1 -- --> v1:172.21.15.1:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f73600af4f0 con 0x7f73600b4670 2024-03-25T18:28:02.563 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:28:02.559+0000 7f73683a8640 1 -- --> v1:172.21.15.115:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f73600ada40 con 0x7f73600b4ff0 2024-03-25T18:28:02.563 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:28:02.559+0000 7f73683a8640 1 -- --> v1:172.21.15.1:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f73600afcd0 con 0x7f736013c480 2024-03-25T18:28:02.563 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:28:02.559+0000 7f73683a8640 10 monclient(hunting): _renew_subs 2024-03-25T18:28:02.563 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:28:02.559+0000 7f73683a8640 10 monclient(hunting): authenticate will time out at 7563.030480s 2024-03-25T18:28:02.570 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:28:02.568+0000 7f7365943640 1 --1- >> v1:172.21.15.115:6789/0 conn(0x7f73600b4ff0 0x7f73600b0990 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.115:6789/0 says I am v1:172.21.15.1:48458/0 (socket says 172.21.15.1:48458) 2024-03-25T18:28:02.570 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:28:02.568+0000 7f7365943640 1 -- 172.21.15.1:0/15973693 learned_addr learned my addr 172.21.15.1:0/15973693 (peer_addr_for_me v1:172.21.15.1:0/0) 2024-03-25T18:28:02.570 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:28:02.569+0000 7f7356ffd640 10 client.?.objecter ms_handle_connect 0x7f73600b4ff0 2024-03-25T18:28:02.570 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:28:02.569+0000 7f7356ffd640 10 client.?.objecter resend_mon_ops 2024-03-25T18:28:02.572 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:28:02.570+0000 7f7356ffd640 10 client.?.objecter ms_handle_connect 0x7f73600b4670 2024-03-25T18:28:02.572 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:28:02.570+0000 7f7356ffd640 10 client.?.objecter resend_mon_ops 2024-03-25T18:28:02.576 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:28:02.574+0000 7f7356ffd640 10 client.?.objecter ms_handle_connect 0x7f736013c480 2024-03-25T18:28:02.576 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:28:02.574+0000 7f7356ffd640 10 client.?.objecter resend_mon_ops 2024-03-25T18:28:02.582 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:28:02.580+0000 7f7356ffd640 1 -- 172.21.15.1:0/15973693 <== mon.1 v1:172.21.15.115:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1115717219 0 0) 0x7f73600ada40 con 0x7f73600b4ff0 2024-03-25T18:28:02.582 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:28:02.580+0000 7f7356ffd640 10 monclient(hunting): _negotiate 2024-03-25T18:28:02.582 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:28:02.580+0000 7f7356ffd640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:28:02.582 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:28:02.580+0000 7f7356ffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:28:02.582 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:28:02.580+0000 7f7356ffd640 10 monclient(hunting): my global_id is 4276 2024-03-25T18:28:02.582 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:28:02.580+0000 7f7356ffd640 1 -- 172.21.15.1:0/15973693 --> v1:172.21.15.115:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f7344004eb0 con 0x7f73600b4ff0 2024-03-25T18:28:02.583 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:28:02.581+0000 7f7356ffd640 1 -- 172.21.15.1:0/15973693 <== mon.2 v1:172.21.15.1:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1994570326 0 0) 0x7f73600afcd0 con 0x7f736013c480 2024-03-25T18:28:02.583 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:28:02.581+0000 7f7356ffd640 10 monclient(hunting): _negotiate 2024-03-25T18:28:02.583 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:28:02.581+0000 7f7356ffd640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:28:02.583 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:28:02.581+0000 7f7356ffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:28:02.583 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:28:02.581+0000 7f7356ffd640 10 monclient(hunting): my global_id is 4283 2024-03-25T18:28:02.583 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:28:02.581+0000 7f7356ffd640 1 -- 172.21.15.1:0/15973693 --> v1:172.21.15.1:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f73600ada40 con 0x7f736013c480 2024-03-25T18:28:02.584 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:28:02.582+0000 7f7356ffd640 1 -- 172.21.15.1:0/15973693 <== mon.0 v1:172.21.15.1:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1064208003 0 0) 0x7f73600af4f0 con 0x7f73600b4670 2024-03-25T18:28:02.584 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:28:02.582+0000 7f7356ffd640 10 monclient(hunting): _negotiate 2024-03-25T18:28:02.584 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:28:02.582+0000 7f7356ffd640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:28:02.584 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:28:02.582+0000 7f7356ffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:28:02.584 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:28:02.582+0000 7f7356ffd640 10 monclient(hunting): my global_id is 4281 2024-03-25T18:28:02.584 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:28:02.582+0000 7f7356ffd640 1 -- 172.21.15.1:0/15973693 --> v1:172.21.15.1:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f73600afcd0 con 0x7f73600b4670 2024-03-25T18:28:02.594 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:28:02.593+0000 7f7356ffd640 1 -- 172.21.15.1:0/15973693 <== mon.1 v1:172.21.15.115:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1613208670 0 0) 0x7f7344004eb0 con 0x7f73600b4ff0 2024-03-25T18:28:02.595 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:28:02.593+0000 7f7356ffd640 1 -- 172.21.15.1:0/15973693 --> v1:172.21.15.115:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f73600af4f0 con 0x7f73600b4ff0 2024-03-25T18:28:02.595 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:28:02.593+0000 7f7356ffd640 1 -- 172.21.15.1:0/15973693 <== mon.1 v1:172.21.15.115:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1586965025 0 0) 0x7f7350001bc0 con 0x7f73600b4ff0 2024-03-25T18:28:02.595 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:28:02.593+0000 7f7356ffd640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:28:02.595 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:28:02.593+0000 7f7356ffd640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-03-25T18:28:02.595 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:28:02.593+0000 7f7356ffd640 10 monclient(hunting): dump: 2024-03-25T18:28:02.595 INFO:teuthology.misc.health.smithi001.stderr:epoch 1 2024-03-25T18:28:02.595 INFO:teuthology.misc.health.smithi001.stderr:fsid c8ea72cf-9e99-4383-9b52-6c23d365fdfc 2024-03-25T18:28:02.595 INFO:teuthology.misc.health.smithi001.stderr:last_changed 2024-03-25T18:25:50.510519+0000 2024-03-25T18:28:02.595 INFO:teuthology.misc.health.smithi001.stderr:created 2024-03-25T18:25:50.510519+0000 2024-03-25T18:28:02.595 INFO:teuthology.misc.health.smithi001.stderr:min_mon_release 19 (squid) 2024-03-25T18:28:02.595 INFO:teuthology.misc.health.smithi001.stderr:election_strategy: 3 2024-03-25T18:28:02.595 INFO:teuthology.misc.health.smithi001.stderr:0: v1:172.21.15.1:6789/0 mon.a 2024-03-25T18:28:02.596 INFO:teuthology.misc.health.smithi001.stderr:1: v1:172.21.15.115:6789/0 mon.b 2024-03-25T18:28:02.596 INFO:teuthology.misc.health.smithi001.stderr:2: v1:172.21.15.1:6790/0 mon.c 2024-03-25T18:28:02.596 INFO:teuthology.misc.health.smithi001.stderr: 2024-03-25T18:28:02.596 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:28:02.593+0000 7f7356ffd640 10 monclient(hunting): _finish_auth 0 2024-03-25T18:28:02.597 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:28:02.595+0000 7f7356ffd640 1 -- 172.21.15.1:0/15973693 <== mon.2 v1:172.21.15.1:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1603943182 0 0) 0x7f73600ada40 con 0x7f736013c480 2024-03-25T18:28:02.597 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:28:02.595+0000 7f7356ffd640 1 -- 172.21.15.1:0/15973693 --> v1:172.21.15.1:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f7344004eb0 con 0x7f736013c480 2024-03-25T18:28:02.597 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:28:02.595+0000 7f7356ffd640 1 -- 172.21.15.1:0/15973693 <== mon.2 v1:172.21.15.1:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1586965025 0 0) 0x7f7358001b50 con 0x7f736013c480 2024-03-25T18:28:02.597 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:28:02.595+0000 7f7356ffd640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:28:02.597 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:28:02.595+0000 7f7356ffd640 10 monclient(hunting): got monmap 1 from mon.c (according to old e1) 2024-03-25T18:28:02.597 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:28:02.595+0000 7f7356ffd640 10 monclient(hunting): dump: 2024-03-25T18:28:02.597 INFO:teuthology.misc.health.smithi001.stderr:epoch 1 2024-03-25T18:28:02.597 INFO:teuthology.misc.health.smithi001.stderr:fsid c8ea72cf-9e99-4383-9b52-6c23d365fdfc 2024-03-25T18:28:02.597 INFO:teuthology.misc.health.smithi001.stderr:last_changed 2024-03-25T18:25:50.510519+0000 2024-03-25T18:28:02.598 INFO:teuthology.misc.health.smithi001.stderr:created 2024-03-25T18:25:50.510519+0000 2024-03-25T18:28:02.598 INFO:teuthology.misc.health.smithi001.stderr:min_mon_release 19 (squid) 2024-03-25T18:28:02.598 INFO:teuthology.misc.health.smithi001.stderr:election_strategy: 3 2024-03-25T18:28:02.598 INFO:teuthology.misc.health.smithi001.stderr:0: v1:172.21.15.1:6789/0 mon.a 2024-03-25T18:28:02.598 INFO:teuthology.misc.health.smithi001.stderr:1: v1:172.21.15.115:6789/0 mon.b 2024-03-25T18:28:02.598 INFO:teuthology.misc.health.smithi001.stderr:2: v1:172.21.15.1:6790/0 mon.c 2024-03-25T18:28:02.598 INFO:teuthology.misc.health.smithi001.stderr: 2024-03-25T18:28:02.606 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:28:02.605+0000 7f7356ffd640 1 -- 172.21.15.1:0/15973693 <== mon.0 v1:172.21.15.1:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 487093675 0 0) 0x7f73600afcd0 con 0x7f73600b4670 2024-03-25T18:28:02.607 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:28:02.605+0000 7f7356ffd640 1 -- 172.21.15.1:0/15973693 --> v1:172.21.15.1:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f73600ada40 con 0x7f73600b4670 2024-03-25T18:28:02.607 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:28:02.605+0000 7f7356ffd640 1 -- 172.21.15.1:0/15973693 <== mon.0 v1:172.21.15.1:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1586965025 0 0) 0x7f735c0018f0 con 0x7f73600b4670 2024-03-25T18:28:02.607 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:28:02.605+0000 7f7356ffd640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:28:02.607 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:28:02.605+0000 7f7356ffd640 10 monclient(hunting): got monmap 1 from mon.a (according to old e1) 2024-03-25T18:28:02.607 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:28:02.605+0000 7f7356ffd640 10 monclient(hunting): dump: 2024-03-25T18:28:02.607 INFO:teuthology.misc.health.smithi001.stderr:epoch 1 2024-03-25T18:28:02.607 INFO:teuthology.misc.health.smithi001.stderr:fsid c8ea72cf-9e99-4383-9b52-6c23d365fdfc 2024-03-25T18:28:02.607 INFO:teuthology.misc.health.smithi001.stderr:last_changed 2024-03-25T18:25:50.510519+0000 2024-03-25T18:28:02.607 INFO:teuthology.misc.health.smithi001.stderr:created 2024-03-25T18:25:50.510519+0000 2024-03-25T18:28:02.607 INFO:teuthology.misc.health.smithi001.stderr:min_mon_release 19 (squid) 2024-03-25T18:28:02.607 INFO:teuthology.misc.health.smithi001.stderr:election_strategy: 3 2024-03-25T18:28:02.607 INFO:teuthology.misc.health.smithi001.stderr:0: v1:172.21.15.1:6789/0 mon.a 2024-03-25T18:28:02.607 INFO:teuthology.misc.health.smithi001.stderr:1: v1:172.21.15.115:6789/0 mon.b 2024-03-25T18:28:02.608 INFO:teuthology.misc.health.smithi001.stderr:2: v1:172.21.15.1:6790/0 mon.c 2024-03-25T18:28:02.608 INFO:teuthology.misc.health.smithi001.stderr: 2024-03-25T18:28:02.615 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:28:02.613+0000 7f7356ffd640 1 -- 172.21.15.1:0/15973693 <== mon.1 v1:172.21.15.115:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 378469312 0 0) 0x7f73600af4f0 con 0x7f73600b4ff0 2024-03-25T18:28:02.615 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:28:02.613+0000 7f7356ffd640 1 -- 172.21.15.1:0/15973693 >> v1:172.21.15.1:6790/0 conn(0x7f736013c480 legacy=0x7f73600b10a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:28:02.616 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:28:02.613+0000 7f7356ffd640 1 -- 172.21.15.1:0/15973693 >> v1:172.21.15.1:6789/0 conn(0x7f73600b4670 legacy=0x7f736013a0f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:28:02.616 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:28:02.613+0000 7f7356ffd640 10 monclient: _finish_hunting 0 2024-03-25T18:28:02.616 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:28:02.613+0000 7f7356ffd640 1 monclient: found mon.b 2024-03-25T18:28:02.616 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:28:02.613+0000 7f7356ffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:28:02.616 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:28:02.613+0000 7f7356ffd640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.115:6789/0 2024-03-25T18:28:02.616 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:28:02.613+0000 7f7356ffd640 1 -- 172.21.15.1:0/15973693 --> v1:172.21.15.115:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f73600affb0 con 0x7f73600b4ff0 2024-03-25T18:28:02.616 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:28:02.613+0000 7f7356ffd640 10 monclient: _finish_auth 0 2024-03-25T18:28:02.616 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:28:02.613+0000 7f7356ffd640 10 monclient: _check_auth_tickets 2024-03-25T18:28:02.616 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:28:02.613+0000 7f7356ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:28:02.616 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:28:02.614+0000 7f73683a8640 5 monclient: authenticate success, global_id 4276 2024-03-25T18:28:02.616 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:28:02.614+0000 7f73683a8640 10 monclient: _renew_subs 2024-03-25T18:28:02.616 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:28:02.614+0000 7f73683a8640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.115:6789/0 2024-03-25T18:28:02.616 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:28:02.614+0000 7f73683a8640 1 -- 172.21.15.1:0/15973693 --> v1:172.21.15.115:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f73600b0290 con 0x7f73600b4ff0 2024-03-25T18:28:02.617 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:28:02.614+0000 7f73683a8640 10 client.4276.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T18:28:02.617 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:28:02.614+0000 7f73683a8640 10 monclient: _renew_subs 2024-03-25T18:28:02.617 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:28:02.614+0000 7f73683a8640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.115:6789/0 2024-03-25T18:28:02.617 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:28:02.614+0000 7f73683a8640 1 -- 172.21.15.1:0/15973693 --> v1:172.21.15.115:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f73600aefd0 con 0x7f73600b4ff0 2024-03-25T18:28:02.617 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:28:02.614+0000 7f73683a8640 1 librados: init done 2024-03-25T18:28:02.617 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:28:02.615+0000 7f73683a8640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T18:28:02.617 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:28:02.615+0000 7f73683a8640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:28:02.618 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:28:02.615+0000 7f73683a8640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.115:6789/0 2024-03-25T18:28:02.618 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:28:02.615+0000 7f73683a8640 1 -- 172.21.15.1:0/15973693 --> v1:172.21.15.115:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f73600af1c0 con 0x7f73600b4ff0 2024-03-25T18:28:02.884 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:28:02.882+0000 7f7356ffd640 1 -- 172.21.15.1:0/15973693 <== mon.1 v1:172.21.15.115:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f73500011f0 con 0x7f73600b4ff0 2024-03-25T18:28:02.884 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:28:02.882+0000 7f7356ffd640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:28:02.884 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:28:02.882+0000 7f7356ffd640 1 -- 172.21.15.1:0/15973693 <== mon.1 v1:172.21.15.115:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1586965025 0 0) 0x7f7350003b00 con 0x7f73600b4ff0 2024-03-25T18:28:02.884 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:28:02.882+0000 7f7356ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:28:02.884 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:28:02.882+0000 7f7356ffd640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-25T18:28:02.884 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:28:02.882+0000 7f7356ffd640 10 monclient: dump: 2024-03-25T18:28:02.884 INFO:teuthology.misc.health.smithi001.stderr:epoch 1 2024-03-25T18:28:02.884 INFO:teuthology.misc.health.smithi001.stderr:fsid c8ea72cf-9e99-4383-9b52-6c23d365fdfc 2024-03-25T18:28:02.884 INFO:teuthology.misc.health.smithi001.stderr:last_changed 2024-03-25T18:25:50.510519+0000 2024-03-25T18:28:02.884 INFO:teuthology.misc.health.smithi001.stderr:created 2024-03-25T18:25:50.510519+0000 2024-03-25T18:28:02.884 INFO:teuthology.misc.health.smithi001.stderr:min_mon_release 19 (squid) 2024-03-25T18:28:02.885 INFO:teuthology.misc.health.smithi001.stderr:election_strategy: 3 2024-03-25T18:28:02.885 INFO:teuthology.misc.health.smithi001.stderr:0: v1:172.21.15.1:6789/0 mon.a 2024-03-25T18:28:02.885 INFO:teuthology.misc.health.smithi001.stderr:1: v1:172.21.15.115:6789/0 mon.b 2024-03-25T18:28:02.885 INFO:teuthology.misc.health.smithi001.stderr:2: v1:172.21.15.1:6790/0 mon.c 2024-03-25T18:28:02.885 INFO:teuthology.misc.health.smithi001.stderr: 2024-03-25T18:28:02.885 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:28:02.882+0000 7f7364941640 4 set_mon_vals no callback set 2024-03-25T18:28:02.885 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:28:02.883+0000 7f7356ffd640 1 -- 172.21.15.1:0/15973693 <== mon.1 v1:172.21.15.115:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 21787975 0 0) 0x7f7350003d20 con 0x7f73600b4ff0 2024-03-25T18:28:02.885 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:28:02.883+0000 7f7366144640 10 monclient: get_auth_request con 0x7f734406e0f0 auth_method 0 2024-03-25T18:28:02.886 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:28:02.883+0000 7f7356ffd640 10 client.4276.objecter ms_handle_connect 0x7f734406e0f0 2024-03-25T18:28:02.890 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:28:02.888+0000 7f7356ffd640 1 -- 172.21.15.1:0/15973693 <== mon.1 v1:172.21.15.115:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 996254230 0 0) 0x7f7350054880 con 0x7f73600b4ff0 2024-03-25T18:28:02.890 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:28:02.888+0000 7f7356ffd640 10 client.4276.objecter ms_dispatch 0x7f7360000b90 osd_map(10..10 src has 1..10) v4 2024-03-25T18:28:02.890 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:28:02.888+0000 7f7356ffd640 3 client.4276.objecter handle_osd_map got epochs [10,10] > 0 2024-03-25T18:28:02.890 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:28:02.888+0000 7f7356ffd640 3 client.4276.objecter handle_osd_map decoding full epoch 10 2024-03-25T18:28:02.890 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:28:02.888+0000 7f7356ffd640 20 client.4276.objecter dump_active .. 0 homeless 2024-03-25T18:28:02.890 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:28:02.888+0000 7f7356ffd640 1 -- 172.21.15.1:0/15973693 <== mon.1 v1:172.21.15.115:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f7350050390 con 0x7f73600b4ff0 2024-03-25T18:28:02.890 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:28:02.888+0000 7f7356ffd640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:28:02.890 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:28:02.888+0000 7f7356ffd640 10 monclient: _finish_command 1 = system:0 2024-03-25T18:28:03.113 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:28:03.111+0000 7f73683a8640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:28:03.113 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:28:03.111+0000 7f73683a8640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:28:03.113 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:28:03.111+0000 7f73683a8640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.115:6789/0 2024-03-25T18:28:03.113 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:28:03.111+0000 7f73683a8640 1 -- 172.21.15.1:0/15973693 --> v1:172.21.15.115:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f73600688c0 con 0x7f73600b4ff0 2024-03-25T18:28:03.147 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:28:03.146+0000 7f7356ffd640 1 -- 172.21.15.1:0/15973693 <== mon.1 v1:172.21.15.115:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 4092213348) 0x7f73500522c0 con 0x7f73600b4ff0 2024-03-25T18:28:03.148 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:28:03.146+0000 7f7356ffd640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:28:03.148 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:28:03.146+0000 7f7356ffd640 10 monclient: _finish_command 2 = system:0 2024-03-25T18:28:03.148 INFO:teuthology.misc.health.smithi001.stdout: 2024-03-25T18:28:03.148 INFO:teuthology.misc.health.smithi001.stdout:{"epoch":10,"fsid":"c8ea72cf-9e99-4383-9b52-6c23d365fdfc","created":"2024-03-25T18:26:38.856638+0000","modified":"2024-03-25T18:26:44.224208+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T18:26:44.224208+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"b9f34922-3a84-49be-8426-807c1eceaefc","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"42bf306f-55e3-4634-9874-8b4b73ac6644","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"292cb23a-18a9-4207-a7cc-4d5727feaec0","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"45e0e9ab-898d-4673-9590-b4117bc5e84a","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"831a5a09-73ab-4a61-9872-088e76bdbc66","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"88cd903d-3b55-4840-9396-6e3cfe005282","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"89db7355-bc9d-46f4-8ab0-e8d9e744e026","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"cac47bee-0914-4f08-896f-348838882866","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T18:28:03.150 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:28:03.147+0000 7f73683a8640 10 librados: watch_flush enter 2024-03-25T18:28:03.150 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:28:03.148+0000 7f73683a8640 10 librados: watch_flush exit 2024-03-25T18:28:03.150 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:28:03.148+0000 7f73683a8640 20 client.4276.objecter shutdown clearing up homeless session... 2024-03-25T18:28:03.150 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:28:03.148+0000 7f73683a8640 10 client.4276.objecter successfully canceled tick 2024-03-25T18:28:03.150 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:28:03.148+0000 7f73683a8640 1 -- 172.21.15.1:0/15973693 >> v1:172.21.15.115:6800/4237379176 conn(0x7f734406e0f0 legacy=0x7f73440705b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:28:03.150 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:28:03.148+0000 7f73683a8640 10 monclient: shutdown 2024-03-25T18:28:03.150 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:28:03.148+0000 7f73683a8640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:28:03.150 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:28:03.148+0000 7f73683a8640 1 -- 172.21.15.1:0/15973693 >> v1:172.21.15.115:6789/0 conn(0x7f73600b4ff0 legacy=0x7f73600b0990 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:28:03.150 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:28:03.148+0000 7f73683a8640 1 -- 172.21.15.1:0/15973693 shutdown_connections 2024-03-25T18:28:03.150 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:28:03.148+0000 7f73683a8640 1 -- 172.21.15.1:0/15973693 >> 172.21.15.1:0/15973693 conn(0x7f73600617a0 msgr2=0x7f73600b71c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:28:03.150 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:28:03.148+0000 7f73683a8640 1 -- 172.21.15.1:0/15973693 shutdown_connections 2024-03-25T18:28:03.151 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:28:03.148+0000 7f73683a8640 1 -- 172.21.15.1:0/15973693 wait complete. 2024-03-25T18:28:03.151 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:28:03.148+0000 7f73683a8640 1 librados: shutdown 2024-03-25T18:28:03.161 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T18:28:09.968 DEBUG:teuthology.orchestra.run.smithi001:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T18:28:10.085 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:28:10.083+0000 7f2846d2b640 10 monclient: get_monmap_and_config 2024-03-25T18:28:10.085 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:28:10.083+0000 7f2846d2b640 10 monclient: build_initial_monmap 2024-03-25T18:28:10.085 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:28:10.083+0000 7f2846d2b640 10 monclient: monmap: 2024-03-25T18:28:10.085 INFO:teuthology.misc.health.smithi001.stderr:epoch 0 2024-03-25T18:28:10.086 INFO:teuthology.misc.health.smithi001.stderr:fsid c8ea72cf-9e99-4383-9b52-6c23d365fdfc 2024-03-25T18:28:10.086 INFO:teuthology.misc.health.smithi001.stderr:last_changed 2024-03-25T18:28:10.085478+0000 2024-03-25T18:28:10.086 INFO:teuthology.misc.health.smithi001.stderr:created 2024-03-25T18:28:10.085478+0000 2024-03-25T18:28:10.086 INFO:teuthology.misc.health.smithi001.stderr:min_mon_release 0 (unknown) 2024-03-25T18:28:10.086 INFO:teuthology.misc.health.smithi001.stderr:election_strategy: 3 2024-03-25T18:28:10.086 INFO:teuthology.misc.health.smithi001.stderr:0: v1:172.21.15.1:6789/0 mon.noname-a 2024-03-25T18:28:10.086 INFO:teuthology.misc.health.smithi001.stderr:1: v1:172.21.15.115:6789/0 mon.noname-c 2024-03-25T18:28:10.086 INFO:teuthology.misc.health.smithi001.stderr:2: v1:172.21.15.1:6790/0 mon.noname-b 2024-03-25T18:28:10.086 INFO:teuthology.misc.health.smithi001.stderr: 2024-03-25T18:28:10.087 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:28:10.084+0000 7f2846d2b640 1 Processor -- start 2024-03-25T18:28:10.087 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:28:10.084+0000 7f2846d2b640 1 -- start start 2024-03-25T18:28:10.088 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:28:10.084+0000 7f2846d2b640 10 monclient: init 2024-03-25T18:28:10.088 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:28:10.085+0000 7f2846d2b640 10 monclient: _reopen_session rank -1 2024-03-25T18:28:10.088 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:28:10.085+0000 7f2846d2b640 10 monclient: _add_conns ranks=[2,0,1] 2024-03-25T18:28:10.088 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:28:10.085+0000 7f2846d2b640 10 monclient(hunting): picked mon.noname-b con 0x7f28400b4ff0 addr v1:172.21.15.1:6790/0 2024-03-25T18:28:10.088 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:28:10.085+0000 7f2846d2b640 10 monclient(hunting): picked mon.noname-a con 0x7f28400b4670 addr v1:172.21.15.1:6789/0 2024-03-25T18:28:10.088 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:28:10.085+0000 7f2846d2b640 10 monclient(hunting): picked mon.noname-c con 0x7f284013c480 addr v1:172.21.15.115:6789/0 2024-03-25T18:28:10.088 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:28:10.085+0000 7f2846d2b640 1 -- --> v1:172.21.15.1:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f28400ab0e0 con 0x7f28400b4670 2024-03-25T18:28:10.088 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:28:10.085+0000 7f2846d2b640 1 -- --> v1:172.21.15.115:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f28400ab3c0 con 0x7f284013c480 2024-03-25T18:28:10.088 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:28:10.085+0000 7f2846d2b640 1 -- --> v1:172.21.15.1:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f28400a9f30 con 0x7f28400b4ff0 2024-03-25T18:28:10.088 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:28:10.085+0000 7f2846d2b640 10 monclient(hunting): _renew_subs 2024-03-25T18:28:10.088 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:28:10.085+0000 7f2846d2b640 10 monclient(hunting): authenticate will time out at 7570.556474s 2024-03-25T18:28:10.091 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:28:10.089+0000 7f28452c8640 1 --1- >> v1:172.21.15.115:6789/0 conn(0x7f284013c480 0x7f284014dc00 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.115:6789/0 says I am v1:172.21.15.1:53050/0 (socket says 172.21.15.1:53050) 2024-03-25T18:28:10.091 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:28:10.090+0000 7f28452c8640 1 -- 172.21.15.1:0/367354554 learned_addr learned my addr 172.21.15.1:0/367354554 (peer_addr_for_me v1:172.21.15.1:0/0) 2024-03-25T18:28:10.101 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:28:10.099+0000 7f283f7fe640 1 -- 172.21.15.1:0/367354554 <== mon.1 v1:172.21.15.115:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2426240506 0 0) 0x7f28400ab3c0 con 0x7f284013c480 2024-03-25T18:28:10.101 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:28:10.099+0000 7f283f7fe640 10 monclient(hunting): _negotiate 2024-03-25T18:28:10.101 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:28:10.099+0000 7f283f7fe640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:28:10.101 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:28:10.099+0000 7f283f7fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:28:10.101 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:28:10.099+0000 7f283f7fe640 10 monclient(hunting): my global_id is 4279 2024-03-25T18:28:10.101 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:28:10.099+0000 7f283f7fe640 1 -- 172.21.15.1:0/367354554 --> v1:172.21.15.115:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f2830003520 con 0x7f284013c480 2024-03-25T18:28:10.105 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:28:10.103+0000 7f283f7fe640 1 -- 172.21.15.1:0/367354554 <== mon.2 v1:172.21.15.1:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3640730693 0 0) 0x7f28400a9f30 con 0x7f28400b4ff0 2024-03-25T18:28:10.105 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:28:10.103+0000 7f283f7fe640 10 monclient(hunting): _negotiate 2024-03-25T18:28:10.105 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:28:10.103+0000 7f283f7fe640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:28:10.105 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:28:10.103+0000 7f283f7fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:28:10.105 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:28:10.103+0000 7f283f7fe640 10 monclient(hunting): my global_id is 4286 2024-03-25T18:28:10.105 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:28:10.103+0000 7f283f7fe640 1 -- 172.21.15.1:0/367354554 --> v1:172.21.15.1:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f28400ab3c0 con 0x7f28400b4ff0 2024-03-25T18:28:10.110 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:28:10.108+0000 7f283f7fe640 1 -- 172.21.15.1:0/367354554 <== mon.0 v1:172.21.15.1:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 591730894 0 0) 0x7f28400ab0e0 con 0x7f28400b4670 2024-03-25T18:28:10.110 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:28:10.108+0000 7f283f7fe640 10 monclient(hunting): _negotiate 2024-03-25T18:28:10.110 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:28:10.108+0000 7f283f7fe640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:28:10.110 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:28:10.108+0000 7f283f7fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:28:10.110 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:28:10.108+0000 7f283f7fe640 10 monclient(hunting): my global_id is 4284 2024-03-25T18:28:10.110 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:28:10.108+0000 7f283f7fe640 1 -- 172.21.15.1:0/367354554 --> v1:172.21.15.1:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f28400a9f30 con 0x7f28400b4670 2024-03-25T18:28:10.114 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:28:10.112+0000 7f283f7fe640 1 -- 172.21.15.1:0/367354554 <== mon.1 v1:172.21.15.115:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 4171453269 0 0) 0x7f2830003520 con 0x7f284013c480 2024-03-25T18:28:10.114 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:28:10.112+0000 7f283f7fe640 1 -- 172.21.15.1:0/367354554 >> v1:172.21.15.1:6790/0 conn(0x7f28400b4ff0 legacy=0x7f284013bd70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:28:10.114 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:28:10.112+0000 7f283f7fe640 1 -- 172.21.15.1:0/367354554 >> v1:172.21.15.1:6789/0 conn(0x7f28400b4670 legacy=0x7f28401425c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:28:10.114 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:28:10.112+0000 7f283f7fe640 10 monclient: _finish_hunting 0 2024-03-25T18:28:10.114 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:28:10.112+0000 7f283f7fe640 1 monclient: found mon.noname-c 2024-03-25T18:28:10.114 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:28:10.112+0000 7f283f7fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:28:10.114 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:28:10.112+0000 7f283f7fe640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.115:6789/0 2024-03-25T18:28:10.114 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:28:10.112+0000 7f283f7fe640 1 -- 172.21.15.1:0/367354554 --> v1:172.21.15.115:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f28400a9b70 con 0x7f284013c480 2024-03-25T18:28:10.115 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:28:10.112+0000 7f283f7fe640 10 monclient: _finish_auth 0 2024-03-25T18:28:10.115 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:28:10.112+0000 7f283f7fe640 10 monclient: _check_auth_tickets 2024-03-25T18:28:10.115 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:28:10.112+0000 7f283f7fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:28:10.115 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:28:10.112+0000 7f283f7fe640 1 -- 172.21.15.1:0/367354554 <== mon.1 v1:172.21.15.115:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1586965025 0 0) 0x7f2838001a10 con 0x7f284013c480 2024-03-25T18:28:10.115 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:28:10.112+0000 7f283f7fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:28:10.115 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:28:10.112+0000 7f283f7fe640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-03-25T18:28:10.115 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:28:10.112+0000 7f283f7fe640 10 monclient: dump: 2024-03-25T18:28:10.115 INFO:teuthology.misc.health.smithi001.stderr:epoch 1 2024-03-25T18:28:10.115 INFO:teuthology.misc.health.smithi001.stderr:fsid c8ea72cf-9e99-4383-9b52-6c23d365fdfc 2024-03-25T18:28:10.115 INFO:teuthology.misc.health.smithi001.stderr:last_changed 2024-03-25T18:25:50.510519+0000 2024-03-25T18:28:10.115 INFO:teuthology.misc.health.smithi001.stderr:created 2024-03-25T18:25:50.510519+0000 2024-03-25T18:28:10.115 INFO:teuthology.misc.health.smithi001.stderr:min_mon_release 19 (squid) 2024-03-25T18:28:10.115 INFO:teuthology.misc.health.smithi001.stderr:election_strategy: 3 2024-03-25T18:28:10.115 INFO:teuthology.misc.health.smithi001.stderr:0: v1:172.21.15.1:6789/0 mon.a 2024-03-25T18:28:10.115 INFO:teuthology.misc.health.smithi001.stderr:1: v1:172.21.15.115:6789/0 mon.b 2024-03-25T18:28:10.116 INFO:teuthology.misc.health.smithi001.stderr:2: v1:172.21.15.1:6790/0 mon.c 2024-03-25T18:28:10.116 INFO:teuthology.misc.health.smithi001.stderr: 2024-03-25T18:28:10.116 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:28:10.112+0000 7f2846d2b640 5 monclient: authenticate success, global_id 4279 2024-03-25T18:28:10.116 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:28:10.112+0000 7f2846d2b640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T18:28:10.122 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:28:10.120+0000 7f283f7fe640 1 -- 172.21.15.1:0/367354554 <== mon.1 v1:172.21.15.115:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f2838001b70 con 0x7f284013c480 2024-03-25T18:28:10.122 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:28:10.120+0000 7f283f7fe640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:28:10.122 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:28:10.120+0000 7f283f7fe640 1 -- 172.21.15.1:0/367354554 <== mon.1 v1:172.21.15.115:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1586965025 0 0) 0x7f2838001f80 con 0x7f284013c480 2024-03-25T18:28:10.122 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:28:10.120+0000 7f2846d2b640 10 monclient: get_monmap_and_config success 2024-03-25T18:28:10.122 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:28:10.120+0000 7f2846d2b640 4 set_mon_vals no callback set 2024-03-25T18:28:10.122 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:28:10.120+0000 7f283f7fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:28:10.122 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:28:10.120+0000 7f283f7fe640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-25T18:28:10.122 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:28:10.120+0000 7f283f7fe640 10 monclient: dump: 2024-03-25T18:28:10.123 INFO:teuthology.misc.health.smithi001.stderr:epoch 1 2024-03-25T18:28:10.123 INFO:teuthology.misc.health.smithi001.stderr:fsid c8ea72cf-9e99-4383-9b52-6c23d365fdfc 2024-03-25T18:28:10.123 INFO:teuthology.misc.health.smithi001.stderr:last_changed 2024-03-25T18:25:50.510519+0000 2024-03-25T18:28:10.123 INFO:teuthology.misc.health.smithi001.stderr:created 2024-03-25T18:25:50.510519+0000 2024-03-25T18:28:10.123 INFO:teuthology.misc.health.smithi001.stderr:min_mon_release 19 (squid) 2024-03-25T18:28:10.123 INFO:teuthology.misc.health.smithi001.stderr:election_strategy: 3 2024-03-25T18:28:10.123 INFO:teuthology.misc.health.smithi001.stderr:0: v1:172.21.15.1:6789/0 mon.a 2024-03-25T18:28:10.123 INFO:teuthology.misc.health.smithi001.stderr:1: v1:172.21.15.115:6789/0 mon.b 2024-03-25T18:28:10.123 INFO:teuthology.misc.health.smithi001.stderr:2: v1:172.21.15.1:6790/0 mon.c 2024-03-25T18:28:10.123 INFO:teuthology.misc.health.smithi001.stderr: 2024-03-25T18:28:10.124 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:28:10.121+0000 7f2846d2b640 10 monclient: shutdown 2024-03-25T18:28:10.124 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:28:10.121+0000 7f2846d2b640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:28:10.125 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:28:10.121+0000 7f2846d2b640 1 -- 172.21.15.1:0/367354554 >> v1:172.21.15.115:6789/0 conn(0x7f284013c480 legacy=0x7f284014dc00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:28:10.125 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:28:10.121+0000 7f2846d2b640 1 -- 172.21.15.1:0/367354554 shutdown_connections 2024-03-25T18:28:10.125 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:28:10.121+0000 7f2846d2b640 1 -- 172.21.15.1:0/367354554 >> 172.21.15.1:0/367354554 conn(0x7f28400617a0 msgr2=0x7f2840061bb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:28:10.125 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:28:10.121+0000 7f2846d2b640 1 -- 172.21.15.1:0/367354554 shutdown_connections 2024-03-25T18:28:10.125 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:28:10.121+0000 7f2846d2b640 1 -- 172.21.15.1:0/367354554 wait complete. 2024-03-25T18:28:10.125 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:28:10.122+0000 7f2846d2b640 10 monclient: build_initial_monmap 2024-03-25T18:28:10.125 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:28:10.122+0000 7f2846d2b640 10 monclient: monmap: 2024-03-25T18:28:10.125 INFO:teuthology.misc.health.smithi001.stderr:epoch 0 2024-03-25T18:28:10.125 INFO:teuthology.misc.health.smithi001.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T18:28:10.125 INFO:teuthology.misc.health.smithi001.stderr:last_changed 0.000000 2024-03-25T18:28:10.125 INFO:teuthology.misc.health.smithi001.stderr:created 0.000000 2024-03-25T18:28:10.125 INFO:teuthology.misc.health.smithi001.stderr:min_mon_release 0 (unknown) 2024-03-25T18:28:10.125 INFO:teuthology.misc.health.smithi001.stderr:election_strategy: 1 2024-03-25T18:28:10.125 INFO:teuthology.misc.health.smithi001.stderr:0: v1:172.21.15.1:6789/0 mon.noname-a 2024-03-25T18:28:10.125 INFO:teuthology.misc.health.smithi001.stderr:1: v1:172.21.15.115:6789/0 mon.noname-b 2024-03-25T18:28:10.126 INFO:teuthology.misc.health.smithi001.stderr:2: v1:172.21.15.1:6790/0 mon.noname-c 2024-03-25T18:28:10.126 INFO:teuthology.misc.health.smithi001.stderr: 2024-03-25T18:28:10.126 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:28:10.122+0000 7f2846d2b640 1 librados: starting msgr at 2024-03-25T18:28:10.126 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:28:10.122+0000 7f2846d2b640 1 librados: starting objecter 2024-03-25T18:28:10.126 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:28:10.122+0000 7f2846d2b640 1 Processor -- start 2024-03-25T18:28:10.126 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:28:10.122+0000 7f2846d2b640 1 -- start start 2024-03-25T18:28:10.126 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:28:10.122+0000 7f2846d2b640 1 librados: setting wanted keys 2024-03-25T18:28:10.126 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:28:10.122+0000 7f2846d2b640 1 librados: calling monclient init 2024-03-25T18:28:10.126 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:28:10.122+0000 7f2846d2b640 10 monclient: init 2024-03-25T18:28:10.127 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:28:10.123+0000 7f2846d2b640 10 monclient: _reopen_session rank -1 2024-03-25T18:28:10.127 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:28:10.123+0000 7f2846d2b640 10 monclient: _add_conns ranks=[1,0,2] 2024-03-25T18:28:10.127 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:28:10.123+0000 7f2846d2b640 10 monclient(hunting): picked mon.noname-b con 0x7f28400b4670 addr v1:172.21.15.115:6789/0 2024-03-25T18:28:10.127 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:28:10.123+0000 7f2846d2b640 10 monclient(hunting): picked mon.noname-a con 0x7f28400b4ff0 addr v1:172.21.15.1:6789/0 2024-03-25T18:28:10.127 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:28:10.123+0000 7f2846d2b640 10 monclient(hunting): picked mon.noname-c con 0x7f284013c480 addr v1:172.21.15.1:6790/0 2024-03-25T18:28:10.127 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:28:10.123+0000 7f2846d2b640 1 -- --> v1:172.21.15.1:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2840144cc0 con 0x7f28400b4ff0 2024-03-25T18:28:10.128 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:28:10.123+0000 7f2846d2b640 1 -- --> v1:172.21.15.115:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f284013d730 con 0x7f28400b4670 2024-03-25T18:28:10.128 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:28:10.123+0000 7f2846d2b640 1 -- --> v1:172.21.15.1:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f28401440b0 con 0x7f284013c480 2024-03-25T18:28:10.128 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:28:10.123+0000 7f2846d2b640 10 monclient(hunting): _renew_subs 2024-03-25T18:28:10.128 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:28:10.123+0000 7f2846d2b640 10 monclient(hunting): authenticate will time out at 7570.594605s 2024-03-25T18:28:10.133 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:28:10.131+0000 7f28452c8640 1 --1- >> v1:172.21.15.1:6790/0 conn(0x7f284013c480 0x7f28401e2590 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.1:6790/0 says I am v1:172.21.15.1:60660/0 (socket says 172.21.15.1:60660) 2024-03-25T18:28:10.133 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:28:10.131+0000 7f28452c8640 1 -- 172.21.15.1:0/1339540365 learned_addr learned my addr 172.21.15.1:0/1339540365 (peer_addr_for_me v1:172.21.15.1:0/0) 2024-03-25T18:28:10.135 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:28:10.133+0000 7f283d7fa640 10 client.?.objecter ms_handle_connect 0x7f284013c480 2024-03-25T18:28:10.135 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:28:10.134+0000 7f283d7fa640 10 client.?.objecter resend_mon_ops 2024-03-25T18:28:10.144 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:28:10.142+0000 7f283d7fa640 10 client.?.objecter ms_handle_connect 0x7f28400b4670 2024-03-25T18:28:10.144 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:28:10.142+0000 7f283d7fa640 10 client.?.objecter resend_mon_ops 2024-03-25T18:28:10.145 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:28:10.143+0000 7f283d7fa640 10 client.?.objecter ms_handle_connect 0x7f28400b4ff0 2024-03-25T18:28:10.145 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:28:10.143+0000 7f283d7fa640 10 client.?.objecter resend_mon_ops 2024-03-25T18:28:10.154 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:28:10.152+0000 7f283d7fa640 1 -- 172.21.15.1:0/1339540365 <== mon.2 v1:172.21.15.1:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2265098792 0 0) 0x7f28401440b0 con 0x7f284013c480 2024-03-25T18:28:10.154 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:28:10.152+0000 7f283d7fa640 10 monclient(hunting): _negotiate 2024-03-25T18:28:10.154 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:28:10.152+0000 7f283d7fa640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:28:10.154 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:28:10.152+0000 7f283d7fa640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:28:10.154 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:28:10.152+0000 7f283d7fa640 10 monclient(hunting): my global_id is 4289 2024-03-25T18:28:10.154 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:28:10.152+0000 7f283d7fa640 1 -- 172.21.15.1:0/1339540365 --> v1:172.21.15.1:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f2824004eb0 con 0x7f284013c480 2024-03-25T18:28:10.155 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:28:10.153+0000 7f283d7fa640 1 -- 172.21.15.1:0/1339540365 <== mon.1 v1:172.21.15.115:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1671766513 0 0) 0x7f284013d730 con 0x7f28400b4670 2024-03-25T18:28:10.155 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:28:10.153+0000 7f283d7fa640 10 monclient(hunting): _negotiate 2024-03-25T18:28:10.155 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:28:10.153+0000 7f283d7fa640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:28:10.155 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:28:10.153+0000 7f283d7fa640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:28:10.155 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:28:10.153+0000 7f283d7fa640 10 monclient(hunting): my global_id is 4282 2024-03-25T18:28:10.155 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:28:10.153+0000 7f283d7fa640 1 -- 172.21.15.1:0/1339540365 --> v1:172.21.15.115:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f28401440b0 con 0x7f28400b4670 2024-03-25T18:28:10.163 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:28:10.161+0000 7f283d7fa640 1 -- 172.21.15.1:0/1339540365 <== mon.2 v1:172.21.15.1:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 331344565 0 0) 0x7f2824004eb0 con 0x7f284013c480 2024-03-25T18:28:10.163 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:28:10.161+0000 7f283d7fa640 1 -- 172.21.15.1:0/1339540365 --> v1:172.21.15.1:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f284013d730 con 0x7f284013c480 2024-03-25T18:28:10.163 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:28:10.161+0000 7f283d7fa640 1 -- 172.21.15.1:0/1339540365 <== mon.2 v1:172.21.15.1:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1586965025 0 0) 0x7f2838001ec0 con 0x7f284013c480 2024-03-25T18:28:10.164 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:28:10.161+0000 7f283d7fa640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:28:10.164 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:28:10.161+0000 7f283d7fa640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-03-25T18:28:10.164 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:28:10.161+0000 7f283d7fa640 10 monclient(hunting): dump: 2024-03-25T18:28:10.164 INFO:teuthology.misc.health.smithi001.stderr:epoch 1 2024-03-25T18:28:10.164 INFO:teuthology.misc.health.smithi001.stderr:fsid c8ea72cf-9e99-4383-9b52-6c23d365fdfc 2024-03-25T18:28:10.164 INFO:teuthology.misc.health.smithi001.stderr:last_changed 2024-03-25T18:25:50.510519+0000 2024-03-25T18:28:10.164 INFO:teuthology.misc.health.smithi001.stderr:created 2024-03-25T18:25:50.510519+0000 2024-03-25T18:28:10.164 INFO:teuthology.misc.health.smithi001.stderr:min_mon_release 19 (squid) 2024-03-25T18:28:10.164 INFO:teuthology.misc.health.smithi001.stderr:election_strategy: 3 2024-03-25T18:28:10.164 INFO:teuthology.misc.health.smithi001.stderr:0: v1:172.21.15.1:6789/0 mon.a 2024-03-25T18:28:10.164 INFO:teuthology.misc.health.smithi001.stderr:1: v1:172.21.15.115:6789/0 mon.b 2024-03-25T18:28:10.164 INFO:teuthology.misc.health.smithi001.stderr:2: v1:172.21.15.1:6790/0 mon.c 2024-03-25T18:28:10.164 INFO:teuthology.misc.health.smithi001.stderr: 2024-03-25T18:28:10.164 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:28:10.161+0000 7f283d7fa640 10 monclient(hunting): _finish_auth 0 2024-03-25T18:28:10.165 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:28:10.161+0000 7f283d7fa640 1 -- 172.21.15.1:0/1339540365 <== mon.0 v1:172.21.15.1:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1161807518 0 0) 0x7f2840144cc0 con 0x7f28400b4ff0 2024-03-25T18:28:10.165 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:28:10.161+0000 7f283d7fa640 10 monclient(hunting): _negotiate 2024-03-25T18:28:10.165 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:28:10.161+0000 7f283d7fa640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:28:10.165 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:28:10.161+0000 7f283d7fa640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:28:10.165 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:28:10.161+0000 7f283d7fa640 10 monclient(hunting): my global_id is 4287 2024-03-25T18:28:10.165 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:28:10.161+0000 7f283d7fa640 1 -- 172.21.15.1:0/1339540365 --> v1:172.21.15.1:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f2824004eb0 con 0x7f28400b4ff0 2024-03-25T18:28:10.175 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:28:10.174+0000 7f283d7fa640 1 -- 172.21.15.1:0/1339540365 <== mon.1 v1:172.21.15.115:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1414476110 0 0) 0x7f28401440b0 con 0x7f28400b4670 2024-03-25T18:28:10.175 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:28:10.174+0000 7f283d7fa640 1 -- 172.21.15.1:0/1339540365 --> v1:172.21.15.115:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f2840144cc0 con 0x7f28400b4670 2024-03-25T18:28:10.175 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:28:10.174+0000 7f283d7fa640 1 -- 172.21.15.1:0/1339540365 <== mon.1 v1:172.21.15.115:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1586965025 0 0) 0x7f2834001970 con 0x7f28400b4670 2024-03-25T18:28:10.175 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:28:10.174+0000 7f283d7fa640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:28:10.175 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:28:10.174+0000 7f283d7fa640 10 monclient(hunting): got monmap 1 from mon.b (according to old e1) 2024-03-25T18:28:10.176 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:28:10.174+0000 7f283d7fa640 10 monclient(hunting): dump: 2024-03-25T18:28:10.176 INFO:teuthology.misc.health.smithi001.stderr:epoch 1 2024-03-25T18:28:10.176 INFO:teuthology.misc.health.smithi001.stderr:fsid c8ea72cf-9e99-4383-9b52-6c23d365fdfc 2024-03-25T18:28:10.176 INFO:teuthology.misc.health.smithi001.stderr:last_changed 2024-03-25T18:25:50.510519+0000 2024-03-25T18:28:10.176 INFO:teuthology.misc.health.smithi001.stderr:created 2024-03-25T18:25:50.510519+0000 2024-03-25T18:28:10.176 INFO:teuthology.misc.health.smithi001.stderr:min_mon_release 19 (squid) 2024-03-25T18:28:10.176 INFO:teuthology.misc.health.smithi001.stderr:election_strategy: 3 2024-03-25T18:28:10.176 INFO:teuthology.misc.health.smithi001.stderr:0: v1:172.21.15.1:6789/0 mon.a 2024-03-25T18:28:10.176 INFO:teuthology.misc.health.smithi001.stderr:1: v1:172.21.15.115:6789/0 mon.b 2024-03-25T18:28:10.176 INFO:teuthology.misc.health.smithi001.stderr:2: v1:172.21.15.1:6790/0 mon.c 2024-03-25T18:28:10.176 INFO:teuthology.misc.health.smithi001.stderr: 2024-03-25T18:28:10.181 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:28:10.178+0000 7f283d7fa640 1 -- 172.21.15.1:0/1339540365 <== mon.2 v1:172.21.15.1:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 663733768 0 0) 0x7f284013d730 con 0x7f284013c480 2024-03-25T18:28:10.181 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:28:10.179+0000 7f283d7fa640 1 -- 172.21.15.1:0/1339540365 >> v1:172.21.15.115:6789/0 conn(0x7f28400b4670 legacy=0x7f284013c910 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:28:10.181 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:28:10.179+0000 7f283d7fa640 1 -- 172.21.15.1:0/1339540365 >> v1:172.21.15.1:6789/0 conn(0x7f28400b4ff0 legacy=0x7f284013d020 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:28:10.181 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:28:10.179+0000 7f283d7fa640 10 monclient: _finish_hunting 0 2024-03-25T18:28:10.182 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:28:10.179+0000 7f283d7fa640 1 monclient: found mon.c 2024-03-25T18:28:10.182 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:28:10.179+0000 7f283d7fa640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:28:10.182 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:28:10.179+0000 7f283d7fa640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.1:6790/0 2024-03-25T18:28:10.182 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:28:10.179+0000 7f283d7fa640 1 -- 172.21.15.1:0/1339540365 --> v1:172.21.15.1:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2840146640 con 0x7f284013c480 2024-03-25T18:28:10.182 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:28:10.179+0000 7f283d7fa640 10 monclient: _finish_auth 0 2024-03-25T18:28:10.182 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:28:10.179+0000 7f283d7fa640 10 monclient: _check_auth_tickets 2024-03-25T18:28:10.182 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:28:10.179+0000 7f283d7fa640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:28:10.182 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:28:10.179+0000 7f2846d2b640 5 monclient: authenticate success, global_id 4289 2024-03-25T18:28:10.183 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:28:10.179+0000 7f2846d2b640 10 monclient: _renew_subs 2024-03-25T18:28:10.183 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:28:10.179+0000 7f2846d2b640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.1:6790/0 2024-03-25T18:28:10.183 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:28:10.179+0000 7f2846d2b640 1 -- 172.21.15.1:0/1339540365 --> v1:172.21.15.1:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2840144f30 con 0x7f284013c480 2024-03-25T18:28:10.183 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:28:10.179+0000 7f2846d2b640 10 client.4289.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T18:28:10.183 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:28:10.179+0000 7f2846d2b640 10 monclient: _renew_subs 2024-03-25T18:28:10.183 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:28:10.179+0000 7f2846d2b640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.1:6790/0 2024-03-25T18:28:10.183 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:28:10.179+0000 7f2846d2b640 1 -- 172.21.15.1:0/1339540365 --> v1:172.21.15.1:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f2840143660 con 0x7f284013c480 2024-03-25T18:28:10.183 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:28:10.179+0000 7f2846d2b640 1 librados: init done 2024-03-25T18:28:10.183 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:28:10.180+0000 7f2846d2b640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T18:28:10.183 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:28:10.180+0000 7f2846d2b640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:28:10.183 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:28:10.180+0000 7f2846d2b640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.1:6790/0 2024-03-25T18:28:10.183 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:28:10.180+0000 7f2846d2b640 1 -- 172.21.15.1:0/1339540365 --> v1:172.21.15.1:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f2840145320 con 0x7f284013c480 2024-03-25T18:28:10.516 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:28:10.515+0000 7f283d7fa640 1 -- 172.21.15.1:0/1339540365 <== mon.2 v1:172.21.15.1:6790/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f2838004e10 con 0x7f284013c480 2024-03-25T18:28:10.516 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:28:10.515+0000 7f283d7fa640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:28:10.516 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:28:10.515+0000 7f283d7fa640 1 -- 172.21.15.1:0/1339540365 <== mon.2 v1:172.21.15.1:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1586965025 0 0) 0x7f28380015d0 con 0x7f284013c480 2024-03-25T18:28:10.516 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:28:10.515+0000 7f283d7fa640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:28:10.516 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:28:10.515+0000 7f283d7fa640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-25T18:28:10.517 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:28:10.515+0000 7f283d7fa640 10 monclient: dump: 2024-03-25T18:28:10.517 INFO:teuthology.misc.health.smithi001.stderr:epoch 1 2024-03-25T18:28:10.517 INFO:teuthology.misc.health.smithi001.stderr:fsid c8ea72cf-9e99-4383-9b52-6c23d365fdfc 2024-03-25T18:28:10.517 INFO:teuthology.misc.health.smithi001.stderr:last_changed 2024-03-25T18:25:50.510519+0000 2024-03-25T18:28:10.517 INFO:teuthology.misc.health.smithi001.stderr:created 2024-03-25T18:25:50.510519+0000 2024-03-25T18:28:10.517 INFO:teuthology.misc.health.smithi001.stderr:min_mon_release 19 (squid) 2024-03-25T18:28:10.517 INFO:teuthology.misc.health.smithi001.stderr:election_strategy: 3 2024-03-25T18:28:10.517 INFO:teuthology.misc.health.smithi001.stderr:0: v1:172.21.15.1:6789/0 mon.a 2024-03-25T18:28:10.517 INFO:teuthology.misc.health.smithi001.stderr:1: v1:172.21.15.115:6789/0 mon.b 2024-03-25T18:28:10.517 INFO:teuthology.misc.health.smithi001.stderr:2: v1:172.21.15.1:6790/0 mon.c 2024-03-25T18:28:10.517 INFO:teuthology.misc.health.smithi001.stderr: 2024-03-25T18:28:10.517 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:28:10.515+0000 7f283effd640 4 set_mon_vals no callback set 2024-03-25T18:28:10.518 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:28:10.516+0000 7f283d7fa640 1 -- 172.21.15.1:0/1339540365 <== mon.2 v1:172.21.15.1:6790/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 21787975 0 0) 0x7f283801af90 con 0x7f284013c480 2024-03-25T18:28:10.518 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:28:10.517+0000 7f2844ac7640 10 monclient: get_auth_request con 0x7f282406df70 auth_method 0 2024-03-25T18:28:10.518 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:28:10.517+0000 7f283d7fa640 10 client.4289.objecter ms_handle_connect 0x7f282406df70 2024-03-25T18:28:10.526 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:28:10.524+0000 7f283d7fa640 1 -- 172.21.15.1:0/1339540365 <== mon.2 v1:172.21.15.1:6790/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 996254230 0 0) 0x7f2838054c00 con 0x7f284013c480 2024-03-25T18:28:10.526 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:28:10.524+0000 7f283d7fa640 10 client.4289.objecter ms_dispatch 0x7f2840000b90 osd_map(10..10 src has 1..10) v4 2024-03-25T18:28:10.526 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:28:10.524+0000 7f283d7fa640 3 client.4289.objecter handle_osd_map got epochs [10,10] > 0 2024-03-25T18:28:10.526 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:28:10.524+0000 7f283d7fa640 3 client.4289.objecter handle_osd_map decoding full epoch 10 2024-03-25T18:28:10.526 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:28:10.524+0000 7f283d7fa640 20 client.4289.objecter dump_active .. 0 homeless 2024-03-25T18:28:10.526 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:28:10.524+0000 7f283d7fa640 1 -- 172.21.15.1:0/1339540365 <== mon.2 v1:172.21.15.1:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f2838050540 con 0x7f284013c480 2024-03-25T18:28:10.526 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:28:10.525+0000 7f283d7fa640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:28:10.526 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:28:10.525+0000 7f283d7fa640 10 monclient: _finish_command 1 = system:0 2024-03-25T18:28:10.777 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:28:10.776+0000 7f2846d2b640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:28:10.777 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:28:10.776+0000 7f2846d2b640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:28:10.777 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:28:10.776+0000 7f2846d2b640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.1:6790/0 2024-03-25T18:28:10.777 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:28:10.776+0000 7f2846d2b640 1 -- 172.21.15.1:0/1339540365 --> v1:172.21.15.1:6790/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f284006dbd0 con 0x7f284013c480 2024-03-25T18:28:10.805 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:28:10.804+0000 7f283d7fa640 1 -- 172.21.15.1:0/1339540365 <== mon.2 v1:172.21.15.1:6790/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 4092213348) 0x7f2838052640 con 0x7f284013c480 2024-03-25T18:28:10.805 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:28:10.804+0000 7f283d7fa640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:28:10.805 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:28:10.804+0000 7f283d7fa640 10 monclient: _finish_command 2 = system:0 2024-03-25T18:28:10.805 INFO:teuthology.misc.health.smithi001.stdout: 2024-03-25T18:28:10.805 INFO:teuthology.misc.health.smithi001.stdout:{"epoch":10,"fsid":"c8ea72cf-9e99-4383-9b52-6c23d365fdfc","created":"2024-03-25T18:26:38.856638+0000","modified":"2024-03-25T18:26:44.224208+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T18:26:44.224208+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"b9f34922-3a84-49be-8426-807c1eceaefc","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"42bf306f-55e3-4634-9874-8b4b73ac6644","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"292cb23a-18a9-4207-a7cc-4d5727feaec0","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"45e0e9ab-898d-4673-9590-b4117bc5e84a","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"831a5a09-73ab-4a61-9872-088e76bdbc66","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"88cd903d-3b55-4840-9396-6e3cfe005282","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"89db7355-bc9d-46f4-8ab0-e8d9e744e026","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"cac47bee-0914-4f08-896f-348838882866","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T18:28:10.807 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:28:10.806+0000 7f2846d2b640 10 librados: watch_flush enter 2024-03-25T18:28:10.807 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:28:10.806+0000 7f2846d2b640 10 librados: watch_flush exit 2024-03-25T18:28:10.807 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:28:10.806+0000 7f2846d2b640 20 client.4289.objecter shutdown clearing up homeless session... 2024-03-25T18:28:10.808 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:28:10.806+0000 7f2846d2b640 10 client.4289.objecter successfully canceled tick 2024-03-25T18:28:10.808 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:28:10.806+0000 7f2846d2b640 1 -- 172.21.15.1:0/1339540365 >> v1:172.21.15.115:6800/4237379176 conn(0x7f282406df70 legacy=0x7f2824070430 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:28:10.808 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:28:10.806+0000 7f2846d2b640 10 monclient: shutdown 2024-03-25T18:28:10.808 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:28:10.806+0000 7f2846d2b640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:28:10.808 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:28:10.806+0000 7f2846d2b640 1 -- 172.21.15.1:0/1339540365 >> v1:172.21.15.1:6790/0 conn(0x7f284013c480 legacy=0x7f28401e2590 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:28:10.808 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:28:10.806+0000 7f2846d2b640 1 -- 172.21.15.1:0/1339540365 shutdown_connections 2024-03-25T18:28:10.808 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:28:10.806+0000 7f2846d2b640 1 -- 172.21.15.1:0/1339540365 >> 172.21.15.1:0/1339540365 conn(0x7f28400617a0 msgr2=0x7f28401407a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:28:10.808 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:28:10.806+0000 7f2846d2b640 1 -- 172.21.15.1:0/1339540365 shutdown_connections 2024-03-25T18:28:10.808 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:28:10.807+0000 7f2846d2b640 1 -- 172.21.15.1:0/1339540365 wait complete. 2024-03-25T18:28:10.808 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:28:10.807+0000 7f2846d2b640 1 librados: shutdown 2024-03-25T18:28:10.818 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T18:28:16.126 INFO:tasks.ceph.mon.a.smithi001.stderr:==00:00:01:54.684 34876== Warning: unimplemented fcntl command: 1036 2024-03-25T18:28:16.129 INFO:tasks.ceph.mon.a.smithi001.stderr:==00:00:01:54.688 34876== Warning: unimplemented fcntl command: 1036 2024-03-25T18:28:16.173 INFO:tasks.ceph.mon.a.smithi001.stderr:==00:00:01:54.732 34876== Warning: unimplemented fcntl command: 1036 2024-03-25T18:28:16.180 INFO:tasks.ceph.mon.b.smithi115.stderr:==00:00:01:54.664 34836== Warning: unimplemented fcntl command: 1036 2024-03-25T18:28:16.184 INFO:tasks.ceph.mon.b.smithi115.stderr:==00:00:01:54.668 34836== Warning: unimplemented fcntl command: 1036 2024-03-25T18:28:16.201 INFO:tasks.ceph.mon.c.smithi001.stderr:==00:00:01:54.759 34877== Warning: unimplemented fcntl command: 1036 2024-03-25T18:28:16.210 INFO:tasks.ceph.mon.b.smithi115.stderr:==00:00:01:54.693 34836== Warning: unimplemented fcntl command: 1036 2024-03-25T18:28:16.222 INFO:tasks.ceph.mon.c.smithi001.stderr:==00:00:01:54.781 34877== Warning: unimplemented fcntl command: 1036 2024-03-25T18:28:16.258 INFO:tasks.ceph.mon.c.smithi001.stderr:==00:00:01:54.815 34877== Warning: unimplemented fcntl command: 1036 2024-03-25T18:28:17.625 DEBUG:teuthology.orchestra.run.smithi001:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T18:28:17.742 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:28:17.742+0000 7f685eba3640 10 monclient: get_monmap_and_config 2024-03-25T18:28:17.742 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:28:17.742+0000 7f685eba3640 10 monclient: build_initial_monmap 2024-03-25T18:28:17.743 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:28:17.742+0000 7f685eba3640 10 monclient: monmap: 2024-03-25T18:28:17.743 INFO:teuthology.misc.health.smithi001.stderr:epoch 0 2024-03-25T18:28:17.743 INFO:teuthology.misc.health.smithi001.stderr:fsid c8ea72cf-9e99-4383-9b52-6c23d365fdfc 2024-03-25T18:28:17.743 INFO:teuthology.misc.health.smithi001.stderr:last_changed 2024-03-25T18:28:17.742644+0000 2024-03-25T18:28:17.743 INFO:teuthology.misc.health.smithi001.stderr:created 2024-03-25T18:28:17.742644+0000 2024-03-25T18:28:17.743 INFO:teuthology.misc.health.smithi001.stderr:min_mon_release 0 (unknown) 2024-03-25T18:28:17.743 INFO:teuthology.misc.health.smithi001.stderr:election_strategy: 3 2024-03-25T18:28:17.743 INFO:teuthology.misc.health.smithi001.stderr:0: v1:172.21.15.1:6789/0 mon.noname-a 2024-03-25T18:28:17.743 INFO:teuthology.misc.health.smithi001.stderr:1: v1:172.21.15.115:6789/0 mon.noname-c 2024-03-25T18:28:17.743 INFO:teuthology.misc.health.smithi001.stderr:2: v1:172.21.15.1:6790/0 mon.noname-b 2024-03-25T18:28:17.743 INFO:teuthology.misc.health.smithi001.stderr: 2024-03-25T18:28:17.744 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:28:17.742+0000 7f685eba3640 1 Processor -- start 2024-03-25T18:28:17.745 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:28:17.743+0000 7f685eba3640 1 -- start start 2024-03-25T18:28:17.745 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:28:17.743+0000 7f685eba3640 10 monclient: init 2024-03-25T18:28:17.745 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:28:17.743+0000 7f685eba3640 10 monclient: _reopen_session rank -1 2024-03-25T18:28:17.745 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:28:17.743+0000 7f685eba3640 10 monclient: _add_conns ranks=[1,2,0] 2024-03-25T18:28:17.745 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:28:17.743+0000 7f685eba3640 10 monclient(hunting): picked mon.noname-c con 0x7f68580b4ff0 addr v1:172.21.15.115:6789/0 2024-03-25T18:28:17.745 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:28:17.743+0000 7f685eba3640 10 monclient(hunting): picked mon.noname-b con 0x7f68580b4670 addr v1:172.21.15.1:6790/0 2024-03-25T18:28:17.745 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:28:17.743+0000 7f685eba3640 10 monclient(hunting): picked mon.noname-a con 0x7f685813c480 addr v1:172.21.15.1:6789/0 2024-03-25T18:28:17.745 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:28:17.743+0000 7f685eba3640 1 -- --> v1:172.21.15.1:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f68580ab0e0 con 0x7f685813c480 2024-03-25T18:28:17.745 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:28:17.743+0000 7f685eba3640 1 -- --> v1:172.21.15.115:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f68580ab3c0 con 0x7f68580b4ff0 2024-03-25T18:28:17.745 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:28:17.743+0000 7f685eba3640 1 -- --> v1:172.21.15.1:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f68580a9f30 con 0x7f68580b4670 2024-03-25T18:28:17.746 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:28:17.743+0000 7f685eba3640 10 monclient(hunting): _renew_subs 2024-03-25T18:28:17.746 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:28:17.743+0000 7f685eba3640 10 monclient(hunting): authenticate will time out at 7578.213479s 2024-03-25T18:28:17.748 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:28:17.748+0000 7f685d140640 1 --1- >> v1:172.21.15.1:6789/0 conn(0x7f685813c480 0x7f685814dc00 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.1:6789/0 says I am v1:172.21.15.1:39414/0 (socket says 172.21.15.1:39414) 2024-03-25T18:28:17.748 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:28:17.748+0000 7f685d140640 1 -- 172.21.15.1:0/1113633690 learned_addr learned my addr 172.21.15.1:0/1113633690 (peer_addr_for_me v1:172.21.15.1:0/0) 2024-03-25T18:28:17.761 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:28:17.760+0000 7f68577fe640 1 -- 172.21.15.1:0/1113633690 <== mon.0 v1:172.21.15.1:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2364501186 0 0) 0x7f68580ab0e0 con 0x7f685813c480 2024-03-25T18:28:17.761 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:28:17.760+0000 7f68577fe640 10 monclient(hunting): _negotiate 2024-03-25T18:28:17.761 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:28:17.760+0000 7f68577fe640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:28:17.761 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:28:17.760+0000 7f68577fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:28:17.762 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:28:17.760+0000 7f68577fe640 10 monclient(hunting): my global_id is 4290 2024-03-25T18:28:17.762 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:28:17.760+0000 7f68577fe640 1 -- 172.21.15.1:0/1113633690 --> v1:172.21.15.1:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f6848003520 con 0x7f685813c480 2024-03-25T18:28:17.769 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:28:17.768+0000 7f68577fe640 1 -- 172.21.15.1:0/1113633690 <== mon.1 v1:172.21.15.115:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1657237730 0 0) 0x7f68580ab3c0 con 0x7f68580b4ff0 2024-03-25T18:28:17.769 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:28:17.768+0000 7f68577fe640 10 monclient(hunting): _negotiate 2024-03-25T18:28:17.769 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:28:17.768+0000 7f68577fe640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:28:17.769 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:28:17.768+0000 7f68577fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:28:17.769 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:28:17.768+0000 7f68577fe640 10 monclient(hunting): my global_id is 4285 2024-03-25T18:28:17.769 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:28:17.768+0000 7f68577fe640 1 -- 172.21.15.1:0/1113633690 --> v1:172.21.15.115:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f68580ab0e0 con 0x7f68580b4ff0 2024-03-25T18:28:17.779 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:28:17.778+0000 7f68577fe640 1 -- 172.21.15.1:0/1113633690 <== mon.1 v1:172.21.15.115:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1137377099 0 0) 0x7f68580ab0e0 con 0x7f68580b4ff0 2024-03-25T18:28:17.779 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:28:17.778+0000 7f68577fe640 1 -- 172.21.15.1:0/1113633690 >> v1:172.21.15.1:6790/0 conn(0x7f68580b4670 legacy=0x7f68581425c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:28:17.779 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:28:17.778+0000 7f68577fe640 1 -- 172.21.15.1:0/1113633690 >> v1:172.21.15.1:6789/0 conn(0x7f685813c480 legacy=0x7f685814dc00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:28:17.779 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:28:17.778+0000 7f68577fe640 10 monclient: _finish_hunting 0 2024-03-25T18:28:17.779 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:28:17.778+0000 7f68577fe640 1 monclient: found mon.noname-c 2024-03-25T18:28:17.779 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:28:17.778+0000 7f68577fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:28:17.779 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:28:17.778+0000 7f68577fe640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.115:6789/0 2024-03-25T18:28:17.779 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:28:17.778+0000 7f68577fe640 1 -- 172.21.15.1:0/1113633690 --> v1:172.21.15.115:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f68580a9b70 con 0x7f68580b4ff0 2024-03-25T18:28:17.779 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:28:17.778+0000 7f68577fe640 10 monclient: _finish_auth 0 2024-03-25T18:28:17.779 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:28:17.778+0000 7f68577fe640 10 monclient: _check_auth_tickets 2024-03-25T18:28:17.780 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:28:17.778+0000 7f68577fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:28:17.780 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:28:17.778+0000 7f68577fe640 1 -- 172.21.15.1:0/1113633690 <== mon.1 v1:172.21.15.115:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1586965025 0 0) 0x7f684c001670 con 0x7f68580b4ff0 2024-03-25T18:28:17.780 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:28:17.778+0000 7f68577fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:28:17.780 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:28:17.778+0000 7f68577fe640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-03-25T18:28:17.780 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:28:17.778+0000 7f68577fe640 10 monclient: dump: 2024-03-25T18:28:17.780 INFO:teuthology.misc.health.smithi001.stderr:epoch 1 2024-03-25T18:28:17.780 INFO:teuthology.misc.health.smithi001.stderr:fsid c8ea72cf-9e99-4383-9b52-6c23d365fdfc 2024-03-25T18:28:17.780 INFO:teuthology.misc.health.smithi001.stderr:last_changed 2024-03-25T18:25:50.510519+0000 2024-03-25T18:28:17.780 INFO:teuthology.misc.health.smithi001.stderr:created 2024-03-25T18:25:50.510519+0000 2024-03-25T18:28:17.780 INFO:teuthology.misc.health.smithi001.stderr:min_mon_release 19 (squid) 2024-03-25T18:28:17.780 INFO:teuthology.misc.health.smithi001.stderr:election_strategy: 3 2024-03-25T18:28:17.780 INFO:teuthology.misc.health.smithi001.stderr:0: v1:172.21.15.1:6789/0 mon.a 2024-03-25T18:28:17.780 INFO:teuthology.misc.health.smithi001.stderr:1: v1:172.21.15.115:6789/0 mon.b 2024-03-25T18:28:17.780 INFO:teuthology.misc.health.smithi001.stderr:2: v1:172.21.15.1:6790/0 mon.c 2024-03-25T18:28:17.780 INFO:teuthology.misc.health.smithi001.stderr: 2024-03-25T18:28:17.781 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:28:17.778+0000 7f685eba3640 5 monclient: authenticate success, global_id 4285 2024-03-25T18:28:17.781 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:28:17.778+0000 7f685eba3640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T18:28:17.787 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:28:17.787+0000 7f68577fe640 1 -- 172.21.15.1:0/1113633690 <== mon.1 v1:172.21.15.115:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f684c0017d0 con 0x7f68580b4ff0 2024-03-25T18:28:17.787 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:28:17.787+0000 7f68577fe640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:28:17.787 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:28:17.787+0000 7f68577fe640 1 -- 172.21.15.1:0/1113633690 <== mon.1 v1:172.21.15.115:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1586965025 0 0) 0x7f684c001b60 con 0x7f68580b4ff0 2024-03-25T18:28:17.788 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:28:17.787+0000 7f68577fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:28:17.788 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:28:17.787+0000 7f68577fe640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-25T18:28:17.788 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:28:17.787+0000 7f68577fe640 10 monclient: dump: 2024-03-25T18:28:17.788 INFO:teuthology.misc.health.smithi001.stderr:epoch 1 2024-03-25T18:28:17.788 INFO:teuthology.misc.health.smithi001.stderr:fsid c8ea72cf-9e99-4383-9b52-6c23d365fdfc 2024-03-25T18:28:17.788 INFO:teuthology.misc.health.smithi001.stderr:last_changed 2024-03-25T18:25:50.510519+0000 2024-03-25T18:28:17.788 INFO:teuthology.misc.health.smithi001.stderr:created 2024-03-25T18:25:50.510519+0000 2024-03-25T18:28:17.788 INFO:teuthology.misc.health.smithi001.stderr:min_mon_release 19 (squid) 2024-03-25T18:28:17.788 INFO:teuthology.misc.health.smithi001.stderr:election_strategy: 3 2024-03-25T18:28:17.788 INFO:teuthology.misc.health.smithi001.stderr:0: v1:172.21.15.1:6789/0 mon.a 2024-03-25T18:28:17.789 INFO:teuthology.misc.health.smithi001.stderr:1: v1:172.21.15.115:6789/0 mon.b 2024-03-25T18:28:17.789 INFO:teuthology.misc.health.smithi001.stderr:2: v1:172.21.15.1:6790/0 mon.c 2024-03-25T18:28:17.789 INFO:teuthology.misc.health.smithi001.stderr: 2024-03-25T18:28:17.789 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:28:17.787+0000 7f685eba3640 10 monclient: get_monmap_and_config success 2024-03-25T18:28:17.789 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:28:17.787+0000 7f685eba3640 4 set_mon_vals no callback set 2024-03-25T18:28:17.792 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:28:17.787+0000 7f685eba3640 10 monclient: shutdown 2024-03-25T18:28:17.792 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:28:17.788+0000 7f685eba3640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:28:17.792 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:28:17.788+0000 7f685eba3640 1 -- 172.21.15.1:0/1113633690 >> v1:172.21.15.115:6789/0 conn(0x7f68580b4ff0 legacy=0x7f685813bd70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:28:17.792 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:28:17.788+0000 7f685eba3640 1 -- 172.21.15.1:0/1113633690 shutdown_connections 2024-03-25T18:28:17.792 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:28:17.788+0000 7f685eba3640 1 -- 172.21.15.1:0/1113633690 >> 172.21.15.1:0/1113633690 conn(0x7f68580617a0 msgr2=0x7f6858061bb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:28:17.793 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:28:17.788+0000 7f685eba3640 1 -- 172.21.15.1:0/1113633690 shutdown_connections 2024-03-25T18:28:17.793 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:28:17.788+0000 7f685eba3640 1 -- 172.21.15.1:0/1113633690 wait complete. 2024-03-25T18:28:17.793 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:28:17.789+0000 7f685eba3640 10 monclient: build_initial_monmap 2024-03-25T18:28:17.793 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:28:17.789+0000 7f685eba3640 10 monclient: monmap: 2024-03-25T18:28:17.793 INFO:teuthology.misc.health.smithi001.stderr:epoch 0 2024-03-25T18:28:17.793 INFO:teuthology.misc.health.smithi001.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T18:28:17.793 INFO:teuthology.misc.health.smithi001.stderr:last_changed 0.000000 2024-03-25T18:28:17.793 INFO:teuthology.misc.health.smithi001.stderr:created 0.000000 2024-03-25T18:28:17.793 INFO:teuthology.misc.health.smithi001.stderr:min_mon_release 0 (unknown) 2024-03-25T18:28:17.793 INFO:teuthology.misc.health.smithi001.stderr:election_strategy: 1 2024-03-25T18:28:17.793 INFO:teuthology.misc.health.smithi001.stderr:0: v1:172.21.15.1:6789/0 mon.noname-a 2024-03-25T18:28:17.793 INFO:teuthology.misc.health.smithi001.stderr:1: v1:172.21.15.115:6789/0 mon.noname-b 2024-03-25T18:28:17.793 INFO:teuthology.misc.health.smithi001.stderr:2: v1:172.21.15.1:6790/0 mon.noname-c 2024-03-25T18:28:17.793 INFO:teuthology.misc.health.smithi001.stderr: 2024-03-25T18:28:17.794 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:28:17.789+0000 7f685eba3640 1 librados: starting msgr at 2024-03-25T18:28:17.794 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:28:17.789+0000 7f685eba3640 1 librados: starting objecter 2024-03-25T18:28:17.794 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:28:17.789+0000 7f685eba3640 1 Processor -- start 2024-03-25T18:28:17.794 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:28:17.789+0000 7f685eba3640 1 -- start start 2024-03-25T18:28:17.794 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:28:17.789+0000 7f685eba3640 1 librados: setting wanted keys 2024-03-25T18:28:17.794 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:28:17.789+0000 7f685eba3640 1 librados: calling monclient init 2024-03-25T18:28:17.794 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:28:17.789+0000 7f685eba3640 10 monclient: init 2024-03-25T18:28:17.794 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:28:17.790+0000 7f685eba3640 10 monclient: _reopen_session rank -1 2024-03-25T18:28:17.794 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:28:17.790+0000 7f685eba3640 10 monclient: _add_conns ranks=[2,1,0] 2024-03-25T18:28:17.794 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:28:17.790+0000 7f685eba3640 10 monclient(hunting): picked mon.noname-c con 0x7f68580b4670 addr v1:172.21.15.1:6790/0 2024-03-25T18:28:17.794 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:28:17.790+0000 7f685eba3640 10 monclient(hunting): picked mon.noname-b con 0x7f68580b4ff0 addr v1:172.21.15.115:6789/0 2024-03-25T18:28:17.794 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:28:17.790+0000 7f685eba3640 10 monclient(hunting): picked mon.noname-a con 0x7f685813c480 addr v1:172.21.15.1:6789/0 2024-03-25T18:28:17.794 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:28:17.790+0000 7f685eba3640 1 -- --> v1:172.21.15.1:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6858144600 con 0x7f685813c480 2024-03-25T18:28:17.794 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:28:17.790+0000 7f685eba3640 1 -- --> v1:172.21.15.115:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f68581448e0 con 0x7f68580b4ff0 2024-03-25T18:28:17.795 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:28:17.791+0000 7f685eba3640 1 -- --> v1:172.21.15.1:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f685813f7e0 con 0x7f68580b4670 2024-03-25T18:28:17.795 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:28:17.791+0000 7f685eba3640 10 monclient(hunting): _renew_subs 2024-03-25T18:28:17.795 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:28:17.791+0000 7f685eba3640 10 monclient(hunting): authenticate will time out at 7578.260718s 2024-03-25T18:28:17.805 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:28:17.804+0000 7f685d140640 1 --1- >> v1:172.21.15.1:6789/0 conn(0x7f685813c480 0x7f68581e2170 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.1:6789/0 says I am v1:172.21.15.1:39466/0 (socket says 172.21.15.1:39466) 2024-03-25T18:28:17.805 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:28:17.804+0000 7f685d140640 1 -- 172.21.15.1:0/2924157926 learned_addr learned my addr 172.21.15.1:0/2924157926 (peer_addr_for_me v1:172.21.15.1:0/0) 2024-03-25T18:28:17.806 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:28:17.806+0000 7f68557fa640 10 client.?.objecter ms_handle_connect 0x7f68580b4ff0 2024-03-25T18:28:17.806 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:28:17.806+0000 7f68557fa640 10 client.?.objecter resend_mon_ops 2024-03-25T18:28:17.809 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:28:17.809+0000 7f68557fa640 10 client.?.objecter ms_handle_connect 0x7f685813c480 2024-03-25T18:28:17.809 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:28:17.809+0000 7f68557fa640 10 client.?.objecter resend_mon_ops 2024-03-25T18:28:17.819 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:28:17.818+0000 7f68557fa640 1 -- 172.21.15.1:0/2924157926 <== mon.1 v1:172.21.15.115:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3546614475 0 0) 0x7f68581448e0 con 0x7f68580b4ff0 2024-03-25T18:28:17.819 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:28:17.819+0000 7f68557fa640 10 monclient(hunting): _negotiate 2024-03-25T18:28:17.819 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:28:17.819+0000 7f68557fa640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:28:17.820 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:28:17.819+0000 7f68557fa640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:28:17.820 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:28:17.819+0000 7f68557fa640 10 monclient(hunting): my global_id is 4288 2024-03-25T18:28:17.820 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:28:17.819+0000 7f68557fa640 1 -- 172.21.15.1:0/2924157926 --> v1:172.21.15.115:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f683c004eb0 con 0x7f68580b4ff0 2024-03-25T18:28:17.828 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:28:17.827+0000 7f68557fa640 10 client.?.objecter ms_handle_connect 0x7f68580b4670 2024-03-25T18:28:17.828 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:28:17.827+0000 7f68557fa640 10 client.?.objecter resend_mon_ops 2024-03-25T18:28:17.830 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:28:17.829+0000 7f68557fa640 1 -- 172.21.15.1:0/2924157926 <== mon.0 v1:172.21.15.1:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 701866719 0 0) 0x7f6858144600 con 0x7f685813c480 2024-03-25T18:28:17.830 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:28:17.829+0000 7f68557fa640 10 monclient(hunting): _negotiate 2024-03-25T18:28:17.830 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:28:17.829+0000 7f68557fa640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:28:17.830 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:28:17.829+0000 7f68557fa640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:28:17.830 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:28:17.829+0000 7f68557fa640 10 monclient(hunting): my global_id is 4293 2024-03-25T18:28:17.830 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:28:17.829+0000 7f68557fa640 1 -- 172.21.15.1:0/2924157926 --> v1:172.21.15.1:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f68581448e0 con 0x7f685813c480 2024-03-25T18:28:17.836 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:28:17.836+0000 7f68557fa640 1 -- 172.21.15.1:0/2924157926 <== mon.1 v1:172.21.15.115:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3962223490 0 0) 0x7f683c004eb0 con 0x7f68580b4ff0 2024-03-25T18:28:17.836 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:28:17.836+0000 7f68557fa640 1 -- 172.21.15.1:0/2924157926 --> v1:172.21.15.115:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f6858144600 con 0x7f68580b4ff0 2024-03-25T18:28:17.837 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:28:17.836+0000 7f68557fa640 1 -- 172.21.15.1:0/2924157926 <== mon.1 v1:172.21.15.115:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1586965025 0 0) 0x7f6844001930 con 0x7f68580b4ff0 2024-03-25T18:28:17.837 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:28:17.836+0000 7f68557fa640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:28:17.837 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:28:17.836+0000 7f68557fa640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-03-25T18:28:17.837 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:28:17.836+0000 7f68557fa640 10 monclient(hunting): dump: 2024-03-25T18:28:17.837 INFO:teuthology.misc.health.smithi001.stderr:epoch 1 2024-03-25T18:28:17.837 INFO:teuthology.misc.health.smithi001.stderr:fsid c8ea72cf-9e99-4383-9b52-6c23d365fdfc 2024-03-25T18:28:17.837 INFO:teuthology.misc.health.smithi001.stderr:last_changed 2024-03-25T18:25:50.510519+0000 2024-03-25T18:28:17.837 INFO:teuthology.misc.health.smithi001.stderr:created 2024-03-25T18:25:50.510519+0000 2024-03-25T18:28:17.837 INFO:teuthology.misc.health.smithi001.stderr:min_mon_release 19 (squid) 2024-03-25T18:28:17.837 INFO:teuthology.misc.health.smithi001.stderr:election_strategy: 3 2024-03-25T18:28:17.837 INFO:teuthology.misc.health.smithi001.stderr:0: v1:172.21.15.1:6789/0 mon.a 2024-03-25T18:28:17.837 INFO:teuthology.misc.health.smithi001.stderr:1: v1:172.21.15.115:6789/0 mon.b 2024-03-25T18:28:17.837 INFO:teuthology.misc.health.smithi001.stderr:2: v1:172.21.15.1:6790/0 mon.c 2024-03-25T18:28:17.837 INFO:teuthology.misc.health.smithi001.stderr: 2024-03-25T18:28:17.838 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:28:17.836+0000 7f68557fa640 10 monclient(hunting): _finish_auth 0 2024-03-25T18:28:17.846 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:28:17.845+0000 7f68557fa640 1 -- 172.21.15.1:0/2924157926 <== mon.0 v1:172.21.15.1:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1517790465 0 0) 0x7f68581448e0 con 0x7f685813c480 2024-03-25T18:28:17.846 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:28:17.846+0000 7f68557fa640 1 -- 172.21.15.1:0/2924157926 --> v1:172.21.15.1:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f683c004eb0 con 0x7f685813c480 2024-03-25T18:28:17.846 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:28:17.846+0000 7f68557fa640 1 -- 172.21.15.1:0/2924157926 <== mon.0 v1:172.21.15.1:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1586965025 0 0) 0x7f6850001b30 con 0x7f685813c480 2024-03-25T18:28:17.846 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:28:17.846+0000 7f68557fa640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:28:17.846 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:28:17.846+0000 7f68557fa640 10 monclient(hunting): got monmap 1 from mon.a (according to old e1) 2024-03-25T18:28:17.846 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:28:17.846+0000 7f68557fa640 10 monclient(hunting): dump: 2024-03-25T18:28:17.847 INFO:teuthology.misc.health.smithi001.stderr:epoch 1 2024-03-25T18:28:17.847 INFO:teuthology.misc.health.smithi001.stderr:fsid c8ea72cf-9e99-4383-9b52-6c23d365fdfc 2024-03-25T18:28:17.847 INFO:teuthology.misc.health.smithi001.stderr:last_changed 2024-03-25T18:25:50.510519+0000 2024-03-25T18:28:17.847 INFO:teuthology.misc.health.smithi001.stderr:created 2024-03-25T18:25:50.510519+0000 2024-03-25T18:28:17.847 INFO:teuthology.misc.health.smithi001.stderr:min_mon_release 19 (squid) 2024-03-25T18:28:17.847 INFO:teuthology.misc.health.smithi001.stderr:election_strategy: 3 2024-03-25T18:28:17.847 INFO:teuthology.misc.health.smithi001.stderr:0: v1:172.21.15.1:6789/0 mon.a 2024-03-25T18:28:17.847 INFO:teuthology.misc.health.smithi001.stderr:1: v1:172.21.15.115:6789/0 mon.b 2024-03-25T18:28:17.847 INFO:teuthology.misc.health.smithi001.stderr:2: v1:172.21.15.1:6790/0 mon.c 2024-03-25T18:28:17.847 INFO:teuthology.misc.health.smithi001.stderr: 2024-03-25T18:28:17.852 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:28:17.851+0000 7f68557fa640 1 -- 172.21.15.1:0/2924157926 <== mon.2 v1:172.21.15.1:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 355489899 0 0) 0x7f685813f7e0 con 0x7f68580b4670 2024-03-25T18:28:17.852 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:28:17.851+0000 7f68557fa640 10 monclient(hunting): _negotiate 2024-03-25T18:28:17.852 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:28:17.851+0000 7f68557fa640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:28:17.852 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:28:17.851+0000 7f68557fa640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:28:17.852 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:28:17.851+0000 7f68557fa640 10 monclient(hunting): my global_id is 4295 2024-03-25T18:28:17.852 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:28:17.851+0000 7f68557fa640 1 -- 172.21.15.1:0/2924157926 --> v1:172.21.15.1:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f68581448e0 con 0x7f68580b4670 2024-03-25T18:28:17.857 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:28:17.856+0000 7f68557fa640 1 -- 172.21.15.1:0/2924157926 <== mon.1 v1:172.21.15.115:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 2699583028 0 0) 0x7f6858144600 con 0x7f68580b4ff0 2024-03-25T18:28:17.857 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:28:17.856+0000 7f68557fa640 1 -- 172.21.15.1:0/2924157926 >> v1:172.21.15.1:6790/0 conn(0x7f68580b4670 legacy=0x7f685813ead0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:28:17.857 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:28:17.856+0000 7f68557fa640 1 -- 172.21.15.1:0/2924157926 >> v1:172.21.15.1:6789/0 conn(0x7f685813c480 legacy=0x7f68581e2170 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:28:17.858 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:28:17.856+0000 7f68557fa640 10 monclient: _finish_hunting 0 2024-03-25T18:28:17.858 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:28:17.856+0000 7f68557fa640 1 monclient: found mon.b 2024-03-25T18:28:17.858 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:28:17.856+0000 7f68557fa640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:28:17.858 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:28:17.856+0000 7f68557fa640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.115:6789/0 2024-03-25T18:28:17.858 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:28:17.856+0000 7f68557fa640 1 -- 172.21.15.1:0/2924157926 --> v1:172.21.15.115:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f685813f970 con 0x7f68580b4ff0 2024-03-25T18:28:17.858 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:28:17.856+0000 7f68557fa640 10 monclient: _finish_auth 0 2024-03-25T18:28:17.858 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:28:17.856+0000 7f68557fa640 10 monclient: _check_auth_tickets 2024-03-25T18:28:17.858 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:28:17.856+0000 7f68557fa640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:28:17.858 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:28:17.856+0000 7f685eba3640 5 monclient: authenticate success, global_id 4288 2024-03-25T18:28:17.858 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:28:17.856+0000 7f685eba3640 10 monclient: _renew_subs 2024-03-25T18:28:17.858 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:28:17.856+0000 7f685eba3640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.115:6789/0 2024-03-25T18:28:17.858 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:28:17.856+0000 7f685eba3640 1 -- 172.21.15.1:0/2924157926 --> v1:172.21.15.115:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6858145410 con 0x7f68580b4ff0 2024-03-25T18:28:17.858 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:28:17.856+0000 7f685eba3640 10 client.4288.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T18:28:17.858 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:28:17.857+0000 7f685eba3640 10 monclient: _renew_subs 2024-03-25T18:28:17.858 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:28:17.857+0000 7f685eba3640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.115:6789/0 2024-03-25T18:28:17.859 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:28:17.857+0000 7f685eba3640 1 -- 172.21.15.1:0/2924157926 --> v1:172.21.15.115:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f6858132dd0 con 0x7f68580b4ff0 2024-03-25T18:28:17.859 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:28:17.857+0000 7f685eba3640 1 librados: init done 2024-03-25T18:28:17.859 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:28:17.858+0000 7f685eba3640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T18:28:17.859 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:28:17.858+0000 7f685eba3640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:28:17.859 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:28:17.858+0000 7f685eba3640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.115:6789/0 2024-03-25T18:28:17.859 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:28:17.858+0000 7f685eba3640 1 -- 172.21.15.1:0/2924157926 --> v1:172.21.15.115:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f685806dbd0 con 0x7f68580b4ff0 2024-03-25T18:28:18.072 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:28:18.071+0000 7f68557fa640 1 -- 172.21.15.1:0/2924157926 <== mon.1 v1:172.21.15.115:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f68440035f0 con 0x7f68580b4ff0 2024-03-25T18:28:18.072 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:28:18.071+0000 7f68557fa640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:28:18.072 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:28:18.071+0000 7f6856ffd640 4 set_mon_vals no callback set 2024-03-25T18:28:18.138 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:28:18.137+0000 7f68557fa640 1 -- 172.21.15.1:0/2924157926 <== mon.1 v1:172.21.15.115:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1586965025 0 0) 0x7f6844003900 con 0x7f68580b4ff0 2024-03-25T18:28:18.139 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:28:18.137+0000 7f68557fa640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:28:18.139 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:28:18.137+0000 7f68557fa640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-25T18:28:18.139 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:28:18.137+0000 7f68557fa640 10 monclient: dump: 2024-03-25T18:28:18.139 INFO:teuthology.misc.health.smithi001.stderr:epoch 1 2024-03-25T18:28:18.139 INFO:teuthology.misc.health.smithi001.stderr:fsid c8ea72cf-9e99-4383-9b52-6c23d365fdfc 2024-03-25T18:28:18.139 INFO:teuthology.misc.health.smithi001.stderr:last_changed 2024-03-25T18:25:50.510519+0000 2024-03-25T18:28:18.139 INFO:teuthology.misc.health.smithi001.stderr:created 2024-03-25T18:25:50.510519+0000 2024-03-25T18:28:18.139 INFO:teuthology.misc.health.smithi001.stderr:min_mon_release 19 (squid) 2024-03-25T18:28:18.139 INFO:teuthology.misc.health.smithi001.stderr:election_strategy: 3 2024-03-25T18:28:18.139 INFO:teuthology.misc.health.smithi001.stderr:0: v1:172.21.15.1:6789/0 mon.a 2024-03-25T18:28:18.140 INFO:teuthology.misc.health.smithi001.stderr:1: v1:172.21.15.115:6789/0 mon.b 2024-03-25T18:28:18.140 INFO:teuthology.misc.health.smithi001.stderr:2: v1:172.21.15.1:6790/0 mon.c 2024-03-25T18:28:18.140 INFO:teuthology.misc.health.smithi001.stderr: 2024-03-25T18:28:18.140 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:28:18.138+0000 7f68557fa640 1 -- 172.21.15.1:0/2924157926 <== mon.1 v1:172.21.15.115:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 21787975 0 0) 0x7f68440198d0 con 0x7f68580b4ff0 2024-03-25T18:28:18.140 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:28:18.139+0000 7f685c93f640 10 monclient: get_auth_request con 0x7f683c06df50 auth_method 0 2024-03-25T18:28:18.141 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:28:18.139+0000 7f68557fa640 10 client.4288.objecter ms_handle_connect 0x7f683c06df50 2024-03-25T18:28:18.142 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:28:18.141+0000 7f68557fa640 1 -- 172.21.15.1:0/2924157926 <== mon.1 v1:172.21.15.115:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 996254230 0 0) 0x7f68440536b0 con 0x7f68580b4ff0 2024-03-25T18:28:18.142 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:28:18.141+0000 7f68557fa640 10 client.4288.objecter ms_dispatch 0x7f6858000b90 osd_map(10..10 src has 1..10) v4 2024-03-25T18:28:18.142 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:28:18.141+0000 7f68557fa640 3 client.4288.objecter handle_osd_map got epochs [10,10] > 0 2024-03-25T18:28:18.142 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:28:18.141+0000 7f68557fa640 3 client.4288.objecter handle_osd_map decoding full epoch 10 2024-03-25T18:28:18.142 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:28:18.141+0000 7f68557fa640 20 client.4288.objecter dump_active .. 0 homeless 2024-03-25T18:28:18.144 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:28:18.144+0000 7f68557fa640 1 -- 172.21.15.1:0/2924157926 <== mon.1 v1:172.21.15.115:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f684404eff0 con 0x7f68580b4ff0 2024-03-25T18:28:18.145 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:28:18.144+0000 7f68557fa640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:28:18.145 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:28:18.144+0000 7f68557fa640 10 monclient: _finish_command 1 = system:0 2024-03-25T18:28:18.400 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:28:18.399+0000 7f685eba3640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:28:18.400 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:28:18.399+0000 7f685eba3640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:28:18.400 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:28:18.399+0000 7f685eba3640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.115:6789/0 2024-03-25T18:28:18.400 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:28:18.399+0000 7f685eba3640 1 -- 172.21.15.1:0/2924157926 --> v1:172.21.15.115:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f68580688c0 con 0x7f68580b4ff0 2024-03-25T18:28:18.439 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:28:18.438+0000 7f68557fa640 1 -- 172.21.15.1:0/2924157926 <== mon.1 v1:172.21.15.115:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 4092213348) 0x7f68440510f0 con 0x7f68580b4ff0 2024-03-25T18:28:18.439 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:28:18.438+0000 7f68557fa640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:28:18.439 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:28:18.438+0000 7f68557fa640 10 monclient: _finish_command 2 = system:0 2024-03-25T18:28:18.439 INFO:teuthology.misc.health.smithi001.stdout: 2024-03-25T18:28:18.439 INFO:teuthology.misc.health.smithi001.stdout:{"epoch":10,"fsid":"c8ea72cf-9e99-4383-9b52-6c23d365fdfc","created":"2024-03-25T18:26:38.856638+0000","modified":"2024-03-25T18:26:44.224208+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T18:26:44.224208+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"b9f34922-3a84-49be-8426-807c1eceaefc","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"42bf306f-55e3-4634-9874-8b4b73ac6644","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"292cb23a-18a9-4207-a7cc-4d5727feaec0","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"45e0e9ab-898d-4673-9590-b4117bc5e84a","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"831a5a09-73ab-4a61-9872-088e76bdbc66","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"88cd903d-3b55-4840-9396-6e3cfe005282","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"89db7355-bc9d-46f4-8ab0-e8d9e744e026","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"cac47bee-0914-4f08-896f-348838882866","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T18:28:18.442 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:28:18.440+0000 7f685eba3640 10 librados: watch_flush enter 2024-03-25T18:28:18.442 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:28:18.440+0000 7f685eba3640 10 librados: watch_flush exit 2024-03-25T18:28:18.442 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:28:18.440+0000 7f685eba3640 20 client.4288.objecter shutdown clearing up homeless session... 2024-03-25T18:28:18.442 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:28:18.440+0000 7f685eba3640 10 client.4288.objecter successfully canceled tick 2024-03-25T18:28:18.442 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:28:18.440+0000 7f685eba3640 1 -- 172.21.15.1:0/2924157926 >> v1:172.21.15.115:6800/4237379176 conn(0x7f683c06df50 legacy=0x7f683c070410 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:28:18.442 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:28:18.440+0000 7f685eba3640 10 monclient: shutdown 2024-03-25T18:28:18.442 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:28:18.440+0000 7f685eba3640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:28:18.442 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:28:18.440+0000 7f685eba3640 1 -- 172.21.15.1:0/2924157926 >> v1:172.21.15.115:6789/0 conn(0x7f68580b4ff0 legacy=0x7f6858144d00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:28:18.442 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:28:18.441+0000 7f685eba3640 1 -- 172.21.15.1:0/2924157926 shutdown_connections 2024-03-25T18:28:18.442 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:28:18.441+0000 7f685eba3640 1 -- 172.21.15.1:0/2924157926 >> 172.21.15.1:0/2924157926 conn(0x7f68580617a0 msgr2=0x7f6858140670 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:28:18.442 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:28:18.441+0000 7f685eba3640 1 -- 172.21.15.1:0/2924157926 shutdown_connections 2024-03-25T18:28:18.442 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:28:18.441+0000 7f685eba3640 1 -- 172.21.15.1:0/2924157926 wait complete. 2024-03-25T18:28:18.442 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:28:18.441+0000 7f685eba3640 1 librados: shutdown 2024-03-25T18:28:18.454 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T18:28:25.262 DEBUG:teuthology.orchestra.run.smithi001:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T18:28:25.377 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:28:25.377+0000 7f3717c84640 10 monclient: get_monmap_and_config 2024-03-25T18:28:25.378 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:28:25.377+0000 7f3717c84640 10 monclient: build_initial_monmap 2024-03-25T18:28:25.378 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:28:25.377+0000 7f3717c84640 10 monclient: monmap: 2024-03-25T18:28:25.378 INFO:teuthology.misc.health.smithi001.stderr:epoch 0 2024-03-25T18:28:25.378 INFO:teuthology.misc.health.smithi001.stderr:fsid c8ea72cf-9e99-4383-9b52-6c23d365fdfc 2024-03-25T18:28:25.378 INFO:teuthology.misc.health.smithi001.stderr:last_changed 2024-03-25T18:28:25.377849+0000 2024-03-25T18:28:25.378 INFO:teuthology.misc.health.smithi001.stderr:created 2024-03-25T18:28:25.377849+0000 2024-03-25T18:28:25.378 INFO:teuthology.misc.health.smithi001.stderr:min_mon_release 0 (unknown) 2024-03-25T18:28:25.378 INFO:teuthology.misc.health.smithi001.stderr:election_strategy: 3 2024-03-25T18:28:25.378 INFO:teuthology.misc.health.smithi001.stderr:0: v1:172.21.15.1:6789/0 mon.noname-a 2024-03-25T18:28:25.378 INFO:teuthology.misc.health.smithi001.stderr:1: v1:172.21.15.115:6789/0 mon.noname-c 2024-03-25T18:28:25.379 INFO:teuthology.misc.health.smithi001.stderr:2: v1:172.21.15.1:6790/0 mon.noname-b 2024-03-25T18:28:25.379 INFO:teuthology.misc.health.smithi001.stderr: 2024-03-25T18:28:25.380 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:28:25.378+0000 7f3717c84640 1 Processor -- start 2024-03-25T18:28:25.380 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:28:25.378+0000 7f3717c84640 1 -- start start 2024-03-25T18:28:25.380 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:28:25.378+0000 7f3717c84640 10 monclient: init 2024-03-25T18:28:25.380 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:28:25.378+0000 7f3717c84640 10 monclient: _reopen_session rank -1 2024-03-25T18:28:25.380 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:28:25.378+0000 7f3717c84640 10 monclient: _add_conns ranks=[2,1,0] 2024-03-25T18:28:25.380 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:28:25.378+0000 7f3717c84640 10 monclient(hunting): picked mon.noname-b con 0x7f37100b4ff0 addr v1:172.21.15.1:6790/0 2024-03-25T18:28:25.380 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:28:25.378+0000 7f3717c84640 10 monclient(hunting): picked mon.noname-c con 0x7f37100b4670 addr v1:172.21.15.115:6789/0 2024-03-25T18:28:25.380 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:28:25.378+0000 7f3717c84640 10 monclient(hunting): picked mon.noname-a con 0x7f371013c480 addr v1:172.21.15.1:6789/0 2024-03-25T18:28:25.380 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:28:25.379+0000 7f3717c84640 1 -- --> v1:172.21.15.1:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f37100ab0e0 con 0x7f371013c480 2024-03-25T18:28:25.380 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:28:25.379+0000 7f3717c84640 1 -- --> v1:172.21.15.115:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f37100ab3c0 con 0x7f37100b4670 2024-03-25T18:28:25.380 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:28:25.379+0000 7f3717c84640 1 -- --> v1:172.21.15.1:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f37100a9f30 con 0x7f37100b4ff0 2024-03-25T18:28:25.380 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:28:25.379+0000 7f3717c84640 10 monclient(hunting): _renew_subs 2024-03-25T18:28:25.380 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:28:25.379+0000 7f3717c84640 10 monclient(hunting): authenticate will time out at 7585.848710s 2024-03-25T18:28:25.389 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:28:25.388+0000 7f3715a20640 1 --1- >> v1:172.21.15.1:6790/0 conn(0x7f37100b4ff0 0x7f371013bd70 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.1:6790/0 says I am v1:172.21.15.1:41664/0 (socket says 172.21.15.1:41664) 2024-03-25T18:28:25.389 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:28:25.388+0000 7f3715a20640 1 -- 172.21.15.1:0/328153412 learned_addr learned my addr 172.21.15.1:0/328153412 (peer_addr_for_me v1:172.21.15.1:0/0) 2024-03-25T18:28:25.404 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:28:25.404+0000 7f3714a1e640 1 -- 172.21.15.1:0/328153412 <== mon.1 v1:172.21.15.115:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 539624147 0 0) 0x7f37100ab3c0 con 0x7f37100b4670 2024-03-25T18:28:25.405 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:28:25.404+0000 7f3714a1e640 10 monclient(hunting): _negotiate 2024-03-25T18:28:25.405 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:28:25.404+0000 7f3714a1e640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:28:25.405 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:28:25.404+0000 7f3714a1e640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:28:25.405 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:28:25.404+0000 7f3714a1e640 10 monclient(hunting): my global_id is 4291 2024-03-25T18:28:25.405 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:28:25.404+0000 7f3714a1e640 1 -- 172.21.15.1:0/328153412 --> v1:172.21.15.115:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f36fc003520 con 0x7f37100b4670 2024-03-25T18:28:25.408 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:28:25.408+0000 7f3714a1e640 1 -- 172.21.15.1:0/328153412 <== mon.2 v1:172.21.15.1:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1238271994 0 0) 0x7f37100a9f30 con 0x7f37100b4ff0 2024-03-25T18:28:25.408 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:28:25.408+0000 7f3714a1e640 10 monclient(hunting): _negotiate 2024-03-25T18:28:25.408 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:28:25.408+0000 7f3714a1e640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:28:25.409 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:28:25.408+0000 7f3714a1e640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:28:25.409 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:28:25.408+0000 7f3714a1e640 10 monclient(hunting): my global_id is 4298 2024-03-25T18:28:25.409 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:28:25.408+0000 7f3714a1e640 1 -- 172.21.15.1:0/328153412 --> v1:172.21.15.1:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f37100ab3c0 con 0x7f37100b4ff0 2024-03-25T18:28:25.411 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:28:25.411+0000 7f3714a1e640 1 -- 172.21.15.1:0/328153412 <== mon.0 v1:172.21.15.1:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2886300932 0 0) 0x7f37100ab0e0 con 0x7f371013c480 2024-03-25T18:28:25.412 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:28:25.411+0000 7f3714a1e640 10 monclient(hunting): _negotiate 2024-03-25T18:28:25.412 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:28:25.411+0000 7f3714a1e640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:28:25.412 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:28:25.411+0000 7f3714a1e640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:28:25.412 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:28:25.411+0000 7f3714a1e640 10 monclient(hunting): my global_id is 4296 2024-03-25T18:28:25.412 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:28:25.411+0000 7f3714a1e640 1 -- 172.21.15.1:0/328153412 --> v1:172.21.15.1:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f37100a9f30 con 0x7f371013c480 2024-03-25T18:28:25.417 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:28:25.416+0000 7f3714a1e640 1 -- 172.21.15.1:0/328153412 <== mon.0 v1:172.21.15.1:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1724316699 0 0) 0x7f37100a9f30 con 0x7f371013c480 2024-03-25T18:28:25.417 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:28:25.416+0000 7f3714a1e640 1 -- 172.21.15.1:0/328153412 >> v1:172.21.15.1:6790/0 conn(0x7f37100b4ff0 legacy=0x7f371013bd70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:28:25.417 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:28:25.416+0000 7f3714a1e640 1 -- 172.21.15.1:0/328153412 >> v1:172.21.15.115:6789/0 conn(0x7f37100b4670 legacy=0x7f37101425c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:28:25.417 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:28:25.416+0000 7f3714a1e640 10 monclient: _finish_hunting 0 2024-03-25T18:28:25.417 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:28:25.416+0000 7f3714a1e640 1 monclient: found mon.noname-a 2024-03-25T18:28:25.417 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:28:25.416+0000 7f3714a1e640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:28:25.418 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:28:25.416+0000 7f3714a1e640 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.1:6789/0 2024-03-25T18:28:25.418 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:28:25.416+0000 7f3714a1e640 1 -- 172.21.15.1:0/328153412 --> v1:172.21.15.1:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f37100a9b70 con 0x7f371013c480 2024-03-25T18:28:25.418 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:28:25.416+0000 7f3714a1e640 10 monclient: _finish_auth 0 2024-03-25T18:28:25.418 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:28:25.416+0000 7f3714a1e640 10 monclient: _check_auth_tickets 2024-03-25T18:28:25.418 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:28:25.416+0000 7f3714a1e640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:28:25.418 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:28:25.416+0000 7f3714a1e640 1 -- 172.21.15.1:0/328153412 <== mon.0 v1:172.21.15.1:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1586965025 0 0) 0x7f3708001930 con 0x7f371013c480 2024-03-25T18:28:25.418 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:28:25.416+0000 7f3714a1e640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:28:25.418 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:28:25.416+0000 7f3714a1e640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-03-25T18:28:25.418 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:28:25.416+0000 7f3714a1e640 10 monclient: dump: 2024-03-25T18:28:25.418 INFO:teuthology.misc.health.smithi001.stderr:epoch 1 2024-03-25T18:28:25.418 INFO:teuthology.misc.health.smithi001.stderr:fsid c8ea72cf-9e99-4383-9b52-6c23d365fdfc 2024-03-25T18:28:25.418 INFO:teuthology.misc.health.smithi001.stderr:last_changed 2024-03-25T18:25:50.510519+0000 2024-03-25T18:28:25.418 INFO:teuthology.misc.health.smithi001.stderr:created 2024-03-25T18:25:50.510519+0000 2024-03-25T18:28:25.418 INFO:teuthology.misc.health.smithi001.stderr:min_mon_release 19 (squid) 2024-03-25T18:28:25.419 INFO:teuthology.misc.health.smithi001.stderr:election_strategy: 3 2024-03-25T18:28:25.419 INFO:teuthology.misc.health.smithi001.stderr:0: v1:172.21.15.1:6789/0 mon.a 2024-03-25T18:28:25.419 INFO:teuthology.misc.health.smithi001.stderr:1: v1:172.21.15.115:6789/0 mon.b 2024-03-25T18:28:25.419 INFO:teuthology.misc.health.smithi001.stderr:2: v1:172.21.15.1:6790/0 mon.c 2024-03-25T18:28:25.419 INFO:teuthology.misc.health.smithi001.stderr: 2024-03-25T18:28:25.419 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:28:25.416+0000 7f3717c84640 5 monclient: authenticate success, global_id 4296 2024-03-25T18:28:25.419 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:28:25.416+0000 7f3717c84640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T18:28:25.447 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:28:25.446+0000 7f3714a1e640 1 -- 172.21.15.1:0/328153412 <== mon.0 v1:172.21.15.1:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f3708001a90 con 0x7f371013c480 2024-03-25T18:28:25.447 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:28:25.446+0000 7f3714a1e640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:28:25.447 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:28:25.446+0000 7f3714a1e640 1 -- 172.21.15.1:0/328153412 <== mon.0 v1:172.21.15.1:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1586965025 0 0) 0x7f3708001ee0 con 0x7f371013c480 2024-03-25T18:28:25.447 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:28:25.446+0000 7f3714a1e640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:28:25.447 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:28:25.446+0000 7f3714a1e640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-25T18:28:25.447 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:28:25.446+0000 7f3714a1e640 10 monclient: dump: 2024-03-25T18:28:25.447 INFO:teuthology.misc.health.smithi001.stderr:epoch 1 2024-03-25T18:28:25.447 INFO:teuthology.misc.health.smithi001.stderr:fsid c8ea72cf-9e99-4383-9b52-6c23d365fdfc 2024-03-25T18:28:25.448 INFO:teuthology.misc.health.smithi001.stderr:last_changed 2024-03-25T18:25:50.510519+0000 2024-03-25T18:28:25.448 INFO:teuthology.misc.health.smithi001.stderr:created 2024-03-25T18:25:50.510519+0000 2024-03-25T18:28:25.448 INFO:teuthology.misc.health.smithi001.stderr:min_mon_release 19 (squid) 2024-03-25T18:28:25.448 INFO:teuthology.misc.health.smithi001.stderr:election_strategy: 3 2024-03-25T18:28:25.448 INFO:teuthology.misc.health.smithi001.stderr:0: v1:172.21.15.1:6789/0 mon.a 2024-03-25T18:28:25.448 INFO:teuthology.misc.health.smithi001.stderr:1: v1:172.21.15.115:6789/0 mon.b 2024-03-25T18:28:25.448 INFO:teuthology.misc.health.smithi001.stderr:2: v1:172.21.15.1:6790/0 mon.c 2024-03-25T18:28:25.448 INFO:teuthology.misc.health.smithi001.stderr: 2024-03-25T18:28:25.448 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:28:25.446+0000 7f3717c84640 10 monclient: get_monmap_and_config success 2024-03-25T18:28:25.448 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:28:25.446+0000 7f3717c84640 4 set_mon_vals no callback set 2024-03-25T18:28:25.450 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:28:25.447+0000 7f3717c84640 10 monclient: shutdown 2024-03-25T18:28:25.450 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:28:25.447+0000 7f3717c84640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:28:25.450 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:28:25.447+0000 7f3717c84640 1 -- 172.21.15.1:0/328153412 >> v1:172.21.15.1:6789/0 conn(0x7f371013c480 legacy=0x7f371014dc00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:28:25.450 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:28:25.447+0000 7f3717c84640 1 -- 172.21.15.1:0/328153412 shutdown_connections 2024-03-25T18:28:25.450 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:28:25.447+0000 7f3717c84640 1 -- 172.21.15.1:0/328153412 >> 172.21.15.1:0/328153412 conn(0x7f37100617a0 msgr2=0x7f3710061bb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:28:25.450 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:28:25.447+0000 7f3717c84640 1 -- 172.21.15.1:0/328153412 shutdown_connections 2024-03-25T18:28:25.451 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:28:25.447+0000 7f3717c84640 1 -- 172.21.15.1:0/328153412 wait complete. 2024-03-25T18:28:25.451 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:28:25.448+0000 7f3717c84640 10 monclient: build_initial_monmap 2024-03-25T18:28:25.451 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:28:25.448+0000 7f3717c84640 10 monclient: monmap: 2024-03-25T18:28:25.451 INFO:teuthology.misc.health.smithi001.stderr:epoch 0 2024-03-25T18:28:25.451 INFO:teuthology.misc.health.smithi001.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T18:28:25.451 INFO:teuthology.misc.health.smithi001.stderr:last_changed 0.000000 2024-03-25T18:28:25.451 INFO:teuthology.misc.health.smithi001.stderr:created 0.000000 2024-03-25T18:28:25.451 INFO:teuthology.misc.health.smithi001.stderr:min_mon_release 0 (unknown) 2024-03-25T18:28:25.451 INFO:teuthology.misc.health.smithi001.stderr:election_strategy: 1 2024-03-25T18:28:25.451 INFO:teuthology.misc.health.smithi001.stderr:0: v1:172.21.15.1:6789/0 mon.noname-a 2024-03-25T18:28:25.451 INFO:teuthology.misc.health.smithi001.stderr:1: v1:172.21.15.115:6789/0 mon.noname-b 2024-03-25T18:28:25.451 INFO:teuthology.misc.health.smithi001.stderr:2: v1:172.21.15.1:6790/0 mon.noname-c 2024-03-25T18:28:25.451 INFO:teuthology.misc.health.smithi001.stderr: 2024-03-25T18:28:25.451 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:28:25.448+0000 7f3717c84640 1 librados: starting msgr at 2024-03-25T18:28:25.452 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:28:25.448+0000 7f3717c84640 1 librados: starting objecter 2024-03-25T18:28:25.452 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:28:25.448+0000 7f3717c84640 1 Processor -- start 2024-03-25T18:28:25.452 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:28:25.448+0000 7f3717c84640 1 -- start start 2024-03-25T18:28:25.452 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:28:25.448+0000 7f3717c84640 1 librados: setting wanted keys 2024-03-25T18:28:25.452 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:28:25.448+0000 7f3717c84640 1 librados: calling monclient init 2024-03-25T18:28:25.452 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:28:25.448+0000 7f3717c84640 10 monclient: init 2024-03-25T18:28:25.452 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:28:25.449+0000 7f3717c84640 10 monclient: _reopen_session rank -1 2024-03-25T18:28:25.452 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:28:25.449+0000 7f3717c84640 10 monclient: _add_conns ranks=[0,2,1] 2024-03-25T18:28:25.452 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:28:25.449+0000 7f3717c84640 10 monclient(hunting): picked mon.noname-a con 0x7f37100b4670 addr v1:172.21.15.1:6789/0 2024-03-25T18:28:25.452 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:28:25.449+0000 7f3717c84640 10 monclient(hunting): picked mon.noname-c con 0x7f37100b4ff0 addr v1:172.21.15.1:6790/0 2024-03-25T18:28:25.452 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:28:25.449+0000 7f3717c84640 10 monclient(hunting): picked mon.noname-b con 0x7f371013c480 addr v1:172.21.15.115:6789/0 2024-03-25T18:28:25.452 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:28:25.449+0000 7f3717c84640 1 -- --> v1:172.21.15.1:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3710130ad0 con 0x7f37100b4670 2024-03-25T18:28:25.452 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:28:25.449+0000 7f3717c84640 1 -- --> v1:172.21.15.115:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f371013f930 con 0x7f371013c480 2024-03-25T18:28:25.452 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:28:25.449+0000 7f3717c84640 1 -- --> v1:172.21.15.1:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3710144550 con 0x7f37100b4ff0 2024-03-25T18:28:25.453 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:28:25.449+0000 7f3717c84640 10 monclient(hunting): _renew_subs 2024-03-25T18:28:25.453 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:28:25.449+0000 7f3717c84640 10 monclient(hunting): authenticate will time out at 7585.919301s 2024-03-25T18:28:25.464 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:28:25.463+0000 7f3716221640 1 --1- >> v1:172.21.15.115:6789/0 conn(0x7f371013c480 0x7f37101303c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.115:6789/0 says I am v1:172.21.15.1:42604/0 (socket says 172.21.15.1:42604) 2024-03-25T18:28:25.464 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:28:25.463+0000 7f3716221640 1 -- 172.21.15.1:0/1724883746 learned_addr learned my addr 172.21.15.1:0/1724883746 (peer_addr_for_me v1:172.21.15.1:0/0) 2024-03-25T18:28:25.464 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:28:25.464+0000 7f37067fc640 10 client.?.objecter ms_handle_connect 0x7f371013c480 2024-03-25T18:28:25.465 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:28:25.464+0000 7f37067fc640 10 client.?.objecter resend_mon_ops 2024-03-25T18:28:25.467 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:28:25.467+0000 7f37067fc640 10 client.?.objecter ms_handle_connect 0x7f37100b4670 2024-03-25T18:28:25.468 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:28:25.467+0000 7f37067fc640 10 client.?.objecter resend_mon_ops 2024-03-25T18:28:25.470 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:28:25.469+0000 7f37067fc640 10 client.?.objecter ms_handle_connect 0x7f37100b4ff0 2024-03-25T18:28:25.470 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:28:25.469+0000 7f37067fc640 10 client.?.objecter resend_mon_ops 2024-03-25T18:28:25.473 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:28:25.473+0000 7f37067fc640 1 -- 172.21.15.1:0/1724883746 <== mon.0 v1:172.21.15.1:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 504225012 0 0) 0x7f3710130ad0 con 0x7f37100b4670 2024-03-25T18:28:25.473 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:28:25.473+0000 7f37067fc640 10 monclient(hunting): _negotiate 2024-03-25T18:28:25.473 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:28:25.473+0000 7f37067fc640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:28:25.473 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:28:25.473+0000 7f37067fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:28:25.474 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:28:25.473+0000 7f37067fc640 10 monclient(hunting): my global_id is 4299 2024-03-25T18:28:25.474 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:28:25.473+0000 7f37067fc640 1 -- 172.21.15.1:0/1724883746 --> v1:172.21.15.1:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f36f4004eb0 con 0x7f37100b4670 2024-03-25T18:28:25.478 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:28:25.477+0000 7f37067fc640 1 -- 172.21.15.1:0/1724883746 <== mon.2 v1:172.21.15.1:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 406536557 0 0) 0x7f3710144550 con 0x7f37100b4ff0 2024-03-25T18:28:25.478 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:28:25.477+0000 7f37067fc640 10 monclient(hunting): _negotiate 2024-03-25T18:28:25.478 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:28:25.477+0000 7f37067fc640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:28:25.478 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:28:25.477+0000 7f37067fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:28:25.478 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:28:25.477+0000 7f37067fc640 10 monclient(hunting): my global_id is 4301 2024-03-25T18:28:25.478 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:28:25.477+0000 7f37067fc640 1 -- 172.21.15.1:0/1724883746 --> v1:172.21.15.1:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f3710130ad0 con 0x7f37100b4ff0 2024-03-25T18:28:25.482 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:28:25.482+0000 7f37067fc640 1 -- 172.21.15.1:0/1724883746 <== mon.1 v1:172.21.15.115:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2598673807 0 0) 0x7f371013f930 con 0x7f371013c480 2024-03-25T18:28:25.483 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:28:25.482+0000 7f37067fc640 10 monclient(hunting): _negotiate 2024-03-25T18:28:25.483 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:28:25.482+0000 7f37067fc640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:28:25.483 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:28:25.482+0000 7f37067fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:28:25.483 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:28:25.482+0000 7f37067fc640 10 monclient(hunting): my global_id is 4294 2024-03-25T18:28:25.483 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:28:25.482+0000 7f37067fc640 1 -- 172.21.15.1:0/1724883746 --> v1:172.21.15.115:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f3710144550 con 0x7f371013c480 2024-03-25T18:28:25.498 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:28:25.496+0000 7f37067fc640 1 -- 172.21.15.1:0/1724883746 <== mon.1 v1:172.21.15.115:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2539371701 0 0) 0x7f3710144550 con 0x7f371013c480 2024-03-25T18:28:25.498 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:28:25.496+0000 7f37067fc640 1 -- 172.21.15.1:0/1724883746 --> v1:172.21.15.115:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f371013f930 con 0x7f371013c480 2024-03-25T18:28:25.498 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:28:25.496+0000 7f37067fc640 1 -- 172.21.15.1:0/1724883746 <== mon.1 v1:172.21.15.115:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1586965025 0 0) 0x7f3708001e60 con 0x7f371013c480 2024-03-25T18:28:25.498 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:28:25.496+0000 7f37067fc640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:28:25.498 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:28:25.496+0000 7f37067fc640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-03-25T18:28:25.498 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:28:25.496+0000 7f37067fc640 10 monclient(hunting): dump: 2024-03-25T18:28:25.498 INFO:teuthology.misc.health.smithi001.stderr:epoch 1 2024-03-25T18:28:25.498 INFO:teuthology.misc.health.smithi001.stderr:fsid c8ea72cf-9e99-4383-9b52-6c23d365fdfc 2024-03-25T18:28:25.498 INFO:teuthology.misc.health.smithi001.stderr:last_changed 2024-03-25T18:25:50.510519+0000 2024-03-25T18:28:25.498 INFO:teuthology.misc.health.smithi001.stderr:created 2024-03-25T18:25:50.510519+0000 2024-03-25T18:28:25.498 INFO:teuthology.misc.health.smithi001.stderr:min_mon_release 19 (squid) 2024-03-25T18:28:25.499 INFO:teuthology.misc.health.smithi001.stderr:election_strategy: 3 2024-03-25T18:28:25.499 INFO:teuthology.misc.health.smithi001.stderr:0: v1:172.21.15.1:6789/0 mon.a 2024-03-25T18:28:25.499 INFO:teuthology.misc.health.smithi001.stderr:1: v1:172.21.15.115:6789/0 mon.b 2024-03-25T18:28:25.499 INFO:teuthology.misc.health.smithi001.stderr:2: v1:172.21.15.1:6790/0 mon.c 2024-03-25T18:28:25.499 INFO:teuthology.misc.health.smithi001.stderr: 2024-03-25T18:28:25.499 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:28:25.496+0000 7f37067fc640 10 monclient(hunting): _finish_auth 0 2024-03-25T18:28:25.499 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:28:25.497+0000 7f371521f640 1 -- 172.21.15.1:0/1724883746 >> v1:172.21.15.1:6790/0 conn(0x7f37100b4ff0 legacy=0x7f371012fcb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 17 2024-03-25T18:28:25.499 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:28:25.497+0000 7f371521f640 1 -- 172.21.15.1:0/1724883746 >> v1:172.21.15.1:6790/0 conn(0x7f37100b4ff0 legacy=0x7f371012fcb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-03-25T18:28:25.499 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:28:25.497+0000 7f371521f640 1 --1- 172.21.15.1:0/1724883746 >> v1:172.21.15.1:6790/0 conn(0x7f37100b4ff0 0x7f371012fcb0 :-1 s=OPENED pgs=8270 cs=1 l=1).handle_message read tag failed 2024-03-25T18:28:25.499 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:28:25.497+0000 7f371521f640 1 --1- 172.21.15.1:0/1724883746 >> v1:172.21.15.1:6790/0 conn(0x7f37100b4ff0 0x7f371012fcb0 :-1 s=OPENED pgs=8270 cs=1 l=1).fault on lossy channel, failing 2024-03-25T18:28:25.499 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:28:25.497+0000 7f37067fc640 10 monclient(hunting): ms_handle_reset hunted mon v1:172.21.15.1:6790/0 2024-03-25T18:28:25.505 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:28:25.504+0000 7f37067fc640 1 -- 172.21.15.1:0/1724883746 <== mon.0 v1:172.21.15.1:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1725028857 0 0) 0x7f36f4004eb0 con 0x7f37100b4670 2024-03-25T18:28:25.505 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:28:25.504+0000 7f37067fc640 1 -- 172.21.15.1:0/1724883746 --> v1:172.21.15.1:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f3710144550 con 0x7f37100b4670 2024-03-25T18:28:25.505 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:28:25.504+0000 7f37067fc640 1 -- 172.21.15.1:0/1724883746 <== mon.0 v1:172.21.15.1:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1586965025 0 0) 0x7f370c001930 con 0x7f37100b4670 2024-03-25T18:28:25.505 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:28:25.504+0000 7f37067fc640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:28:25.505 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:28:25.504+0000 7f37067fc640 10 monclient(hunting): got monmap 1 from mon.a (according to old e1) 2024-03-25T18:28:25.505 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:28:25.504+0000 7f37067fc640 10 monclient(hunting): dump: 2024-03-25T18:28:25.506 INFO:teuthology.misc.health.smithi001.stderr:epoch 1 2024-03-25T18:28:25.506 INFO:teuthology.misc.health.smithi001.stderr:fsid c8ea72cf-9e99-4383-9b52-6c23d365fdfc 2024-03-25T18:28:25.506 INFO:teuthology.misc.health.smithi001.stderr:last_changed 2024-03-25T18:25:50.510519+0000 2024-03-25T18:28:25.506 INFO:teuthology.misc.health.smithi001.stderr:created 2024-03-25T18:25:50.510519+0000 2024-03-25T18:28:25.506 INFO:teuthology.misc.health.smithi001.stderr:min_mon_release 19 (squid) 2024-03-25T18:28:25.506 INFO:teuthology.misc.health.smithi001.stderr:election_strategy: 3 2024-03-25T18:28:25.506 INFO:teuthology.misc.health.smithi001.stderr:0: v1:172.21.15.1:6789/0 mon.a 2024-03-25T18:28:25.506 INFO:teuthology.misc.health.smithi001.stderr:1: v1:172.21.15.115:6789/0 mon.b 2024-03-25T18:28:25.506 INFO:teuthology.misc.health.smithi001.stderr:2: v1:172.21.15.1:6790/0 mon.c 2024-03-25T18:28:25.506 INFO:teuthology.misc.health.smithi001.stderr: 2024-03-25T18:28:25.527 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:28:25.526+0000 7f37067fc640 1 -- 172.21.15.1:0/1724883746 <== mon.1 v1:172.21.15.115:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 1041145557 0 0) 0x7f371013f930 con 0x7f371013c480 2024-03-25T18:28:25.527 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:28:25.526+0000 7f37067fc640 1 -- 172.21.15.1:0/1724883746 >> v1:172.21.15.1:6790/0 conn(0x7f37100b4ff0 legacy=0x7f371012fcb0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-03-25T18:28:25.527 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:28:25.526+0000 7f37067fc640 1 -- 172.21.15.1:0/1724883746 >> v1:172.21.15.1:6789/0 conn(0x7f37100b4670 legacy=0x7f3710144f30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:28:25.527 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:28:25.526+0000 7f37067fc640 10 monclient: _finish_hunting 0 2024-03-25T18:28:25.527 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:28:25.526+0000 7f37067fc640 1 monclient: found mon.b 2024-03-25T18:28:25.528 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:28:25.526+0000 7f37067fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:28:25.528 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:28:25.526+0000 7f37067fc640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.115:6789/0 2024-03-25T18:28:25.528 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:28:25.526+0000 7f37067fc640 1 -- 172.21.15.1:0/1724883746 --> v1:172.21.15.115:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3710144830 con 0x7f371013c480 2024-03-25T18:28:25.528 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:28:25.526+0000 7f37067fc640 10 monclient: _finish_auth 0 2024-03-25T18:28:25.528 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:28:25.526+0000 7f37067fc640 10 monclient: _check_auth_tickets 2024-03-25T18:28:25.528 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:28:25.526+0000 7f37067fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:28:25.528 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:28:25.526+0000 7f3717c84640 5 monclient: authenticate success, global_id 4294 2024-03-25T18:28:25.528 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:28:25.526+0000 7f3717c84640 10 monclient: _renew_subs 2024-03-25T18:28:25.528 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:28:25.526+0000 7f3717c84640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.115:6789/0 2024-03-25T18:28:25.528 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:28:25.526+0000 7f3717c84640 1 -- 172.21.15.1:0/1724883746 --> v1:172.21.15.115:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3710144b10 con 0x7f371013c480 2024-03-25T18:28:25.528 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:28:25.526+0000 7f3717c84640 10 client.4294.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T18:28:25.528 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:28:25.526+0000 7f3717c84640 10 monclient: _renew_subs 2024-03-25T18:28:25.528 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:28:25.526+0000 7f3717c84640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.115:6789/0 2024-03-25T18:28:25.529 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:28:25.526+0000 7f3717c84640 1 -- 172.21.15.1:0/1724883746 --> v1:172.21.15.115:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f3710146410 con 0x7f371013c480 2024-03-25T18:28:25.529 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:28:25.526+0000 7f3717c84640 1 librados: init done 2024-03-25T18:28:25.529 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:28:25.527+0000 7f3717c84640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T18:28:25.529 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:28:25.527+0000 7f3717c84640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:28:25.529 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:28:25.527+0000 7f3717c84640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.115:6789/0 2024-03-25T18:28:25.529 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:28:25.527+0000 7f3717c84640 1 -- 172.21.15.1:0/1724883746 --> v1:172.21.15.115:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f37101466c0 con 0x7f371013c480 2024-03-25T18:28:25.801 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:28:25.801+0000 7f37067fc640 1 -- 172.21.15.1:0/1724883746 <== mon.1 v1:172.21.15.115:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f3708002040 con 0x7f371013c480 2024-03-25T18:28:25.802 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:28:25.801+0000 7f37067fc640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:28:25.802 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:28:25.801+0000 7f37067fc640 1 -- 172.21.15.1:0/1724883746 <== mon.1 v1:172.21.15.115:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1586965025 0 0) 0x7f3708003dc0 con 0x7f371013c480 2024-03-25T18:28:25.802 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:28:25.801+0000 7f37067fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:28:25.802 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:28:25.801+0000 7f37067fc640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-25T18:28:25.802 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:28:25.801+0000 7f37067fc640 10 monclient: dump: 2024-03-25T18:28:25.802 INFO:teuthology.misc.health.smithi001.stderr:epoch 1 2024-03-25T18:28:25.802 INFO:teuthology.misc.health.smithi001.stderr:fsid c8ea72cf-9e99-4383-9b52-6c23d365fdfc 2024-03-25T18:28:25.802 INFO:teuthology.misc.health.smithi001.stderr:last_changed 2024-03-25T18:25:50.510519+0000 2024-03-25T18:28:25.802 INFO:teuthology.misc.health.smithi001.stderr:created 2024-03-25T18:25:50.510519+0000 2024-03-25T18:28:25.802 INFO:teuthology.misc.health.smithi001.stderr:min_mon_release 19 (squid) 2024-03-25T18:28:25.802 INFO:teuthology.misc.health.smithi001.stderr:election_strategy: 3 2024-03-25T18:28:25.802 INFO:teuthology.misc.health.smithi001.stderr:0: v1:172.21.15.1:6789/0 mon.a 2024-03-25T18:28:25.802 INFO:teuthology.misc.health.smithi001.stderr:1: v1:172.21.15.115:6789/0 mon.b 2024-03-25T18:28:25.803 INFO:teuthology.misc.health.smithi001.stderr:2: v1:172.21.15.1:6790/0 mon.c 2024-03-25T18:28:25.803 INFO:teuthology.misc.health.smithi001.stderr: 2024-03-25T18:28:25.803 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:28:25.801+0000 7f3707fff640 4 set_mon_vals no callback set 2024-03-25T18:28:25.803 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:28:25.802+0000 7f37067fc640 1 -- 172.21.15.1:0/1724883746 <== mon.1 v1:172.21.15.115:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 21787975 0 0) 0x7f3708003fe0 con 0x7f371013c480 2024-03-25T18:28:25.803 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:28:25.802+0000 7f3715a20640 10 monclient: get_auth_request con 0x7f36f406df10 auth_method 0 2024-03-25T18:28:25.803 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:28:25.803+0000 7f37067fc640 10 client.4294.objecter ms_handle_connect 0x7f36f406df10 2024-03-25T18:28:25.808 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:28:25.807+0000 7f37067fc640 1 -- 172.21.15.1:0/1724883746 <== mon.1 v1:172.21.15.115:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 996254230 0 0) 0x7f3708054a40 con 0x7f371013c480 2024-03-25T18:28:25.808 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:28:25.807+0000 7f37067fc640 10 client.4294.objecter ms_dispatch 0x7f3710000b90 osd_map(10..10 src has 1..10) v4 2024-03-25T18:28:25.808 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:28:25.807+0000 7f37067fc640 3 client.4294.objecter handle_osd_map got epochs [10,10] > 0 2024-03-25T18:28:25.808 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:28:25.807+0000 7f37067fc640 3 client.4294.objecter handle_osd_map decoding full epoch 10 2024-03-25T18:28:25.808 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:28:25.807+0000 7f37067fc640 20 client.4294.objecter dump_active .. 0 homeless 2024-03-25T18:28:25.808 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:28:25.807+0000 7f37067fc640 1 -- 172.21.15.1:0/1724883746 <== mon.1 v1:172.21.15.115:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f3708050550 con 0x7f371013c480 2024-03-25T18:28:25.808 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:28:25.807+0000 7f37067fc640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:28:25.808 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:28:25.807+0000 7f37067fc640 10 monclient: _finish_command 1 = system:0 2024-03-25T18:28:26.027 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:28:26.026+0000 7f3717c84640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:28:26.027 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:28:26.026+0000 7f3717c84640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:28:26.027 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:28:26.026+0000 7f3717c84640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.115:6789/0 2024-03-25T18:28:26.027 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:28:26.026+0000 7f3717c84640 1 -- 172.21.15.1:0/1724883746 --> v1:172.21.15.115:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f37100688c0 con 0x7f371013c480 2024-03-25T18:28:26.055 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:28:26.054+0000 7f37067fc640 1 -- 172.21.15.1:0/1724883746 <== mon.1 v1:172.21.15.115:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 4092213348) 0x7f3708052480 con 0x7f371013c480 2024-03-25T18:28:26.055 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:28:26.054+0000 7f37067fc640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:28:26.055 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:28:26.054+0000 7f37067fc640 10 monclient: _finish_command 2 = system:0 2024-03-25T18:28:26.055 INFO:teuthology.misc.health.smithi001.stdout: 2024-03-25T18:28:26.055 INFO:teuthology.misc.health.smithi001.stdout:{"epoch":10,"fsid":"c8ea72cf-9e99-4383-9b52-6c23d365fdfc","created":"2024-03-25T18:26:38.856638+0000","modified":"2024-03-25T18:26:44.224208+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T18:26:44.224208+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"b9f34922-3a84-49be-8426-807c1eceaefc","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"42bf306f-55e3-4634-9874-8b4b73ac6644","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"292cb23a-18a9-4207-a7cc-4d5727feaec0","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"45e0e9ab-898d-4673-9590-b4117bc5e84a","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"831a5a09-73ab-4a61-9872-088e76bdbc66","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"88cd903d-3b55-4840-9396-6e3cfe005282","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"89db7355-bc9d-46f4-8ab0-e8d9e744e026","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"cac47bee-0914-4f08-896f-348838882866","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T18:28:26.058 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:28:26.056+0000 7f3717c84640 10 librados: watch_flush enter 2024-03-25T18:28:26.058 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:28:26.056+0000 7f3717c84640 10 librados: watch_flush exit 2024-03-25T18:28:26.058 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:28:26.056+0000 7f3717c84640 20 client.4294.objecter shutdown clearing up homeless session... 2024-03-25T18:28:26.058 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:28:26.056+0000 7f3717c84640 10 client.4294.objecter successfully canceled tick 2024-03-25T18:28:26.058 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:28:26.057+0000 7f3717c84640 1 -- 172.21.15.1:0/1724883746 >> v1:172.21.15.115:6800/4237379176 conn(0x7f36f406df10 legacy=0x7f36f40703d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:28:26.058 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:28:26.057+0000 7f3717c84640 10 monclient: shutdown 2024-03-25T18:28:26.058 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:28:26.057+0000 7f3717c84640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:28:26.058 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:28:26.057+0000 7f3717c84640 1 -- 172.21.15.1:0/1724883746 >> v1:172.21.15.115:6789/0 conn(0x7f371013c480 legacy=0x7f37101303c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:28:26.058 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:28:26.057+0000 7f3717c84640 1 -- 172.21.15.1:0/1724883746 shutdown_connections 2024-03-25T18:28:26.058 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:28:26.057+0000 7f3717c84640 1 -- 172.21.15.1:0/1724883746 >> 172.21.15.1:0/1724883746 conn(0x7f37100617a0 msgr2=0x7f3710141ad0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:28:26.058 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:28:26.057+0000 7f3717c84640 1 -- 172.21.15.1:0/1724883746 shutdown_connections 2024-03-25T18:28:26.058 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:28:26.057+0000 7f3717c84640 1 -- 172.21.15.1:0/1724883746 wait complete. 2024-03-25T18:28:26.058 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:28:26.057+0000 7f3717c84640 1 librados: shutdown 2024-03-25T18:28:26.068 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T18:28:32.876 DEBUG:teuthology.orchestra.run.smithi001:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T18:28:32.973 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:28:32.972+0000 7fdbf410d640 10 monclient: get_monmap_and_config 2024-03-25T18:28:32.973 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:28:32.972+0000 7fdbf410d640 10 monclient: build_initial_monmap 2024-03-25T18:28:32.973 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:28:32.972+0000 7fdbf410d640 10 monclient: monmap: 2024-03-25T18:28:32.974 INFO:teuthology.misc.health.smithi001.stderr:epoch 0 2024-03-25T18:28:32.974 INFO:teuthology.misc.health.smithi001.stderr:fsid c8ea72cf-9e99-4383-9b52-6c23d365fdfc 2024-03-25T18:28:32.974 INFO:teuthology.misc.health.smithi001.stderr:last_changed 2024-03-25T18:28:32.973516+0000 2024-03-25T18:28:32.974 INFO:teuthology.misc.health.smithi001.stderr:created 2024-03-25T18:28:32.973516+0000 2024-03-25T18:28:32.974 INFO:teuthology.misc.health.smithi001.stderr:min_mon_release 0 (unknown) 2024-03-25T18:28:32.974 INFO:teuthology.misc.health.smithi001.stderr:election_strategy: 3 2024-03-25T18:28:32.974 INFO:teuthology.misc.health.smithi001.stderr:0: v1:172.21.15.1:6789/0 mon.noname-a 2024-03-25T18:28:32.974 INFO:teuthology.misc.health.smithi001.stderr:1: v1:172.21.15.115:6789/0 mon.noname-c 2024-03-25T18:28:32.974 INFO:teuthology.misc.health.smithi001.stderr:2: v1:172.21.15.1:6790/0 mon.noname-b 2024-03-25T18:28:32.974 INFO:teuthology.misc.health.smithi001.stderr: 2024-03-25T18:28:32.975 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:28:32.973+0000 7fdbf410d640 1 Processor -- start 2024-03-25T18:28:32.975 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:28:32.973+0000 7fdbf410d640 1 -- start start 2024-03-25T18:28:32.975 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:28:32.973+0000 7fdbf410d640 10 monclient: init 2024-03-25T18:28:32.975 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:28:32.974+0000 7fdbf410d640 10 monclient: _reopen_session rank -1 2024-03-25T18:28:32.975 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:28:32.974+0000 7fdbf410d640 10 monclient: _add_conns ranks=[1,0,2] 2024-03-25T18:28:32.976 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:28:32.974+0000 7fdbf410d640 10 monclient(hunting): picked mon.noname-c con 0x7fdbec0b4ff0 addr v1:172.21.15.115:6789/0 2024-03-25T18:28:32.976 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:28:32.974+0000 7fdbf410d640 10 monclient(hunting): picked mon.noname-a con 0x7fdbec0b4670 addr v1:172.21.15.1:6789/0 2024-03-25T18:28:32.976 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:28:32.974+0000 7fdbf410d640 10 monclient(hunting): picked mon.noname-b con 0x7fdbec13c480 addr v1:172.21.15.1:6790/0 2024-03-25T18:28:32.976 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:28:32.974+0000 7fdbf410d640 1 -- --> v1:172.21.15.1:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fdbec0ab0e0 con 0x7fdbec0b4670 2024-03-25T18:28:32.976 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:28:32.974+0000 7fdbf410d640 1 -- --> v1:172.21.15.115:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fdbec0ab3c0 con 0x7fdbec0b4ff0 2024-03-25T18:28:32.976 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:28:32.974+0000 7fdbf410d640 1 -- --> v1:172.21.15.1:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fdbec0a9f30 con 0x7fdbec13c480 2024-03-25T18:28:32.976 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:28:32.974+0000 7fdbf410d640 10 monclient(hunting): _renew_subs 2024-03-25T18:28:32.976 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:28:32.974+0000 7fdbf410d640 10 monclient(hunting): authenticate will time out at 7593.444235s 2024-03-25T18:28:32.989 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:28:32.989+0000 7fdbf16a8640 1 --1- >> v1:172.21.15.1:6789/0 conn(0x7fdbec0b4670 0x7fdbec1425c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.1:6789/0 says I am v1:172.21.15.1:55100/0 (socket says 172.21.15.1:55100) 2024-03-25T18:28:32.989 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:28:32.989+0000 7fdbf16a8640 1 -- 172.21.15.1:0/3939709864 learned_addr learned my addr 172.21.15.1:0/3939709864 (peer_addr_for_me v1:172.21.15.1:0/0) 2024-03-25T18:28:33.018 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:28:33.017+0000 7fdbf0ea7640 1 -- 172.21.15.1:0/3939709864 <== mon.0 v1:172.21.15.1:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2588409796 0 0) 0x7fdbec0ab0e0 con 0x7fdbec0b4670 2024-03-25T18:28:33.018 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:28:33.017+0000 7fdbf0ea7640 10 monclient(hunting): _negotiate 2024-03-25T18:28:33.018 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:28:33.017+0000 7fdbf0ea7640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:28:33.018 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:28:33.017+0000 7fdbf0ea7640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:28:33.018 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:28:33.017+0000 7fdbf0ea7640 10 monclient(hunting): my global_id is 4302 2024-03-25T18:28:33.018 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:28:33.017+0000 7fdbf0ea7640 1 -- 172.21.15.1:0/3939709864 --> v1:172.21.15.1:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fdbd8003520 con 0x7fdbec0b4670 2024-03-25T18:28:33.025 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:28:33.024+0000 7fdbf0ea7640 1 -- 172.21.15.1:0/3939709864 <== mon.0 v1:172.21.15.1:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3902044038 0 0) 0x7fdbd8003520 con 0x7fdbec0b4670 2024-03-25T18:28:33.025 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:28:33.024+0000 7fdbf0ea7640 1 -- 172.21.15.1:0/3939709864 >> v1:172.21.15.1:6790/0 conn(0x7fdbec13c480 legacy=0x7fdbec14dc00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-25T18:28:33.025 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:28:33.025+0000 7fdbf0ea7640 1 -- 172.21.15.1:0/3939709864 >> v1:172.21.15.115:6789/0 conn(0x7fdbec0b4ff0 legacy=0x7fdbec13bd70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:28:33.025 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:28:33.025+0000 7fdbf0ea7640 10 monclient: _finish_hunting 0 2024-03-25T18:28:33.026 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:28:33.025+0000 7fdbf0ea7640 1 monclient: found mon.noname-a 2024-03-25T18:28:33.026 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:28:33.025+0000 7fdbf0ea7640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:28:33.026 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:28:33.025+0000 7fdbf0ea7640 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.1:6789/0 2024-03-25T18:28:33.026 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:28:33.025+0000 7fdbf0ea7640 1 -- 172.21.15.1:0/3939709864 --> v1:172.21.15.1:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdbec0a9b70 con 0x7fdbec0b4670 2024-03-25T18:28:33.026 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:28:33.025+0000 7fdbf0ea7640 10 monclient: _finish_auth 0 2024-03-25T18:28:33.026 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:28:33.025+0000 7fdbf0ea7640 10 monclient: _check_auth_tickets 2024-03-25T18:28:33.026 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:28:33.025+0000 7fdbf0ea7640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:28:33.026 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:28:33.025+0000 7fdbf0ea7640 1 -- 172.21.15.1:0/3939709864 <== mon.0 v1:172.21.15.1:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1586965025 0 0) 0x7fdbdc001690 con 0x7fdbec0b4670 2024-03-25T18:28:33.026 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:28:33.025+0000 7fdbf0ea7640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:28:33.026 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:28:33.025+0000 7fdbf0ea7640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-03-25T18:28:33.026 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:28:33.025+0000 7fdbf0ea7640 10 monclient: dump: 2024-03-25T18:28:33.026 INFO:teuthology.misc.health.smithi001.stderr:epoch 1 2024-03-25T18:28:33.026 INFO:teuthology.misc.health.smithi001.stderr:fsid c8ea72cf-9e99-4383-9b52-6c23d365fdfc 2024-03-25T18:28:33.027 INFO:teuthology.misc.health.smithi001.stderr:last_changed 2024-03-25T18:25:50.510519+0000 2024-03-25T18:28:33.027 INFO:teuthology.misc.health.smithi001.stderr:created 2024-03-25T18:25:50.510519+0000 2024-03-25T18:28:33.027 INFO:teuthology.misc.health.smithi001.stderr:min_mon_release 19 (squid) 2024-03-25T18:28:33.027 INFO:teuthology.misc.health.smithi001.stderr:election_strategy: 3 2024-03-25T18:28:33.027 INFO:teuthology.misc.health.smithi001.stderr:0: v1:172.21.15.1:6789/0 mon.a 2024-03-25T18:28:33.027 INFO:teuthology.misc.health.smithi001.stderr:1: v1:172.21.15.115:6789/0 mon.b 2024-03-25T18:28:33.027 INFO:teuthology.misc.health.smithi001.stderr:2: v1:172.21.15.1:6790/0 mon.c 2024-03-25T18:28:33.027 INFO:teuthology.misc.health.smithi001.stderr: 2024-03-25T18:28:33.027 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:28:33.025+0000 7fdbf410d640 5 monclient: authenticate success, global_id 4302 2024-03-25T18:28:33.027 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:28:33.025+0000 7fdbf410d640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T18:28:33.035 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:28:33.034+0000 7fdbf0ea7640 1 -- 172.21.15.1:0/3939709864 <== mon.0 v1:172.21.15.1:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fdbdc001860 con 0x7fdbec0b4670 2024-03-25T18:28:33.035 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:28:33.034+0000 7fdbf0ea7640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:28:33.035 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:28:33.034+0000 7fdbf0ea7640 1 -- 172.21.15.1:0/3939709864 <== mon.0 v1:172.21.15.1:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1586965025 0 0) 0x7fdbdc001cb0 con 0x7fdbec0b4670 2024-03-25T18:28:33.035 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:28:33.034+0000 7fdbf410d640 10 monclient: get_monmap_and_config success 2024-03-25T18:28:33.035 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:28:33.034+0000 7fdbf410d640 4 set_mon_vals no callback set 2024-03-25T18:28:33.035 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:28:33.034+0000 7fdbf0ea7640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:28:33.035 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:28:33.034+0000 7fdbf0ea7640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-25T18:28:33.035 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:28:33.034+0000 7fdbf0ea7640 10 monclient: dump: 2024-03-25T18:28:33.035 INFO:teuthology.misc.health.smithi001.stderr:epoch 1 2024-03-25T18:28:33.035 INFO:teuthology.misc.health.smithi001.stderr:fsid c8ea72cf-9e99-4383-9b52-6c23d365fdfc 2024-03-25T18:28:33.035 INFO:teuthology.misc.health.smithi001.stderr:last_changed 2024-03-25T18:25:50.510519+0000 2024-03-25T18:28:33.035 INFO:teuthology.misc.health.smithi001.stderr:created 2024-03-25T18:25:50.510519+0000 2024-03-25T18:28:33.035 INFO:teuthology.misc.health.smithi001.stderr:min_mon_release 19 (squid) 2024-03-25T18:28:33.036 INFO:teuthology.misc.health.smithi001.stderr:election_strategy: 3 2024-03-25T18:28:33.036 INFO:teuthology.misc.health.smithi001.stderr:0: v1:172.21.15.1:6789/0 mon.a 2024-03-25T18:28:33.036 INFO:teuthology.misc.health.smithi001.stderr:1: v1:172.21.15.115:6789/0 mon.b 2024-03-25T18:28:33.036 INFO:teuthology.misc.health.smithi001.stderr:2: v1:172.21.15.1:6790/0 mon.c 2024-03-25T18:28:33.036 INFO:teuthology.misc.health.smithi001.stderr: 2024-03-25T18:28:33.038 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:28:33.035+0000 7fdbf410d640 10 monclient: shutdown 2024-03-25T18:28:33.038 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:28:33.035+0000 7fdbf410d640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:28:33.039 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:28:33.035+0000 7fdbf410d640 1 -- 172.21.15.1:0/3939709864 >> v1:172.21.15.1:6789/0 conn(0x7fdbec0b4670 legacy=0x7fdbec1425c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:28:33.039 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:28:33.035+0000 7fdbf410d640 1 -- 172.21.15.1:0/3939709864 shutdown_connections 2024-03-25T18:28:33.039 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:28:33.035+0000 7fdbf410d640 1 -- 172.21.15.1:0/3939709864 >> 172.21.15.1:0/3939709864 conn(0x7fdbec0617a0 msgr2=0x7fdbec061bb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:28:33.039 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:28:33.035+0000 7fdbf410d640 1 -- 172.21.15.1:0/3939709864 shutdown_connections 2024-03-25T18:28:33.039 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:28:33.035+0000 7fdbf410d640 1 -- 172.21.15.1:0/3939709864 wait complete. 2024-03-25T18:28:33.039 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:28:33.036+0000 7fdbf410d640 10 monclient: build_initial_monmap 2024-03-25T18:28:33.039 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:28:33.036+0000 7fdbf410d640 10 monclient: monmap: 2024-03-25T18:28:33.039 INFO:teuthology.misc.health.smithi001.stderr:epoch 0 2024-03-25T18:28:33.039 INFO:teuthology.misc.health.smithi001.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T18:28:33.039 INFO:teuthology.misc.health.smithi001.stderr:last_changed 0.000000 2024-03-25T18:28:33.039 INFO:teuthology.misc.health.smithi001.stderr:created 0.000000 2024-03-25T18:28:33.039 INFO:teuthology.misc.health.smithi001.stderr:min_mon_release 0 (unknown) 2024-03-25T18:28:33.039 INFO:teuthology.misc.health.smithi001.stderr:election_strategy: 1 2024-03-25T18:28:33.039 INFO:teuthology.misc.health.smithi001.stderr:0: v1:172.21.15.1:6789/0 mon.noname-a 2024-03-25T18:28:33.039 INFO:teuthology.misc.health.smithi001.stderr:1: v1:172.21.15.115:6789/0 mon.noname-b 2024-03-25T18:28:33.039 INFO:teuthology.misc.health.smithi001.stderr:2: v1:172.21.15.1:6790/0 mon.noname-c 2024-03-25T18:28:33.040 INFO:teuthology.misc.health.smithi001.stderr: 2024-03-25T18:28:33.040 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:28:33.036+0000 7fdbf410d640 1 librados: starting msgr at 2024-03-25T18:28:33.040 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:28:33.036+0000 7fdbf410d640 1 librados: starting objecter 2024-03-25T18:28:33.040 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:28:33.036+0000 7fdbf410d640 1 Processor -- start 2024-03-25T18:28:33.040 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:28:33.036+0000 7fdbf410d640 1 -- start start 2024-03-25T18:28:33.040 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:28:33.036+0000 7fdbf410d640 1 librados: setting wanted keys 2024-03-25T18:28:33.040 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:28:33.036+0000 7fdbf410d640 1 librados: calling monclient init 2024-03-25T18:28:33.040 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:28:33.036+0000 7fdbf410d640 10 monclient: init 2024-03-25T18:28:33.040 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:28:33.037+0000 7fdbf410d640 10 monclient: _reopen_session rank -1 2024-03-25T18:28:33.040 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:28:33.037+0000 7fdbf410d640 10 monclient: _add_conns ranks=[2,1,0] 2024-03-25T18:28:33.040 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:28:33.037+0000 7fdbf410d640 10 monclient(hunting): picked mon.noname-c con 0x7fdbec0b4670 addr v1:172.21.15.1:6790/0 2024-03-25T18:28:33.040 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:28:33.037+0000 7fdbf410d640 10 monclient(hunting): picked mon.noname-b con 0x7fdbec0b4ff0 addr v1:172.21.15.115:6789/0 2024-03-25T18:28:33.040 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:28:33.037+0000 7fdbf410d640 10 monclient(hunting): picked mon.noname-a con 0x7fdbec13c480 addr v1:172.21.15.1:6789/0 2024-03-25T18:28:33.040 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:28:33.037+0000 7fdbf410d640 1 -- --> v1:172.21.15.1:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fdbec143790 con 0x7fdbec13c480 2024-03-25T18:28:33.040 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:28:33.037+0000 7fdbf410d640 1 -- --> v1:172.21.15.115:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fdbec1446c0 con 0x7fdbec0b4ff0 2024-03-25T18:28:33.041 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:28:33.037+0000 7fdbf410d640 1 -- --> v1:172.21.15.1:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fdbec144930 con 0x7fdbec0b4670 2024-03-25T18:28:33.041 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:28:33.037+0000 7fdbf410d640 10 monclient(hunting): _renew_subs 2024-03-25T18:28:33.041 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:28:33.037+0000 7fdbf410d640 10 monclient(hunting): authenticate will time out at 7593.506908s 2024-03-25T18:28:33.041 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:28:33.037+0000 7fdbf16a8640 1 --1- >> v1:172.21.15.115:6789/0 conn(0x7fdbec0b4ff0 0x7fdbec145820 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.115:6789/0 says I am v1:172.21.15.1:52908/0 (socket says 172.21.15.1:52908) 2024-03-25T18:28:33.041 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:28:33.037+0000 7fdbf16a8640 1 -- 172.21.15.1:0/1832734026 learned_addr learned my addr 172.21.15.1:0/1832734026 (peer_addr_for_me v1:172.21.15.1:0/0) 2024-03-25T18:28:33.041 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:28:33.037+0000 7fdbe27fc640 10 client.?.objecter ms_handle_connect 0x7fdbec0b4ff0 2024-03-25T18:28:33.041 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:28:33.037+0000 7fdbe27fc640 10 client.?.objecter resend_mon_ops 2024-03-25T18:28:33.053 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:28:33.052+0000 7fdbe27fc640 10 client.?.objecter ms_handle_connect 0x7fdbec0b4670 2024-03-25T18:28:33.053 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:28:33.052+0000 7fdbe27fc640 10 client.?.objecter resend_mon_ops 2024-03-25T18:28:33.058 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:28:33.058+0000 7fdbe27fc640 10 client.?.objecter ms_handle_connect 0x7fdbec13c480 2024-03-25T18:28:33.059 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:28:33.058+0000 7fdbe27fc640 10 client.?.objecter resend_mon_ops 2024-03-25T18:28:33.124 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:28:33.123+0000 7fdbe27fc640 1 -- 172.21.15.1:0/1832734026 <== mon.0 v1:172.21.15.1:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3986754692 0 0) 0x7fdbec143790 con 0x7fdbec13c480 2024-03-25T18:28:33.124 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:28:33.123+0000 7fdbe27fc640 10 monclient(hunting): _negotiate 2024-03-25T18:28:33.124 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:28:33.123+0000 7fdbe27fc640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:28:33.124 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:28:33.123+0000 7fdbe27fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:28:33.124 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:28:33.123+0000 7fdbe27fc640 10 monclient(hunting): my global_id is 4305 2024-03-25T18:28:33.124 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:28:33.123+0000 7fdbe27fc640 1 -- 172.21.15.1:0/1832734026 --> v1:172.21.15.1:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fdbd0004eb0 con 0x7fdbec13c480 2024-03-25T18:28:33.137 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:28:33.136+0000 7fdbe27fc640 1 -- 172.21.15.1:0/1832734026 <== mon.2 v1:172.21.15.1:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1174846102 0 0) 0x7fdbec144930 con 0x7fdbec0b4670 2024-03-25T18:28:33.137 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:28:33.136+0000 7fdbe27fc640 10 monclient(hunting): _negotiate 2024-03-25T18:28:33.137 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:28:33.136+0000 7fdbe27fc640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:28:33.137 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:28:33.136+0000 7fdbe27fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:28:33.137 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:28:33.136+0000 7fdbe27fc640 10 monclient(hunting): my global_id is 4304 2024-03-25T18:28:33.137 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:28:33.136+0000 7fdbe27fc640 1 -- 172.21.15.1:0/1832734026 --> v1:172.21.15.1:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fdbec143790 con 0x7fdbec0b4670 2024-03-25T18:28:33.151 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:28:33.150+0000 7fdbe27fc640 1 -- 172.21.15.1:0/1832734026 <== mon.0 v1:172.21.15.1:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 4212307274 0 0) 0x7fdbd0004eb0 con 0x7fdbec13c480 2024-03-25T18:28:33.151 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:28:33.150+0000 7fdbe27fc640 1 -- 172.21.15.1:0/1832734026 --> v1:172.21.15.1:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fdbec144930 con 0x7fdbec13c480 2024-03-25T18:28:33.151 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:28:33.150+0000 7fdbe27fc640 1 -- 172.21.15.1:0/1832734026 <== mon.0 v1:172.21.15.1:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1586965025 0 0) 0x7fdbe4001ad0 con 0x7fdbec13c480 2024-03-25T18:28:33.151 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:28:33.150+0000 7fdbe27fc640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:28:33.151 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:28:33.150+0000 7fdbe27fc640 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2024-03-25T18:28:33.151 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:28:33.150+0000 7fdbe27fc640 10 monclient(hunting): dump: 2024-03-25T18:28:33.151 INFO:teuthology.misc.health.smithi001.stderr:epoch 1 2024-03-25T18:28:33.151 INFO:teuthology.misc.health.smithi001.stderr:fsid c8ea72cf-9e99-4383-9b52-6c23d365fdfc 2024-03-25T18:28:33.151 INFO:teuthology.misc.health.smithi001.stderr:last_changed 2024-03-25T18:25:50.510519+0000 2024-03-25T18:28:33.151 INFO:teuthology.misc.health.smithi001.stderr:created 2024-03-25T18:25:50.510519+0000 2024-03-25T18:28:33.151 INFO:teuthology.misc.health.smithi001.stderr:min_mon_release 19 (squid) 2024-03-25T18:28:33.151 INFO:teuthology.misc.health.smithi001.stderr:election_strategy: 3 2024-03-25T18:28:33.151 INFO:teuthology.misc.health.smithi001.stderr:0: v1:172.21.15.1:6789/0 mon.a 2024-03-25T18:28:33.152 INFO:teuthology.misc.health.smithi001.stderr:1: v1:172.21.15.115:6789/0 mon.b 2024-03-25T18:28:33.152 INFO:teuthology.misc.health.smithi001.stderr:2: v1:172.21.15.1:6790/0 mon.c 2024-03-25T18:28:33.152 INFO:teuthology.misc.health.smithi001.stderr: 2024-03-25T18:28:33.152 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:28:33.150+0000 7fdbe27fc640 10 monclient(hunting): _finish_auth 0 2024-03-25T18:28:33.168 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:28:33.167+0000 7fdbe27fc640 1 -- 172.21.15.1:0/1832734026 <== mon.2 v1:172.21.15.1:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2719918457 0 0) 0x7fdbec143790 con 0x7fdbec0b4670 2024-03-25T18:28:33.168 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:28:33.167+0000 7fdbe27fc640 1 -- 172.21.15.1:0/1832734026 --> v1:172.21.15.1:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fdbd0004eb0 con 0x7fdbec0b4670 2024-03-25T18:28:33.168 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:28:33.167+0000 7fdbe27fc640 1 -- 172.21.15.1:0/1832734026 <== mon.2 v1:172.21.15.1:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1586965025 0 0) 0x7fdbe80018b0 con 0x7fdbec0b4670 2024-03-25T18:28:33.168 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:28:33.167+0000 7fdbe27fc640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:28:33.168 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:28:33.167+0000 7fdbe27fc640 10 monclient(hunting): got monmap 1 from mon.c (according to old e1) 2024-03-25T18:28:33.168 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:28:33.167+0000 7fdbe27fc640 10 monclient(hunting): dump: 2024-03-25T18:28:33.168 INFO:teuthology.misc.health.smithi001.stderr:epoch 1 2024-03-25T18:28:33.168 INFO:teuthology.misc.health.smithi001.stderr:fsid c8ea72cf-9e99-4383-9b52-6c23d365fdfc 2024-03-25T18:28:33.169 INFO:teuthology.misc.health.smithi001.stderr:last_changed 2024-03-25T18:25:50.510519+0000 2024-03-25T18:28:33.169 INFO:teuthology.misc.health.smithi001.stderr:created 2024-03-25T18:25:50.510519+0000 2024-03-25T18:28:33.169 INFO:teuthology.misc.health.smithi001.stderr:min_mon_release 19 (squid) 2024-03-25T18:28:33.169 INFO:teuthology.misc.health.smithi001.stderr:election_strategy: 3 2024-03-25T18:28:33.169 INFO:teuthology.misc.health.smithi001.stderr:0: v1:172.21.15.1:6789/0 mon.a 2024-03-25T18:28:33.169 INFO:teuthology.misc.health.smithi001.stderr:1: v1:172.21.15.115:6789/0 mon.b 2024-03-25T18:28:33.169 INFO:teuthology.misc.health.smithi001.stderr:2: v1:172.21.15.1:6790/0 mon.c 2024-03-25T18:28:33.169 INFO:teuthology.misc.health.smithi001.stderr: 2024-03-25T18:28:33.185 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:28:33.184+0000 7fdbe27fc640 1 -- 172.21.15.1:0/1832734026 <== mon.2 v1:172.21.15.1:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 232496480 0 0) 0x7fdbd0004eb0 con 0x7fdbec0b4670 2024-03-25T18:28:33.185 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:28:33.184+0000 7fdbe27fc640 1 -- 172.21.15.1:0/1832734026 >> v1:172.21.15.115:6789/0 conn(0x7fdbec0b4ff0 legacy=0x7fdbec145820 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:28:33.185 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:28:33.184+0000 7fdbe27fc640 1 -- 172.21.15.1:0/1832734026 >> v1:172.21.15.1:6789/0 conn(0x7fdbec13c480 legacy=0x7fdbec1e2600 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:28:33.186 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:28:33.184+0000 7fdbe27fc640 10 monclient: _finish_hunting 0 2024-03-25T18:28:33.186 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:28:33.184+0000 7fdbe27fc640 1 monclient: found mon.c 2024-03-25T18:28:33.187 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:28:33.184+0000 7fdbe27fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:28:33.187 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:28:33.184+0000 7fdbe27fc640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.1:6790/0 2024-03-25T18:28:33.187 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:28:33.184+0000 7fdbe27fc640 1 -- 172.21.15.1:0/1832734026 --> v1:172.21.15.1:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdbec144c10 con 0x7fdbec0b4670 2024-03-25T18:28:33.187 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:28:33.184+0000 7fdbe27fc640 10 monclient: _finish_auth 0 2024-03-25T18:28:33.187 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:28:33.184+0000 7fdbe27fc640 10 monclient: _check_auth_tickets 2024-03-25T18:28:33.187 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:28:33.184+0000 7fdbe27fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:28:33.187 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:28:33.184+0000 7fdbf410d640 5 monclient: authenticate success, global_id 4304 2024-03-25T18:28:33.187 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:28:33.184+0000 7fdbf410d640 10 monclient: _renew_subs 2024-03-25T18:28:33.187 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:28:33.184+0000 7fdbf410d640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.1:6790/0 2024-03-25T18:28:33.187 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:28:33.184+0000 7fdbf410d640 1 -- 172.21.15.1:0/1832734026 --> v1:172.21.15.1:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fdbec13c190 con 0x7fdbec0b4670 2024-03-25T18:28:33.188 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:28:33.184+0000 7fdbf410d640 10 client.4304.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T18:28:33.188 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:28:33.184+0000 7fdbf410d640 10 monclient: _renew_subs 2024-03-25T18:28:33.188 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:28:33.184+0000 7fdbf410d640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.1:6790/0 2024-03-25T18:28:33.188 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:28:33.184+0000 7fdbf410d640 1 -- 172.21.15.1:0/1832734026 --> v1:172.21.15.1:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7fdbec137340 con 0x7fdbec0b4670 2024-03-25T18:28:33.188 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:28:33.184+0000 7fdbf410d640 1 librados: init done 2024-03-25T18:28:33.188 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:28:33.185+0000 7fdbf410d640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T18:28:33.188 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:28:33.185+0000 7fdbf410d640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:28:33.188 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:28:33.185+0000 7fdbf410d640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.1:6790/0 2024-03-25T18:28:33.188 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:28:33.185+0000 7fdbf410d640 1 -- 172.21.15.1:0/1832734026 --> v1:172.21.15.1:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fdbec06dbd0 con 0x7fdbec0b4670 2024-03-25T18:28:33.493 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:28:33.493+0000 7fdbe27fc640 1 -- 172.21.15.1:0/1832734026 <== mon.2 v1:172.21.15.1:6790/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fdbe8003610 con 0x7fdbec0b4670 2024-03-25T18:28:33.494 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:28:33.493+0000 7fdbe27fc640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:28:33.494 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:28:33.493+0000 7fdbe27fc640 1 -- 172.21.15.1:0/1832734026 <== mon.2 v1:172.21.15.1:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1586965025 0 0) 0x7fdbe8003a60 con 0x7fdbec0b4670 2024-03-25T18:28:33.494 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:28:33.493+0000 7fdbe27fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:28:33.494 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:28:33.493+0000 7fdbe27fc640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-25T18:28:33.494 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:28:33.493+0000 7fdbe27fc640 10 monclient: dump: 2024-03-25T18:28:33.494 INFO:teuthology.misc.health.smithi001.stderr:epoch 1 2024-03-25T18:28:33.494 INFO:teuthology.misc.health.smithi001.stderr:fsid c8ea72cf-9e99-4383-9b52-6c23d365fdfc 2024-03-25T18:28:33.494 INFO:teuthology.misc.health.smithi001.stderr:last_changed 2024-03-25T18:25:50.510519+0000 2024-03-25T18:28:33.494 INFO:teuthology.misc.health.smithi001.stderr:created 2024-03-25T18:25:50.510519+0000 2024-03-25T18:28:33.494 INFO:teuthology.misc.health.smithi001.stderr:min_mon_release 19 (squid) 2024-03-25T18:28:33.495 INFO:teuthology.misc.health.smithi001.stderr:election_strategy: 3 2024-03-25T18:28:33.495 INFO:teuthology.misc.health.smithi001.stderr:0: v1:172.21.15.1:6789/0 mon.a 2024-03-25T18:28:33.495 INFO:teuthology.misc.health.smithi001.stderr:1: v1:172.21.15.115:6789/0 mon.b 2024-03-25T18:28:33.495 INFO:teuthology.misc.health.smithi001.stderr:2: v1:172.21.15.1:6790/0 mon.c 2024-03-25T18:28:33.495 INFO:teuthology.misc.health.smithi001.stderr: 2024-03-25T18:28:33.495 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:28:33.493+0000 7fdbe3fff640 4 set_mon_vals no callback set 2024-03-25T18:28:33.495 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:28:33.494+0000 7fdbe27fc640 1 -- 172.21.15.1:0/1832734026 <== mon.2 v1:172.21.15.1:6790/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 21787975 0 0) 0x7fdbe8003c80 con 0x7fdbec0b4670 2024-03-25T18:28:33.495 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:28:33.494+0000 7fdbf16a8640 10 monclient: get_auth_request con 0x7fdbd006de90 auth_method 0 2024-03-25T18:28:33.496 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:28:33.495+0000 7fdbe27fc640 10 client.4304.objecter ms_handle_connect 0x7fdbd006de90 2024-03-25T18:28:33.503 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:28:33.502+0000 7fdbe27fc640 1 -- 172.21.15.1:0/1832734026 <== mon.2 v1:172.21.15.1:6790/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 996254230 0 0) 0x7fdbe80535b0 con 0x7fdbec0b4670 2024-03-25T18:28:33.503 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:28:33.502+0000 7fdbe27fc640 10 client.4304.objecter ms_dispatch 0x7fdbec000b90 osd_map(10..10 src has 1..10) v4 2024-03-25T18:28:33.503 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:28:33.502+0000 7fdbe27fc640 3 client.4304.objecter handle_osd_map got epochs [10,10] > 0 2024-03-25T18:28:33.503 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:28:33.502+0000 7fdbe27fc640 3 client.4304.objecter handle_osd_map decoding full epoch 10 2024-03-25T18:28:33.503 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:28:33.503+0000 7fdbe27fc640 20 client.4304.objecter dump_active .. 0 homeless 2024-03-25T18:28:33.504 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:28:33.503+0000 7fdbe27fc640 1 -- 172.21.15.1:0/1832734026 <== mon.2 v1:172.21.15.1:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7fdbe804f0c0 con 0x7fdbec0b4670 2024-03-25T18:28:33.504 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:28:33.503+0000 7fdbe27fc640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:28:33.504 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:28:33.503+0000 7fdbe27fc640 10 monclient: _finish_command 1 = system:0 2024-03-25T18:28:33.766 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:28:33.765+0000 7fdbf410d640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:28:33.766 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:28:33.765+0000 7fdbf410d640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:28:33.766 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:28:33.765+0000 7fdbf410d640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.1:6790/0 2024-03-25T18:28:33.766 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:28:33.765+0000 7fdbf410d640 1 -- 172.21.15.1:0/1832734026 --> v1:172.21.15.1:6790/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fdbec0688c0 con 0x7fdbec0b4670 2024-03-25T18:28:33.798 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:28:33.797+0000 7fdbe27fc640 1 -- 172.21.15.1:0/1832734026 <== mon.2 v1:172.21.15.1:6790/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 4092213348) 0x7fdbe8050ff0 con 0x7fdbec0b4670 2024-03-25T18:28:33.798 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:28:33.797+0000 7fdbe27fc640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:28:33.798 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:28:33.797+0000 7fdbe27fc640 10 monclient: _finish_command 2 = system:0 2024-03-25T18:28:33.798 INFO:teuthology.misc.health.smithi001.stdout: 2024-03-25T18:28:33.798 INFO:teuthology.misc.health.smithi001.stdout:{"epoch":10,"fsid":"c8ea72cf-9e99-4383-9b52-6c23d365fdfc","created":"2024-03-25T18:26:38.856638+0000","modified":"2024-03-25T18:26:44.224208+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T18:26:44.224208+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"b9f34922-3a84-49be-8426-807c1eceaefc","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"42bf306f-55e3-4634-9874-8b4b73ac6644","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"292cb23a-18a9-4207-a7cc-4d5727feaec0","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"45e0e9ab-898d-4673-9590-b4117bc5e84a","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"831a5a09-73ab-4a61-9872-088e76bdbc66","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"88cd903d-3b55-4840-9396-6e3cfe005282","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"89db7355-bc9d-46f4-8ab0-e8d9e744e026","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"cac47bee-0914-4f08-896f-348838882866","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T18:28:33.800 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:28:33.799+0000 7fdbf410d640 10 librados: watch_flush enter 2024-03-25T18:28:33.801 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:28:33.799+0000 7fdbf410d640 10 librados: watch_flush exit 2024-03-25T18:28:33.801 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:28:33.799+0000 7fdbf410d640 20 client.4304.objecter shutdown clearing up homeless session... 2024-03-25T18:28:33.801 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:28:33.799+0000 7fdbf410d640 10 client.4304.objecter successfully canceled tick 2024-03-25T18:28:33.801 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:28:33.799+0000 7fdbf410d640 1 -- 172.21.15.1:0/1832734026 >> v1:172.21.15.115:6800/4237379176 conn(0x7fdbd006de90 legacy=0x7fdbd0070350 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:28:33.801 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:28:33.799+0000 7fdbf410d640 10 monclient: shutdown 2024-03-25T18:28:33.801 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:28:33.799+0000 7fdbf410d640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:28:33.801 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:28:33.799+0000 7fdbf410d640 1 -- 172.21.15.1:0/1832734026 >> v1:172.21.15.1:6790/0 conn(0x7fdbec0b4670 legacy=0x7fdbec145110 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:28:33.801 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:28:33.799+0000 7fdbf410d640 1 -- 172.21.15.1:0/1832734026 shutdown_connections 2024-03-25T18:28:33.801 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:28:33.799+0000 7fdbf410d640 1 -- 172.21.15.1:0/1832734026 >> 172.21.15.1:0/1832734026 conn(0x7fdbec0617a0 msgr2=0x7fdbec0b7030 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:28:33.801 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:28:33.799+0000 7fdbf410d640 1 -- 172.21.15.1:0/1832734026 shutdown_connections 2024-03-25T18:28:33.801 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:28:33.799+0000 7fdbf410d640 1 -- 172.21.15.1:0/1832734026 wait complete. 2024-03-25T18:28:33.801 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:28:33.800+0000 7fdbf410d640 1 librados: shutdown 2024-03-25T18:28:33.812 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T18:28:36.198 INFO:tasks.ceph.mon.a.smithi001.stderr:==00:00:02:14.756 34876== Warning: unimplemented fcntl command: 1036 2024-03-25T18:28:36.204 INFO:tasks.ceph.mon.a.smithi001.stderr:==00:00:02:14.762 34876== Warning: unimplemented fcntl command: 1036 2024-03-25T18:28:36.240 INFO:tasks.ceph.mon.a.smithi001.stderr:==00:00:02:14.798 34876== Warning: unimplemented fcntl command: 1036 2024-03-25T18:28:36.252 INFO:tasks.ceph.mon.b.smithi115.stderr:==00:00:02:14.735 34836== Warning: unimplemented fcntl command: 1036 2024-03-25T18:28:36.264 INFO:tasks.ceph.mon.b.smithi115.stderr:==00:00:02:14.748 34836== Warning: unimplemented fcntl command: 1036 2024-03-25T18:28:36.267 INFO:tasks.ceph.mon.c.smithi001.stderr:==00:00:02:14.825 34877== Warning: unimplemented fcntl command: 1036 2024-03-25T18:28:36.275 INFO:tasks.ceph.mon.c.smithi001.stderr:==00:00:02:14.833 34877== Warning: unimplemented fcntl command: 1036 2024-03-25T18:28:36.285 INFO:tasks.ceph.mon.b.smithi115.stderr:==00:00:02:14.769 34836== Warning: unimplemented fcntl command: 1036 2024-03-25T18:28:36.306 INFO:tasks.ceph.mon.c.smithi001.stderr:==00:00:02:14.865 34877== Warning: unimplemented fcntl command: 1036 2024-03-25T18:28:40.620 DEBUG:teuthology.orchestra.run.smithi001:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T18:28:40.736 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:28:40.735+0000 7f203e5f4640 10 monclient: get_monmap_and_config 2024-03-25T18:28:40.736 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:28:40.735+0000 7f203e5f4640 10 monclient: build_initial_monmap 2024-03-25T18:28:40.736 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:28:40.735+0000 7f203e5f4640 10 monclient: monmap: 2024-03-25T18:28:40.736 INFO:teuthology.misc.health.smithi001.stderr:epoch 0 2024-03-25T18:28:40.736 INFO:teuthology.misc.health.smithi001.stderr:fsid c8ea72cf-9e99-4383-9b52-6c23d365fdfc 2024-03-25T18:28:40.737 INFO:teuthology.misc.health.smithi001.stderr:last_changed 2024-03-25T18:28:40.736270+0000 2024-03-25T18:28:40.737 INFO:teuthology.misc.health.smithi001.stderr:created 2024-03-25T18:28:40.736270+0000 2024-03-25T18:28:40.737 INFO:teuthology.misc.health.smithi001.stderr:min_mon_release 0 (unknown) 2024-03-25T18:28:40.737 INFO:teuthology.misc.health.smithi001.stderr:election_strategy: 3 2024-03-25T18:28:40.737 INFO:teuthology.misc.health.smithi001.stderr:0: v1:172.21.15.1:6789/0 mon.noname-a 2024-03-25T18:28:40.737 INFO:teuthology.misc.health.smithi001.stderr:1: v1:172.21.15.115:6789/0 mon.noname-c 2024-03-25T18:28:40.737 INFO:teuthology.misc.health.smithi001.stderr:2: v1:172.21.15.1:6790/0 mon.noname-b 2024-03-25T18:28:40.737 INFO:teuthology.misc.health.smithi001.stderr: 2024-03-25T18:28:40.738 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:28:40.736+0000 7f203e5f4640 1 Processor -- start 2024-03-25T18:28:40.738 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:28:40.736+0000 7f203e5f4640 1 -- start start 2024-03-25T18:28:40.738 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:28:40.736+0000 7f203e5f4640 10 monclient: init 2024-03-25T18:28:40.739 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:28:40.737+0000 7f203e5f4640 10 monclient: _reopen_session rank -1 2024-03-25T18:28:40.739 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:28:40.737+0000 7f203e5f4640 10 monclient: _add_conns ranks=[1,2,0] 2024-03-25T18:28:40.739 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:28:40.737+0000 7f203e5f4640 10 monclient(hunting): picked mon.noname-c con 0x7f203806f820 addr v1:172.21.15.115:6789/0 2024-03-25T18:28:40.739 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:28:40.737+0000 7f203e5f4640 10 monclient(hunting): picked mon.noname-b con 0x7f203813f720 addr v1:172.21.15.1:6790/0 2024-03-25T18:28:40.739 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:28:40.737+0000 7f203e5f4640 10 monclient(hunting): picked mon.noname-a con 0x7f203813c8d0 addr v1:172.21.15.1:6789/0 2024-03-25T18:28:40.739 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:28:40.737+0000 7f203e5f4640 1 -- --> v1:172.21.15.1:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2038071a70 con 0x7f203813c8d0 2024-03-25T18:28:40.739 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:28:40.737+0000 7f203e5f4640 1 -- --> v1:172.21.15.115:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2038072820 con 0x7f203806f820 2024-03-25T18:28:40.739 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:28:40.737+0000 7f203e5f4640 1 -- --> v1:172.21.15.1:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2038072d40 con 0x7f203813f720 2024-03-25T18:28:40.739 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:28:40.737+0000 7f203e5f4640 10 monclient(hunting): _renew_subs 2024-03-25T18:28:40.739 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:28:40.737+0000 7f203e5f4640 10 monclient(hunting): authenticate will time out at 7601.207474s 2024-03-25T18:28:40.739 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:28:40.738+0000 7f2037fff640 1 --1- >> v1:172.21.15.115:6789/0 conn(0x7f203806f820 0x7f203813f010 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.115:6789/0 says I am v1:172.21.15.1:57448/0 (socket says 172.21.15.1:57448) 2024-03-25T18:28:40.740 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:28:40.738+0000 7f2037fff640 1 -- 172.21.15.1:0/1737422684 learned_addr learned my addr 172.21.15.1:0/1737422684 (peer_addr_for_me v1:172.21.15.1:0/0) 2024-03-25T18:28:40.786 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:28:40.786+0000 7f2036ffd640 1 -- 172.21.15.1:0/1737422684 <== mon.0 v1:172.21.15.1:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4084704700 0 0) 0x7f2038071a70 con 0x7f203813c8d0 2024-03-25T18:28:40.787 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:28:40.786+0000 7f2036ffd640 10 monclient(hunting): _negotiate 2024-03-25T18:28:40.787 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:28:40.786+0000 7f2036ffd640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:28:40.787 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:28:40.786+0000 7f2036ffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:28:40.787 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:28:40.786+0000 7f2036ffd640 10 monclient(hunting): my global_id is 4308 2024-03-25T18:28:40.787 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:28:40.786+0000 7f2036ffd640 1 -- 172.21.15.1:0/1737422684 --> v1:172.21.15.1:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f2020003520 con 0x7f203813c8d0 2024-03-25T18:28:40.864 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:28:40.863+0000 7f2036ffd640 1 -- 172.21.15.1:0/1737422684 <== mon.0 v1:172.21.15.1:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1613694508 0 0) 0x7f2020003520 con 0x7f203813c8d0 2024-03-25T18:28:40.864 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:28:40.863+0000 7f2036ffd640 1 -- 172.21.15.1:0/1737422684 >> v1:172.21.15.1:6790/0 conn(0x7f203813f720 legacy=0x7f203813c1c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:28:40.864 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:28:40.863+0000 7f2036ffd640 1 -- 172.21.15.1:0/1737422684 >> v1:172.21.15.115:6789/0 conn(0x7f203806f820 legacy=0x7f203813f010 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:28:40.864 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:28:40.863+0000 7f2036ffd640 10 monclient: _finish_hunting 0 2024-03-25T18:28:40.864 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:28:40.863+0000 7f2036ffd640 1 monclient: found mon.noname-a 2024-03-25T18:28:40.864 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:28:40.863+0000 7f2036ffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:28:40.864 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:28:40.863+0000 7f2036ffd640 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.1:6789/0 2024-03-25T18:28:40.864 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:28:40.863+0000 7f2036ffd640 1 -- 172.21.15.1:0/1737422684 --> v1:172.21.15.1:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f20380709a0 con 0x7f203813c8d0 2024-03-25T18:28:40.864 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:28:40.863+0000 7f2036ffd640 10 monclient: _finish_auth 0 2024-03-25T18:28:40.864 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:28:40.863+0000 7f2036ffd640 10 monclient: _check_auth_tickets 2024-03-25T18:28:40.864 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:28:40.863+0000 7f2036ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:28:40.865 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:28:40.863+0000 7f2036ffd640 1 -- 172.21.15.1:0/1737422684 <== mon.0 v1:172.21.15.1:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1586965025 0 0) 0x7f2030001a10 con 0x7f203813c8d0 2024-03-25T18:28:40.865 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:28:40.863+0000 7f2036ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:28:40.865 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:28:40.863+0000 7f2036ffd640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-03-25T18:28:40.865 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:28:40.863+0000 7f2036ffd640 10 monclient: dump: 2024-03-25T18:28:40.865 INFO:teuthology.misc.health.smithi001.stderr:epoch 1 2024-03-25T18:28:40.865 INFO:teuthology.misc.health.smithi001.stderr:fsid c8ea72cf-9e99-4383-9b52-6c23d365fdfc 2024-03-25T18:28:40.865 INFO:teuthology.misc.health.smithi001.stderr:last_changed 2024-03-25T18:25:50.510519+0000 2024-03-25T18:28:40.865 INFO:teuthology.misc.health.smithi001.stderr:created 2024-03-25T18:25:50.510519+0000 2024-03-25T18:28:40.865 INFO:teuthology.misc.health.smithi001.stderr:min_mon_release 19 (squid) 2024-03-25T18:28:40.865 INFO:teuthology.misc.health.smithi001.stderr:election_strategy: 3 2024-03-25T18:28:40.865 INFO:teuthology.misc.health.smithi001.stderr:0: v1:172.21.15.1:6789/0 mon.a 2024-03-25T18:28:40.865 INFO:teuthology.misc.health.smithi001.stderr:1: v1:172.21.15.115:6789/0 mon.b 2024-03-25T18:28:40.865 INFO:teuthology.misc.health.smithi001.stderr:2: v1:172.21.15.1:6790/0 mon.c 2024-03-25T18:28:40.865 INFO:teuthology.misc.health.smithi001.stderr: 2024-03-25T18:28:40.866 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:28:40.863+0000 7f203e5f4640 5 monclient: authenticate success, global_id 4308 2024-03-25T18:28:40.866 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:28:40.863+0000 7f203e5f4640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T18:28:40.876 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:28:40.875+0000 7f2036ffd640 1 -- 172.21.15.1:0/1737422684 <== mon.0 v1:172.21.15.1:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f2030001b70 con 0x7f203813c8d0 2024-03-25T18:28:40.876 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:28:40.875+0000 7f2036ffd640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:28:40.876 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:28:40.875+0000 7f2036ffd640 1 -- 172.21.15.1:0/1737422684 <== mon.0 v1:172.21.15.1:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1586965025 0 0) 0x7f2030001fc0 con 0x7f203813c8d0 2024-03-25T18:28:40.876 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:28:40.875+0000 7f203e5f4640 10 monclient: get_monmap_and_config success 2024-03-25T18:28:40.876 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:28:40.875+0000 7f203e5f4640 4 set_mon_vals no callback set 2024-03-25T18:28:40.876 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:28:40.875+0000 7f2036ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:28:40.876 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:28:40.875+0000 7f2036ffd640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-25T18:28:40.876 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:28:40.875+0000 7f2036ffd640 10 monclient: dump: 2024-03-25T18:28:40.876 INFO:teuthology.misc.health.smithi001.stderr:epoch 1 2024-03-25T18:28:40.876 INFO:teuthology.misc.health.smithi001.stderr:fsid c8ea72cf-9e99-4383-9b52-6c23d365fdfc 2024-03-25T18:28:40.876 INFO:teuthology.misc.health.smithi001.stderr:last_changed 2024-03-25T18:25:50.510519+0000 2024-03-25T18:28:40.876 INFO:teuthology.misc.health.smithi001.stderr:created 2024-03-25T18:25:50.510519+0000 2024-03-25T18:28:40.876 INFO:teuthology.misc.health.smithi001.stderr:min_mon_release 19 (squid) 2024-03-25T18:28:40.877 INFO:teuthology.misc.health.smithi001.stderr:election_strategy: 3 2024-03-25T18:28:40.877 INFO:teuthology.misc.health.smithi001.stderr:0: v1:172.21.15.1:6789/0 mon.a 2024-03-25T18:28:40.877 INFO:teuthology.misc.health.smithi001.stderr:1: v1:172.21.15.115:6789/0 mon.b 2024-03-25T18:28:40.877 INFO:teuthology.misc.health.smithi001.stderr:2: v1:172.21.15.1:6790/0 mon.c 2024-03-25T18:28:40.877 INFO:teuthology.misc.health.smithi001.stderr: 2024-03-25T18:28:40.879 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:28:40.876+0000 7f203e5f4640 10 monclient: shutdown 2024-03-25T18:28:40.879 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:28:40.876+0000 7f203e5f4640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:28:40.879 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:28:40.876+0000 7f203e5f4640 1 -- 172.21.15.1:0/1737422684 >> v1:172.21.15.1:6789/0 conn(0x7f203813c8d0 legacy=0x7f203814a930 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:28:40.879 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:28:40.876+0000 7f203e5f4640 1 -- 172.21.15.1:0/1737422684 shutdown_connections 2024-03-25T18:28:40.879 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:28:40.876+0000 7f203e5f4640 1 -- 172.21.15.1:0/1737422684 >> 172.21.15.1:0/1737422684 conn(0x7f20380b4d70 msgr2=0x7f20380b5180 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:28:40.879 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:28:40.876+0000 7f203e5f4640 1 -- 172.21.15.1:0/1737422684 shutdown_connections 2024-03-25T18:28:40.879 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:28:40.876+0000 7f203e5f4640 1 -- 172.21.15.1:0/1737422684 wait complete. 2024-03-25T18:28:40.879 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:28:40.877+0000 7f203e5f4640 10 monclient: build_initial_monmap 2024-03-25T18:28:40.879 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:28:40.877+0000 7f203e5f4640 10 monclient: monmap: 2024-03-25T18:28:40.879 INFO:teuthology.misc.health.smithi001.stderr:epoch 0 2024-03-25T18:28:40.879 INFO:teuthology.misc.health.smithi001.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T18:28:40.880 INFO:teuthology.misc.health.smithi001.stderr:last_changed 0.000000 2024-03-25T18:28:40.880 INFO:teuthology.misc.health.smithi001.stderr:created 0.000000 2024-03-25T18:28:40.880 INFO:teuthology.misc.health.smithi001.stderr:min_mon_release 0 (unknown) 2024-03-25T18:28:40.880 INFO:teuthology.misc.health.smithi001.stderr:election_strategy: 1 2024-03-25T18:28:40.880 INFO:teuthology.misc.health.smithi001.stderr:0: v1:172.21.15.1:6789/0 mon.noname-a 2024-03-25T18:28:40.880 INFO:teuthology.misc.health.smithi001.stderr:1: v1:172.21.15.115:6789/0 mon.noname-b 2024-03-25T18:28:40.880 INFO:teuthology.misc.health.smithi001.stderr:2: v1:172.21.15.1:6790/0 mon.noname-c 2024-03-25T18:28:40.880 INFO:teuthology.misc.health.smithi001.stderr: 2024-03-25T18:28:40.880 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:28:40.877+0000 7f203e5f4640 1 librados: starting msgr at 2024-03-25T18:28:40.880 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:28:40.877+0000 7f203e5f4640 1 librados: starting objecter 2024-03-25T18:28:40.880 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:28:40.877+0000 7f203e5f4640 1 Processor -- start 2024-03-25T18:28:40.880 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:28:40.877+0000 7f203e5f4640 1 -- start start 2024-03-25T18:28:40.880 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:28:40.877+0000 7f203e5f4640 1 librados: setting wanted keys 2024-03-25T18:28:40.880 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:28:40.877+0000 7f203e5f4640 1 librados: calling monclient init 2024-03-25T18:28:40.880 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:28:40.877+0000 7f203e5f4640 10 monclient: init 2024-03-25T18:28:40.881 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:28:40.878+0000 7f203e5f4640 10 monclient: _reopen_session rank -1 2024-03-25T18:28:40.881 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:28:40.878+0000 7f203e5f4640 10 monclient: _add_conns ranks=[0,1,2] 2024-03-25T18:28:40.881 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:28:40.878+0000 7f203e5f4640 10 monclient(hunting): picked mon.noname-a con 0x7f203806f820 addr v1:172.21.15.1:6789/0 2024-03-25T18:28:40.881 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:28:40.878+0000 7f203e5f4640 10 monclient(hunting): picked mon.noname-b con 0x7f203813c8d0 addr v1:172.21.15.115:6789/0 2024-03-25T18:28:40.881 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:28:40.878+0000 7f203e5f4640 10 monclient(hunting): picked mon.noname-c con 0x7f203813f720 addr v1:172.21.15.1:6790/0 2024-03-25T18:28:40.881 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:28:40.878+0000 7f203e5f4640 1 -- --> v1:172.21.15.1:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f20381296e0 con 0x7f203806f820 2024-03-25T18:28:40.881 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:28:40.878+0000 7f203e5f4640 1 -- --> v1:172.21.15.115:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f203814cb90 con 0x7f203813c8d0 2024-03-25T18:28:40.881 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:28:40.878+0000 7f203e5f4640 1 -- --> v1:172.21.15.1:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f203814ce70 con 0x7f203813f720 2024-03-25T18:28:40.881 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:28:40.878+0000 7f203e5f4640 10 monclient(hunting): _renew_subs 2024-03-25T18:28:40.881 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:28:40.878+0000 7f203e5f4640 10 monclient(hunting): authenticate will time out at 7601.348110s 2024-03-25T18:28:40.900 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:28:40.900+0000 7f2037fff640 1 --1- >> v1:172.21.15.1:6789/0 conn(0x7f203806f820 0x7f2038128d40 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.1:6789/0 says I am v1:172.21.15.1:59616/0 (socket says 172.21.15.1:59616) 2024-03-25T18:28:40.900 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:28:40.900+0000 7f2037fff640 1 -- 172.21.15.1:0/760863972 learned_addr learned my addr 172.21.15.1:0/760863972 (peer_addr_for_me v1:172.21.15.1:0/0) 2024-03-25T18:28:40.903 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:28:40.903+0000 7f2034ff9640 10 client.?.objecter ms_handle_connect 0x7f203806f820 2024-03-25T18:28:40.903 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:28:40.903+0000 7f2034ff9640 10 client.?.objecter resend_mon_ops 2024-03-25T18:28:40.925 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:28:40.925+0000 7f2034ff9640 10 client.?.objecter ms_handle_connect 0x7f203813f720 2024-03-25T18:28:40.925 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:28:40.925+0000 7f2034ff9640 10 client.?.objecter resend_mon_ops 2024-03-25T18:28:40.930 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:28:40.930+0000 7f2034ff9640 10 client.?.objecter ms_handle_connect 0x7f203813c8d0 2024-03-25T18:28:40.930 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:28:40.930+0000 7f2034ff9640 10 client.?.objecter resend_mon_ops 2024-03-25T18:28:40.937 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:28:40.936+0000 7f2034ff9640 1 -- 172.21.15.1:0/760863972 <== mon.2 v1:172.21.15.1:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3172631459 0 0) 0x7f203814ce70 con 0x7f203813f720 2024-03-25T18:28:40.937 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:28:40.936+0000 7f2034ff9640 10 monclient(hunting): _negotiate 2024-03-25T18:28:40.937 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:28:40.936+0000 7f2034ff9640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:28:40.937 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:28:40.936+0000 7f2034ff9640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:28:40.937 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:28:40.936+0000 7f2034ff9640 10 monclient(hunting): my global_id is 4310 2024-03-25T18:28:40.938 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:28:40.936+0000 7f2034ff9640 1 -- 172.21.15.1:0/760863972 --> v1:172.21.15.1:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f2024004eb0 con 0x7f203813f720 2024-03-25T18:28:40.938 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:28:40.937+0000 7f2034ff9640 1 -- 172.21.15.1:0/760863972 <== mon.0 v1:172.21.15.1:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3291839692 0 0) 0x7f20381296e0 con 0x7f203806f820 2024-03-25T18:28:40.938 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:28:40.937+0000 7f2034ff9640 10 monclient(hunting): _negotiate 2024-03-25T18:28:40.938 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:28:40.937+0000 7f2034ff9640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:28:40.938 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:28:40.937+0000 7f2034ff9640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:28:40.938 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:28:40.937+0000 7f2034ff9640 10 monclient(hunting): my global_id is 4311 2024-03-25T18:28:40.938 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:28:40.937+0000 7f2034ff9640 1 -- 172.21.15.1:0/760863972 --> v1:172.21.15.1:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f203814ce70 con 0x7f203806f820 2024-03-25T18:28:40.941 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:28:40.940+0000 7f2034ff9640 1 -- 172.21.15.1:0/760863972 <== mon.1 v1:172.21.15.115:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2878642815 0 0) 0x7f203814cb90 con 0x7f203813c8d0 2024-03-25T18:28:40.941 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:28:40.940+0000 7f2034ff9640 10 monclient(hunting): _negotiate 2024-03-25T18:28:40.941 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:28:40.940+0000 7f2034ff9640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:28:40.941 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:28:40.940+0000 7f2034ff9640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:28:40.941 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:28:40.940+0000 7f2034ff9640 10 monclient(hunting): my global_id is 4303 2024-03-25T18:28:40.941 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:28:40.940+0000 7f2034ff9640 1 -- 172.21.15.1:0/760863972 --> v1:172.21.15.115:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f20381296e0 con 0x7f203813c8d0 2024-03-25T18:28:40.948 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:28:40.947+0000 7f2034ff9640 1 -- 172.21.15.1:0/760863972 <== mon.1 v1:172.21.15.115:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2387000936 0 0) 0x7f20381296e0 con 0x7f203813c8d0 2024-03-25T18:28:40.948 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:28:40.947+0000 7f2034ff9640 1 -- 172.21.15.1:0/760863972 --> v1:172.21.15.115:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f203814cb90 con 0x7f203813c8d0 2024-03-25T18:28:40.953 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:28:40.952+0000 7f2034ff9640 1 -- 172.21.15.1:0/760863972 <== mon.1 v1:172.21.15.115:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1586965025 0 0) 0x7f202c001840 con 0x7f203813c8d0 2024-03-25T18:28:40.953 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:28:40.952+0000 7f2034ff9640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:28:40.953 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:28:40.952+0000 7f2034ff9640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-03-25T18:28:40.953 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:28:40.952+0000 7f2034ff9640 10 monclient(hunting): dump: 2024-03-25T18:28:40.953 INFO:teuthology.misc.health.smithi001.stderr:epoch 1 2024-03-25T18:28:40.954 INFO:teuthology.misc.health.smithi001.stderr:fsid c8ea72cf-9e99-4383-9b52-6c23d365fdfc 2024-03-25T18:28:40.954 INFO:teuthology.misc.health.smithi001.stderr:last_changed 2024-03-25T18:25:50.510519+0000 2024-03-25T18:28:40.954 INFO:teuthology.misc.health.smithi001.stderr:created 2024-03-25T18:25:50.510519+0000 2024-03-25T18:28:40.954 INFO:teuthology.misc.health.smithi001.stderr:min_mon_release 19 (squid) 2024-03-25T18:28:40.954 INFO:teuthology.misc.health.smithi001.stderr:election_strategy: 3 2024-03-25T18:28:40.954 INFO:teuthology.misc.health.smithi001.stderr:0: v1:172.21.15.1:6789/0 mon.a 2024-03-25T18:28:40.954 INFO:teuthology.misc.health.smithi001.stderr:1: v1:172.21.15.115:6789/0 mon.b 2024-03-25T18:28:40.954 INFO:teuthology.misc.health.smithi001.stderr:2: v1:172.21.15.1:6790/0 mon.c 2024-03-25T18:28:40.954 INFO:teuthology.misc.health.smithi001.stderr: 2024-03-25T18:28:40.954 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:28:40.952+0000 7f2034ff9640 10 monclient(hunting): _finish_auth 0 2024-03-25T18:28:40.955 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:28:40.953+0000 7f2034ff9640 1 -- 172.21.15.1:0/760863972 <== mon.0 v1:172.21.15.1:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3839783620 0 0) 0x7f203814ce70 con 0x7f203806f820 2024-03-25T18:28:40.955 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:28:40.953+0000 7f2034ff9640 1 -- 172.21.15.1:0/760863972 --> v1:172.21.15.1:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f20381296e0 con 0x7f203806f820 2024-03-25T18:28:40.955 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:28:40.954+0000 7f2034ff9640 1 -- 172.21.15.1:0/760863972 <== mon.0 v1:172.21.15.1:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1586965025 0 0) 0x7f20280018f0 con 0x7f203806f820 2024-03-25T18:28:40.955 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:28:40.954+0000 7f2034ff9640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:28:40.955 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:28:40.954+0000 7f2034ff9640 10 monclient(hunting): got monmap 1 from mon.a (according to old e1) 2024-03-25T18:28:40.955 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:28:40.954+0000 7f2034ff9640 10 monclient(hunting): dump: 2024-03-25T18:28:40.955 INFO:teuthology.misc.health.smithi001.stderr:epoch 1 2024-03-25T18:28:40.955 INFO:teuthology.misc.health.smithi001.stderr:fsid c8ea72cf-9e99-4383-9b52-6c23d365fdfc 2024-03-25T18:28:40.955 INFO:teuthology.misc.health.smithi001.stderr:last_changed 2024-03-25T18:25:50.510519+0000 2024-03-25T18:28:40.955 INFO:teuthology.misc.health.smithi001.stderr:created 2024-03-25T18:25:50.510519+0000 2024-03-25T18:28:40.956 INFO:teuthology.misc.health.smithi001.stderr:min_mon_release 19 (squid) 2024-03-25T18:28:40.956 INFO:teuthology.misc.health.smithi001.stderr:election_strategy: 3 2024-03-25T18:28:40.956 INFO:teuthology.misc.health.smithi001.stderr:0: v1:172.21.15.1:6789/0 mon.a 2024-03-25T18:28:40.956 INFO:teuthology.misc.health.smithi001.stderr:1: v1:172.21.15.115:6789/0 mon.b 2024-03-25T18:28:40.956 INFO:teuthology.misc.health.smithi001.stderr:2: v1:172.21.15.1:6790/0 mon.c 2024-03-25T18:28:40.956 INFO:teuthology.misc.health.smithi001.stderr: 2024-03-25T18:28:40.960 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:28:40.959+0000 7f2034ff9640 1 -- 172.21.15.1:0/760863972 <== mon.2 v1:172.21.15.1:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1983914736 0 0) 0x7f2024004eb0 con 0x7f203813f720 2024-03-25T18:28:40.960 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:28:40.959+0000 7f2034ff9640 1 -- 172.21.15.1:0/760863972 --> v1:172.21.15.1:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f203814ce70 con 0x7f203813f720 2024-03-25T18:28:40.960 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:28:40.959+0000 7f2034ff9640 1 -- 172.21.15.1:0/760863972 <== mon.2 v1:172.21.15.1:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1586965025 0 0) 0x7f2030001f40 con 0x7f203813f720 2024-03-25T18:28:40.960 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:28:40.959+0000 7f2034ff9640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:28:40.960 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:28:40.959+0000 7f2034ff9640 10 monclient(hunting): got monmap 1 from mon.c (according to old e1) 2024-03-25T18:28:40.960 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:28:40.959+0000 7f2034ff9640 10 monclient(hunting): dump: 2024-03-25T18:28:40.961 INFO:teuthology.misc.health.smithi001.stderr:epoch 1 2024-03-25T18:28:40.961 INFO:teuthology.misc.health.smithi001.stderr:fsid c8ea72cf-9e99-4383-9b52-6c23d365fdfc 2024-03-25T18:28:40.961 INFO:teuthology.misc.health.smithi001.stderr:last_changed 2024-03-25T18:25:50.510519+0000 2024-03-25T18:28:40.961 INFO:teuthology.misc.health.smithi001.stderr:created 2024-03-25T18:25:50.510519+0000 2024-03-25T18:28:40.961 INFO:teuthology.misc.health.smithi001.stderr:min_mon_release 19 (squid) 2024-03-25T18:28:40.961 INFO:teuthology.misc.health.smithi001.stderr:election_strategy: 3 2024-03-25T18:28:40.961 INFO:teuthology.misc.health.smithi001.stderr:0: v1:172.21.15.1:6789/0 mon.a 2024-03-25T18:28:40.961 INFO:teuthology.misc.health.smithi001.stderr:1: v1:172.21.15.115:6789/0 mon.b 2024-03-25T18:28:40.961 INFO:teuthology.misc.health.smithi001.stderr:2: v1:172.21.15.1:6790/0 mon.c 2024-03-25T18:28:40.961 INFO:teuthology.misc.health.smithi001.stderr: 2024-03-25T18:28:40.967 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:28:40.966+0000 7f2034ff9640 1 -- 172.21.15.1:0/760863972 <== mon.2 v1:172.21.15.1:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 3513580892 0 0) 0x7f203814ce70 con 0x7f203813f720 2024-03-25T18:28:40.967 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:28:40.966+0000 7f2034ff9640 1 -- 172.21.15.1:0/760863972 >> v1:172.21.15.115:6789/0 conn(0x7f203813c8d0 legacy=0x7f203814d570 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:28:40.967 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:28:40.966+0000 7f2034ff9640 1 -- 172.21.15.1:0/760863972 >> v1:172.21.15.1:6789/0 conn(0x7f203806f820 legacy=0x7f2038128d40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:28:40.967 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:28:40.966+0000 7f2034ff9640 10 monclient: _finish_hunting 0 2024-03-25T18:28:40.967 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:28:40.966+0000 7f2034ff9640 1 monclient: found mon.c 2024-03-25T18:28:40.967 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:28:40.966+0000 7f2034ff9640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:28:40.968 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:28:40.966+0000 7f2034ff9640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.1:6790/0 2024-03-25T18:28:40.968 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:28:40.966+0000 7f2034ff9640 1 -- 172.21.15.1:0/760863972 --> v1:172.21.15.1:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f203814d150 con 0x7f203813f720 2024-03-25T18:28:40.968 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:28:40.966+0000 7f2034ff9640 10 monclient: _finish_auth 0 2024-03-25T18:28:40.968 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:28:40.966+0000 7f2034ff9640 10 monclient: _check_auth_tickets 2024-03-25T18:28:40.968 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:28:40.966+0000 7f2034ff9640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:28:40.968 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:28:40.966+0000 7f203e5f4640 5 monclient: authenticate success, global_id 4310 2024-03-25T18:28:40.968 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:28:40.966+0000 7f203e5f4640 10 monclient: _renew_subs 2024-03-25T18:28:40.968 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:28:40.966+0000 7f203e5f4640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.1:6790/0 2024-03-25T18:28:40.968 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:28:40.966+0000 7f203e5f4640 1 -- 172.21.15.1:0/760863972 --> v1:172.21.15.1:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2038134570 con 0x7f203813f720 2024-03-25T18:28:40.968 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:28:40.966+0000 7f203e5f4640 10 client.4310.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T18:28:40.968 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:28:40.966+0000 7f203e5f4640 10 monclient: _renew_subs 2024-03-25T18:28:40.968 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:28:40.966+0000 7f203e5f4640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.1:6790/0 2024-03-25T18:28:40.968 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:28:40.966+0000 7f203e5f4640 1 -- 172.21.15.1:0/760863972 --> v1:172.21.15.1:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f203814de60 con 0x7f203813f720 2024-03-25T18:28:40.968 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:28:40.966+0000 7f203e5f4640 1 librados: init done 2024-03-25T18:28:40.969 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:28:40.967+0000 7f203e5f4640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T18:28:40.969 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:28:40.967+0000 7f203e5f4640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:28:40.969 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:28:40.967+0000 7f203e5f4640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.1:6790/0 2024-03-25T18:28:40.969 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:28:40.967+0000 7f203e5f4640 1 -- 172.21.15.1:0/760863972 --> v1:172.21.15.1:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f20380730e0 con 0x7f203813f720 2024-03-25T18:28:41.298 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:28:41.297+0000 7f2034ff9640 1 -- 172.21.15.1:0/760863972 <== mon.2 v1:172.21.15.1:6790/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f2030003ce0 con 0x7f203813f720 2024-03-25T18:28:41.298 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:28:41.297+0000 7f2034ff9640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:28:41.298 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:28:41.297+0000 7f2034ff9640 1 -- 172.21.15.1:0/760863972 <== mon.2 v1:172.21.15.1:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1586965025 0 0) 0x7f2030003ff0 con 0x7f203813f720 2024-03-25T18:28:41.298 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:28:41.297+0000 7f2034ff9640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:28:41.298 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:28:41.297+0000 7f2034ff9640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-25T18:28:41.298 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:28:41.297+0000 7f2034ff9640 10 monclient: dump: 2024-03-25T18:28:41.298 INFO:teuthology.misc.health.smithi001.stderr:epoch 1 2024-03-25T18:28:41.298 INFO:teuthology.misc.health.smithi001.stderr:fsid c8ea72cf-9e99-4383-9b52-6c23d365fdfc 2024-03-25T18:28:41.298 INFO:teuthology.misc.health.smithi001.stderr:last_changed 2024-03-25T18:25:50.510519+0000 2024-03-25T18:28:41.299 INFO:teuthology.misc.health.smithi001.stderr:created 2024-03-25T18:25:50.510519+0000 2024-03-25T18:28:41.299 INFO:teuthology.misc.health.smithi001.stderr:min_mon_release 19 (squid) 2024-03-25T18:28:41.299 INFO:teuthology.misc.health.smithi001.stderr:election_strategy: 3 2024-03-25T18:28:41.299 INFO:teuthology.misc.health.smithi001.stderr:0: v1:172.21.15.1:6789/0 mon.a 2024-03-25T18:28:41.299 INFO:teuthology.misc.health.smithi001.stderr:1: v1:172.21.15.115:6789/0 mon.b 2024-03-25T18:28:41.299 INFO:teuthology.misc.health.smithi001.stderr:2: v1:172.21.15.1:6790/0 mon.c 2024-03-25T18:28:41.299 INFO:teuthology.misc.health.smithi001.stderr: 2024-03-25T18:28:41.299 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:28:41.297+0000 7f20367fc640 4 set_mon_vals no callback set 2024-03-25T18:28:41.300 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:28:41.298+0000 7f2034ff9640 1 -- 172.21.15.1:0/760863972 <== mon.2 v1:172.21.15.1:6790/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 21787975 0 0) 0x7f2030019fc0 con 0x7f203813f720 2024-03-25T18:28:41.300 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:28:41.299+0000 7f2037fff640 10 monclient: get_auth_request con 0x7f202406e110 auth_method 0 2024-03-25T18:28:41.300 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:28:41.299+0000 7f2034ff9640 10 client.4310.objecter ms_handle_connect 0x7f202406e110 2024-03-25T18:28:41.304 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:28:41.304+0000 7f2034ff9640 1 -- 172.21.15.1:0/760863972 <== mon.2 v1:172.21.15.1:6790/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 996254230 0 0) 0x7f2030053a30 con 0x7f203813f720 2024-03-25T18:28:41.304 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:28:41.304+0000 7f2034ff9640 10 client.4310.objecter ms_dispatch 0x7f2038000b90 osd_map(10..10 src has 1..10) v4 2024-03-25T18:28:41.305 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:28:41.304+0000 7f2034ff9640 3 client.4310.objecter handle_osd_map got epochs [10,10] > 0 2024-03-25T18:28:41.305 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:28:41.304+0000 7f2034ff9640 3 client.4310.objecter handle_osd_map decoding full epoch 10 2024-03-25T18:28:41.305 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:28:41.304+0000 7f2034ff9640 20 client.4310.objecter dump_active .. 0 homeless 2024-03-25T18:28:41.305 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:28:41.304+0000 7f2034ff9640 1 -- 172.21.15.1:0/760863972 <== mon.2 v1:172.21.15.1:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f203004f540 con 0x7f203813f720 2024-03-25T18:28:41.305 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:28:41.304+0000 7f2034ff9640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:28:41.305 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:28:41.304+0000 7f2034ff9640 10 monclient: _finish_command 1 = system:0 2024-03-25T18:28:41.561 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:28:41.560+0000 7f203e5f4640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:28:41.561 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:28:41.560+0000 7f203e5f4640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:28:41.561 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:28:41.560+0000 7f203e5f4640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.1:6790/0 2024-03-25T18:28:41.561 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:28:41.560+0000 7f203e5f4640 1 -- 172.21.15.1:0/760863972 --> v1:172.21.15.1:6790/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f2038127b00 con 0x7f203813f720 2024-03-25T18:28:41.602 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:28:41.601+0000 7f2034ff9640 1 -- 172.21.15.1:0/760863972 <== mon.2 v1:172.21.15.1:6790/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 4092213348) 0x7f2038127b00 con 0x7f203813f720 2024-03-25T18:28:41.602 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:28:41.601+0000 7f2034ff9640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:28:41.602 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:28:41.601+0000 7f2034ff9640 10 monclient: _finish_command 2 = system:0 2024-03-25T18:28:41.602 INFO:teuthology.misc.health.smithi001.stdout: 2024-03-25T18:28:41.603 INFO:teuthology.misc.health.smithi001.stdout:{"epoch":10,"fsid":"c8ea72cf-9e99-4383-9b52-6c23d365fdfc","created":"2024-03-25T18:26:38.856638+0000","modified":"2024-03-25T18:26:44.224208+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T18:26:44.224208+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"b9f34922-3a84-49be-8426-807c1eceaefc","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"42bf306f-55e3-4634-9874-8b4b73ac6644","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"292cb23a-18a9-4207-a7cc-4d5727feaec0","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"45e0e9ab-898d-4673-9590-b4117bc5e84a","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"831a5a09-73ab-4a61-9872-088e76bdbc66","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"88cd903d-3b55-4840-9396-6e3cfe005282","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"89db7355-bc9d-46f4-8ab0-e8d9e744e026","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"cac47bee-0914-4f08-896f-348838882866","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T18:28:41.605 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:28:41.604+0000 7f203e5f4640 10 librados: watch_flush enter 2024-03-25T18:28:41.605 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:28:41.604+0000 7f203e5f4640 10 librados: watch_flush exit 2024-03-25T18:28:41.606 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:28:41.604+0000 7f203e5f4640 20 client.4310.objecter shutdown clearing up homeless session... 2024-03-25T18:28:41.606 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:28:41.604+0000 7f203e5f4640 10 client.4310.objecter successfully canceled tick 2024-03-25T18:28:41.606 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:28:41.604+0000 7f203e5f4640 1 -- 172.21.15.1:0/760863972 >> v1:172.21.15.115:6800/4237379176 conn(0x7f202406e110 legacy=0x7f20240705d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:28:41.606 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:28:41.604+0000 7f203e5f4640 10 monclient: shutdown 2024-03-25T18:28:41.606 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:28:41.604+0000 7f203e5f4640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:28:41.606 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:28:41.604+0000 7f203e5f4640 1 -- 172.21.15.1:0/760863972 >> v1:172.21.15.1:6790/0 conn(0x7f203813f720 legacy=0x7f20381e2450 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:28:41.606 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:28:41.604+0000 7f203e5f4640 1 -- 172.21.15.1:0/760863972 shutdown_connections 2024-03-25T18:28:41.606 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:28:41.604+0000 7f203e5f4640 1 -- 172.21.15.1:0/760863972 >> 172.21.15.1:0/760863972 conn(0x7f20380b4d70 msgr2=0x7f20380b0000 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:28:41.606 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:28:41.604+0000 7f203e5f4640 1 -- 172.21.15.1:0/760863972 shutdown_connections 2024-03-25T18:28:41.606 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:28:41.604+0000 7f203e5f4640 1 -- 172.21.15.1:0/760863972 wait complete. 2024-03-25T18:28:41.606 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:28:41.604+0000 7f203e5f4640 1 librados: shutdown 2024-03-25T18:28:41.616 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T18:28:48.425 DEBUG:teuthology.orchestra.run.smithi001:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T18:28:48.543 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:28:48.542+0000 7f802c043640 10 monclient: get_monmap_and_config 2024-03-25T18:28:48.543 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:28:48.542+0000 7f802c043640 10 monclient: build_initial_monmap 2024-03-25T18:28:48.543 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:28:48.542+0000 7f802c043640 10 monclient: monmap: 2024-03-25T18:28:48.543 INFO:teuthology.misc.health.smithi001.stderr:epoch 0 2024-03-25T18:28:48.543 INFO:teuthology.misc.health.smithi001.stderr:fsid c8ea72cf-9e99-4383-9b52-6c23d365fdfc 2024-03-25T18:28:48.544 INFO:teuthology.misc.health.smithi001.stderr:last_changed 2024-03-25T18:28:48.543133+0000 2024-03-25T18:28:48.544 INFO:teuthology.misc.health.smithi001.stderr:created 2024-03-25T18:28:48.543133+0000 2024-03-25T18:28:48.544 INFO:teuthology.misc.health.smithi001.stderr:min_mon_release 0 (unknown) 2024-03-25T18:28:48.544 INFO:teuthology.misc.health.smithi001.stderr:election_strategy: 3 2024-03-25T18:28:48.544 INFO:teuthology.misc.health.smithi001.stderr:0: v1:172.21.15.1:6789/0 mon.noname-a 2024-03-25T18:28:48.544 INFO:teuthology.misc.health.smithi001.stderr:1: v1:172.21.15.115:6789/0 mon.noname-c 2024-03-25T18:28:48.544 INFO:teuthology.misc.health.smithi001.stderr:2: v1:172.21.15.1:6790/0 mon.noname-b 2024-03-25T18:28:48.544 INFO:teuthology.misc.health.smithi001.stderr: 2024-03-25T18:28:48.544 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:28:48.543+0000 7f802c043640 1 Processor -- start 2024-03-25T18:28:48.545 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:28:48.543+0000 7f802c043640 1 -- start start 2024-03-25T18:28:48.545 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:28:48.543+0000 7f802c043640 10 monclient: init 2024-03-25T18:28:48.545 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:28:48.544+0000 7f802c043640 10 monclient: _reopen_session rank -1 2024-03-25T18:28:48.546 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:28:48.544+0000 7f802c043640 10 monclient: _add_conns ranks=[2,1,0] 2024-03-25T18:28:48.546 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:28:48.544+0000 7f802c043640 10 monclient(hunting): picked mon.noname-b con 0x7f80240b4ff0 addr v1:172.21.15.1:6790/0 2024-03-25T18:28:48.546 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:28:48.544+0000 7f802c043640 10 monclient(hunting): picked mon.noname-c con 0x7f80240b4670 addr v1:172.21.15.115:6789/0 2024-03-25T18:28:48.546 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:28:48.544+0000 7f802c043640 10 monclient(hunting): picked mon.noname-a con 0x7f802413c480 addr v1:172.21.15.1:6789/0 2024-03-25T18:28:48.546 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:28:48.544+0000 7f802c043640 1 -- --> v1:172.21.15.1:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f80240ab0e0 con 0x7f802413c480 2024-03-25T18:28:48.546 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:28:48.544+0000 7f802c043640 1 -- --> v1:172.21.15.115:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f80240ab3c0 con 0x7f80240b4670 2024-03-25T18:28:48.546 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:28:48.544+0000 7f802c043640 1 -- --> v1:172.21.15.1:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f80240a9f30 con 0x7f80240b4ff0 2024-03-25T18:28:48.546 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:28:48.544+0000 7f802c043640 10 monclient(hunting): _renew_subs 2024-03-25T18:28:48.546 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:28:48.544+0000 7f802c043640 10 monclient(hunting): authenticate will time out at 7609.014224s 2024-03-25T18:28:48.553 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:28:48.553+0000 7f80295de640 1 --1- >> v1:172.21.15.115:6789/0 conn(0x7f80240b4670 0x7f80241425c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.115:6789/0 says I am v1:172.21.15.1:36200/0 (socket says 172.21.15.1:36200) 2024-03-25T18:28:48.554 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:28:48.553+0000 7f80295de640 1 -- 172.21.15.1:0/1886441746 learned_addr learned my addr 172.21.15.1:0/1886441746 (peer_addr_for_me v1:172.21.15.1:0/0) 2024-03-25T18:28:48.563 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:28:48.563+0000 7f8028ddd640 1 -- 172.21.15.1:0/1886441746 <== mon.1 v1:172.21.15.115:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 890189744 0 0) 0x7f80240ab3c0 con 0x7f80240b4670 2024-03-25T18:28:48.563 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:28:48.563+0000 7f8028ddd640 10 monclient(hunting): _negotiate 2024-03-25T18:28:48.564 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:28:48.563+0000 7f8028ddd640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:28:48.564 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:28:48.563+0000 7f8028ddd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:28:48.564 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:28:48.563+0000 7f8028ddd640 10 monclient(hunting): my global_id is 4306 2024-03-25T18:28:48.564 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:28:48.563+0000 7f8028ddd640 1 -- 172.21.15.1:0/1886441746 --> v1:172.21.15.115:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f8010003520 con 0x7f80240b4670 2024-03-25T18:28:48.569 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:28:48.568+0000 7f8028ddd640 1 -- 172.21.15.1:0/1886441746 <== mon.1 v1:172.21.15.115:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 4290850482 0 0) 0x7f8010003520 con 0x7f80240b4670 2024-03-25T18:28:48.569 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:28:48.568+0000 7f8028ddd640 1 -- 172.21.15.1:0/1886441746 >> v1:172.21.15.1:6790/0 conn(0x7f80240b4ff0 legacy=0x7f802413bd70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:28:48.569 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:28:48.568+0000 7f8028ddd640 1 -- 172.21.15.1:0/1886441746 >> v1:172.21.15.1:6789/0 conn(0x7f802413c480 legacy=0x7f802414dc00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-25T18:28:48.569 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:28:48.568+0000 7f8028ddd640 10 monclient: _finish_hunting 0 2024-03-25T18:28:48.569 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:28:48.568+0000 7f8028ddd640 1 monclient: found mon.noname-c 2024-03-25T18:28:48.569 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:28:48.568+0000 7f8028ddd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:28:48.569 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:28:48.568+0000 7f8028ddd640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.115:6789/0 2024-03-25T18:28:48.569 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:28:48.568+0000 7f8028ddd640 1 -- 172.21.15.1:0/1886441746 --> v1:172.21.15.115:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f80240a9b70 con 0x7f80240b4670 2024-03-25T18:28:48.569 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:28:48.568+0000 7f8028ddd640 10 monclient: _finish_auth 0 2024-03-25T18:28:48.569 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:28:48.568+0000 7f8028ddd640 10 monclient: _check_auth_tickets 2024-03-25T18:28:48.569 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:28:48.568+0000 7f8028ddd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:28:48.569 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:28:48.568+0000 7f8028ddd640 1 -- 172.21.15.1:0/1886441746 <== mon.1 v1:172.21.15.115:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1586965025 0 0) 0x7f8014001690 con 0x7f80240b4670 2024-03-25T18:28:48.570 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:28:48.568+0000 7f8028ddd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:28:48.570 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:28:48.568+0000 7f8028ddd640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-03-25T18:28:48.570 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:28:48.568+0000 7f8028ddd640 10 monclient: dump: 2024-03-25T18:28:48.570 INFO:teuthology.misc.health.smithi001.stderr:epoch 1 2024-03-25T18:28:48.570 INFO:teuthology.misc.health.smithi001.stderr:fsid c8ea72cf-9e99-4383-9b52-6c23d365fdfc 2024-03-25T18:28:48.570 INFO:teuthology.misc.health.smithi001.stderr:last_changed 2024-03-25T18:25:50.510519+0000 2024-03-25T18:28:48.570 INFO:teuthology.misc.health.smithi001.stderr:created 2024-03-25T18:25:50.510519+0000 2024-03-25T18:28:48.570 INFO:teuthology.misc.health.smithi001.stderr:min_mon_release 19 (squid) 2024-03-25T18:28:48.570 INFO:teuthology.misc.health.smithi001.stderr:election_strategy: 3 2024-03-25T18:28:48.570 INFO:teuthology.misc.health.smithi001.stderr:0: v1:172.21.15.1:6789/0 mon.a 2024-03-25T18:28:48.570 INFO:teuthology.misc.health.smithi001.stderr:1: v1:172.21.15.115:6789/0 mon.b 2024-03-25T18:28:48.570 INFO:teuthology.misc.health.smithi001.stderr:2: v1:172.21.15.1:6790/0 mon.c 2024-03-25T18:28:48.570 INFO:teuthology.misc.health.smithi001.stderr: 2024-03-25T18:28:48.570 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:28:48.568+0000 7f802c043640 5 monclient: authenticate success, global_id 4306 2024-03-25T18:28:48.571 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:28:48.568+0000 7f802c043640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T18:28:48.596 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:28:48.596+0000 7f8028ddd640 1 -- 172.21.15.1:0/1886441746 <== mon.1 v1:172.21.15.115:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f8014001860 con 0x7f80240b4670 2024-03-25T18:28:48.597 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:28:48.596+0000 7f8028ddd640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:28:48.597 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:28:48.596+0000 7f8028ddd640 1 -- 172.21.15.1:0/1886441746 <== mon.1 v1:172.21.15.115:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1586965025 0 0) 0x7f8014001cb0 con 0x7f80240b4670 2024-03-25T18:28:48.597 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:28:48.596+0000 7f8028ddd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:28:48.597 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:28:48.596+0000 7f8028ddd640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-25T18:28:48.597 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:28:48.596+0000 7f8028ddd640 10 monclient: dump: 2024-03-25T18:28:48.597 INFO:teuthology.misc.health.smithi001.stderr:epoch 1 2024-03-25T18:28:48.597 INFO:teuthology.misc.health.smithi001.stderr:fsid c8ea72cf-9e99-4383-9b52-6c23d365fdfc 2024-03-25T18:28:48.597 INFO:teuthology.misc.health.smithi001.stderr:last_changed 2024-03-25T18:25:50.510519+0000 2024-03-25T18:28:48.597 INFO:teuthology.misc.health.smithi001.stderr:created 2024-03-25T18:25:50.510519+0000 2024-03-25T18:28:48.597 INFO:teuthology.misc.health.smithi001.stderr:min_mon_release 19 (squid) 2024-03-25T18:28:48.597 INFO:teuthology.misc.health.smithi001.stderr:election_strategy: 3 2024-03-25T18:28:48.598 INFO:teuthology.misc.health.smithi001.stderr:0: v1:172.21.15.1:6789/0 mon.a 2024-03-25T18:28:48.598 INFO:teuthology.misc.health.smithi001.stderr:1: v1:172.21.15.115:6789/0 mon.b 2024-03-25T18:28:48.598 INFO:teuthology.misc.health.smithi001.stderr:2: v1:172.21.15.1:6790/0 mon.c 2024-03-25T18:28:48.598 INFO:teuthology.misc.health.smithi001.stderr: 2024-03-25T18:28:48.598 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:28:48.596+0000 7f802c043640 10 monclient: get_monmap_and_config success 2024-03-25T18:28:48.598 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:28:48.596+0000 7f802c043640 4 set_mon_vals no callback set 2024-03-25T18:28:48.600 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:28:48.596+0000 7f802c043640 10 monclient: shutdown 2024-03-25T18:28:48.600 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:28:48.596+0000 7f802c043640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:28:48.600 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:28:48.596+0000 7f802c043640 1 -- 172.21.15.1:0/1886441746 >> v1:172.21.15.115:6789/0 conn(0x7f80240b4670 legacy=0x7f80241425c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:28:48.600 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:28:48.597+0000 7f802c043640 1 -- 172.21.15.1:0/1886441746 shutdown_connections 2024-03-25T18:28:48.600 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:28:48.597+0000 7f802c043640 1 -- 172.21.15.1:0/1886441746 >> 172.21.15.1:0/1886441746 conn(0x7f80240617a0 msgr2=0x7f8024061bb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:28:48.600 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:28:48.597+0000 7f802c043640 1 -- 172.21.15.1:0/1886441746 shutdown_connections 2024-03-25T18:28:48.601 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:28:48.597+0000 7f802c043640 1 -- 172.21.15.1:0/1886441746 wait complete. 2024-03-25T18:28:48.601 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:28:48.597+0000 7f802c043640 10 monclient: build_initial_monmap 2024-03-25T18:28:48.601 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:28:48.597+0000 7f802c043640 10 monclient: monmap: 2024-03-25T18:28:48.601 INFO:teuthology.misc.health.smithi001.stderr:epoch 0 2024-03-25T18:28:48.601 INFO:teuthology.misc.health.smithi001.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T18:28:48.601 INFO:teuthology.misc.health.smithi001.stderr:last_changed 0.000000 2024-03-25T18:28:48.601 INFO:teuthology.misc.health.smithi001.stderr:created 0.000000 2024-03-25T18:28:48.601 INFO:teuthology.misc.health.smithi001.stderr:min_mon_release 0 (unknown) 2024-03-25T18:28:48.601 INFO:teuthology.misc.health.smithi001.stderr:election_strategy: 1 2024-03-25T18:28:48.601 INFO:teuthology.misc.health.smithi001.stderr:0: v1:172.21.15.1:6789/0 mon.noname-a 2024-03-25T18:28:48.601 INFO:teuthology.misc.health.smithi001.stderr:1: v1:172.21.15.115:6789/0 mon.noname-b 2024-03-25T18:28:48.602 INFO:teuthology.misc.health.smithi001.stderr:2: v1:172.21.15.1:6790/0 mon.noname-c 2024-03-25T18:28:48.602 INFO:teuthology.misc.health.smithi001.stderr: 2024-03-25T18:28:48.602 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:28:48.598+0000 7f802c043640 1 librados: starting msgr at 2024-03-25T18:28:48.602 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:28:48.598+0000 7f802c043640 1 librados: starting objecter 2024-03-25T18:28:48.602 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:28:48.598+0000 7f802c043640 1 Processor -- start 2024-03-25T18:28:48.602 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:28:48.598+0000 7f802c043640 1 -- start start 2024-03-25T18:28:48.602 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:28:48.598+0000 7f802c043640 1 librados: setting wanted keys 2024-03-25T18:28:48.602 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:28:48.598+0000 7f802c043640 1 librados: calling monclient init 2024-03-25T18:28:48.602 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:28:48.598+0000 7f802c043640 10 monclient: init 2024-03-25T18:28:48.602 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:28:48.598+0000 7f802c043640 10 monclient: _reopen_session rank -1 2024-03-25T18:28:48.602 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:28:48.598+0000 7f802c043640 10 monclient: _add_conns ranks=[1,2,0] 2024-03-25T18:28:48.602 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:28:48.598+0000 7f802c043640 10 monclient(hunting): picked mon.noname-b con 0x7f80240b4670 addr v1:172.21.15.115:6789/0 2024-03-25T18:28:48.603 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:28:48.598+0000 7f802c043640 10 monclient(hunting): picked mon.noname-c con 0x7f80240b4ff0 addr v1:172.21.15.1:6790/0 2024-03-25T18:28:48.603 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:28:48.599+0000 7f802c043640 10 monclient(hunting): picked mon.noname-a con 0x7f802413c480 addr v1:172.21.15.1:6789/0 2024-03-25T18:28:48.603 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:28:48.599+0000 7f802c043640 1 -- --> v1:172.21.15.1:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f802413f720 con 0x7f802413c480 2024-03-25T18:28:48.603 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:28:48.599+0000 7f802c043640 1 -- --> v1:172.21.15.115:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8024144990 con 0x7f80240b4670 2024-03-25T18:28:48.603 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:28:48.599+0000 7f802c043640 1 -- --> v1:172.21.15.1:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8024144c70 con 0x7f80240b4ff0 2024-03-25T18:28:48.603 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:28:48.599+0000 7f802c043640 10 monclient(hunting): _renew_subs 2024-03-25T18:28:48.603 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:28:48.599+0000 7f802c043640 10 monclient(hunting): authenticate will time out at 7609.068775s 2024-03-25T18:28:48.609 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:28:48.609+0000 7f8029ddf640 1 --1- >> v1:172.21.15.115:6789/0 conn(0x7f80240b4670 0x7f802413c910 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.115:6789/0 says I am v1:172.21.15.1:36248/0 (socket says 172.21.15.1:36248) 2024-03-25T18:28:48.609 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:28:48.609+0000 7f8029ddf640 1 -- 172.21.15.1:0/1669196940 learned_addr learned my addr 172.21.15.1:0/1669196940 (peer_addr_for_me v1:172.21.15.1:0/0) 2024-03-25T18:28:48.611 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:28:48.610+0000 7f801a7fc640 10 client.?.objecter ms_handle_connect 0x7f80240b4670 2024-03-25T18:28:48.611 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:28:48.610+0000 7f801a7fc640 10 client.?.objecter resend_mon_ops 2024-03-25T18:28:48.619 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:28:48.619+0000 7f801a7fc640 10 client.?.objecter ms_handle_connect 0x7f80240b4ff0 2024-03-25T18:28:48.619 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:28:48.619+0000 7f801a7fc640 10 client.?.objecter resend_mon_ops 2024-03-25T18:28:48.622 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:28:48.621+0000 7f801a7fc640 10 client.?.objecter ms_handle_connect 0x7f802413c480 2024-03-25T18:28:48.622 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:28:48.621+0000 7f801a7fc640 10 client.?.objecter resend_mon_ops 2024-03-25T18:28:48.630 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:28:48.627+0000 7f801a7fc640 1 -- 172.21.15.1:0/1669196940 <== mon.1 v1:172.21.15.115:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2937841280 0 0) 0x7f8024144990 con 0x7f80240b4670 2024-03-25T18:28:48.630 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:28:48.627+0000 7f801a7fc640 10 monclient(hunting): _negotiate 2024-03-25T18:28:48.630 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:28:48.627+0000 7f801a7fc640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:28:48.630 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:28:48.627+0000 7f801a7fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:28:48.630 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:28:48.627+0000 7f801a7fc640 10 monclient(hunting): my global_id is 4309 2024-03-25T18:28:48.630 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:28:48.627+0000 7f801a7fc640 1 -- 172.21.15.1:0/1669196940 --> v1:172.21.15.115:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f8008004eb0 con 0x7f80240b4670 2024-03-25T18:28:48.631 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:28:48.631+0000 7f801a7fc640 1 -- 172.21.15.1:0/1669196940 <== mon.0 v1:172.21.15.1:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3590684111 0 0) 0x7f802413f720 con 0x7f802413c480 2024-03-25T18:28:48.632 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:28:48.631+0000 7f801a7fc640 10 monclient(hunting): _negotiate 2024-03-25T18:28:48.632 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:28:48.631+0000 7f801a7fc640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:28:48.632 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:28:48.631+0000 7f801a7fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:28:48.632 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:28:48.631+0000 7f801a7fc640 10 monclient(hunting): my global_id is 4314 2024-03-25T18:28:48.632 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:28:48.631+0000 7f801a7fc640 1 -- 172.21.15.1:0/1669196940 --> v1:172.21.15.1:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f8024144990 con 0x7f802413c480 2024-03-25T18:28:48.635 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:28:48.635+0000 7f801a7fc640 1 -- 172.21.15.1:0/1669196940 <== mon.2 v1:172.21.15.1:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3448080923 0 0) 0x7f8024144c70 con 0x7f80240b4ff0 2024-03-25T18:28:48.635 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:28:48.635+0000 7f801a7fc640 10 monclient(hunting): _negotiate 2024-03-25T18:28:48.635 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:28:48.635+0000 7f801a7fc640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:28:48.636 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:28:48.635+0000 7f801a7fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:28:48.636 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:28:48.635+0000 7f801a7fc640 10 monclient(hunting): my global_id is 4316 2024-03-25T18:28:48.636 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:28:48.635+0000 7f801a7fc640 1 -- 172.21.15.1:0/1669196940 --> v1:172.21.15.1:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f802413f720 con 0x7f80240b4ff0 2024-03-25T18:28:48.722 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:28:48.721+0000 7f801a7fc640 1 -- 172.21.15.1:0/1669196940 <== mon.0 v1:172.21.15.1:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2733526770 0 0) 0x7f8024144990 con 0x7f802413c480 2024-03-25T18:28:48.722 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:28:48.721+0000 7f801a7fc640 1 -- 172.21.15.1:0/1669196940 --> v1:172.21.15.1:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f8024144c70 con 0x7f802413c480 2024-03-25T18:28:48.722 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:28:48.721+0000 7f801a7fc640 1 -- 172.21.15.1:0/1669196940 <== mon.0 v1:172.21.15.1:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1586965025 0 0) 0x7f801c001a70 con 0x7f802413c480 2024-03-25T18:28:48.722 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:28:48.721+0000 7f801a7fc640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:28:48.722 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:28:48.721+0000 7f801a7fc640 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2024-03-25T18:28:48.722 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:28:48.721+0000 7f801a7fc640 10 monclient(hunting): dump: 2024-03-25T18:28:48.722 INFO:teuthology.misc.health.smithi001.stderr:epoch 1 2024-03-25T18:28:48.722 INFO:teuthology.misc.health.smithi001.stderr:fsid c8ea72cf-9e99-4383-9b52-6c23d365fdfc 2024-03-25T18:28:48.722 INFO:teuthology.misc.health.smithi001.stderr:last_changed 2024-03-25T18:25:50.510519+0000 2024-03-25T18:28:48.723 INFO:teuthology.misc.health.smithi001.stderr:created 2024-03-25T18:25:50.510519+0000 2024-03-25T18:28:48.723 INFO:teuthology.misc.health.smithi001.stderr:min_mon_release 19 (squid) 2024-03-25T18:28:48.723 INFO:teuthology.misc.health.smithi001.stderr:election_strategy: 3 2024-03-25T18:28:48.723 INFO:teuthology.misc.health.smithi001.stderr:0: v1:172.21.15.1:6789/0 mon.a 2024-03-25T18:28:48.723 INFO:teuthology.misc.health.smithi001.stderr:1: v1:172.21.15.115:6789/0 mon.b 2024-03-25T18:28:48.723 INFO:teuthology.misc.health.smithi001.stderr:2: v1:172.21.15.1:6790/0 mon.c 2024-03-25T18:28:48.723 INFO:teuthology.misc.health.smithi001.stderr: 2024-03-25T18:28:48.723 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:28:48.721+0000 7f801a7fc640 10 monclient(hunting): _finish_auth 0 2024-03-25T18:28:48.740 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:28:48.739+0000 7f801a7fc640 1 -- 172.21.15.1:0/1669196940 <== mon.0 v1:172.21.15.1:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 3148048274 0 0) 0x7f8024144c70 con 0x7f802413c480 2024-03-25T18:28:48.740 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:28:48.739+0000 7f801a7fc640 1 -- 172.21.15.1:0/1669196940 >> v1:172.21.15.1:6790/0 conn(0x7f80240b4ff0 legacy=0x7f8024145090 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:28:48.740 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:28:48.739+0000 7f801a7fc640 1 -- 172.21.15.1:0/1669196940 >> v1:172.21.15.115:6789/0 conn(0x7f80240b4670 legacy=0x7f802413c910 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:28:48.740 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:28:48.739+0000 7f801a7fc640 10 monclient: _finish_hunting 0 2024-03-25T18:28:48.740 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:28:48.739+0000 7f801a7fc640 1 monclient: found mon.a 2024-03-25T18:28:48.741 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:28:48.739+0000 7f801a7fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:28:48.741 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:28:48.739+0000 7f801a7fc640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.1:6789/0 2024-03-25T18:28:48.741 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:28:48.739+0000 7f801a7fc640 1 -- 172.21.15.1:0/1669196940 --> v1:172.21.15.1:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f802413ed80 con 0x7f802413c480 2024-03-25T18:28:48.741 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:28:48.739+0000 7f801a7fc640 10 monclient: _finish_auth 0 2024-03-25T18:28:48.741 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:28:48.739+0000 7f801a7fc640 10 monclient: _check_auth_tickets 2024-03-25T18:28:48.741 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:28:48.739+0000 7f801a7fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:28:48.741 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:28:48.739+0000 7f802c043640 5 monclient: authenticate success, global_id 4314 2024-03-25T18:28:48.741 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:28:48.739+0000 7f802c043640 10 monclient: _renew_subs 2024-03-25T18:28:48.741 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:28:48.739+0000 7f802c043640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.1:6789/0 2024-03-25T18:28:48.741 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:28:48.739+0000 7f802c043640 1 -- 172.21.15.1:0/1669196940 --> v1:172.21.15.1:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8024145eb0 con 0x7f802413c480 2024-03-25T18:28:48.741 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:28:48.739+0000 7f802c043640 10 client.4314.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T18:28:48.741 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:28:48.739+0000 7f802c043640 10 monclient: _renew_subs 2024-03-25T18:28:48.741 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:28:48.739+0000 7f802c043640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.1:6789/0 2024-03-25T18:28:48.742 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:28:48.739+0000 7f802c043640 1 -- 172.21.15.1:0/1669196940 --> v1:172.21.15.1:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f8024143680 con 0x7f802413c480 2024-03-25T18:28:48.742 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:28:48.739+0000 7f802c043640 1 librados: init done 2024-03-25T18:28:48.742 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:28:48.740+0000 7f802c043640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T18:28:48.742 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:28:48.740+0000 7f802c043640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:28:48.742 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:28:48.740+0000 7f802c043640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.1:6789/0 2024-03-25T18:28:48.742 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:28:48.740+0000 7f802c043640 1 -- 172.21.15.1:0/1669196940 --> v1:172.21.15.1:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f80240688c0 con 0x7f802413c480 2024-03-25T18:28:49.054 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:28:49.054+0000 7f801a7fc640 1 -- 172.21.15.1:0/1669196940 <== mon.0 v1:172.21.15.1:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f801c001f60 con 0x7f802413c480 2024-03-25T18:28:49.055 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:28:49.054+0000 7f801a7fc640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:28:49.055 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:28:49.054+0000 7f801a7fc640 1 -- 172.21.15.1:0/1669196940 <== mon.0 v1:172.21.15.1:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1586965025 0 0) 0x7f801c003b10 con 0x7f802413c480 2024-03-25T18:28:49.055 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:28:49.054+0000 7f801a7fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:28:49.055 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:28:49.054+0000 7f801a7fc640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-25T18:28:49.055 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:28:49.054+0000 7f801a7fc640 10 monclient: dump: 2024-03-25T18:28:49.055 INFO:teuthology.misc.health.smithi001.stderr:epoch 1 2024-03-25T18:28:49.055 INFO:teuthology.misc.health.smithi001.stderr:fsid c8ea72cf-9e99-4383-9b52-6c23d365fdfc 2024-03-25T18:28:49.055 INFO:teuthology.misc.health.smithi001.stderr:last_changed 2024-03-25T18:25:50.510519+0000 2024-03-25T18:28:49.055 INFO:teuthology.misc.health.smithi001.stderr:created 2024-03-25T18:25:50.510519+0000 2024-03-25T18:28:49.055 INFO:teuthology.misc.health.smithi001.stderr:min_mon_release 19 (squid) 2024-03-25T18:28:49.055 INFO:teuthology.misc.health.smithi001.stderr:election_strategy: 3 2024-03-25T18:28:49.055 INFO:teuthology.misc.health.smithi001.stderr:0: v1:172.21.15.1:6789/0 mon.a 2024-03-25T18:28:49.055 INFO:teuthology.misc.health.smithi001.stderr:1: v1:172.21.15.115:6789/0 mon.b 2024-03-25T18:28:49.055 INFO:teuthology.misc.health.smithi001.stderr:2: v1:172.21.15.1:6790/0 mon.c 2024-03-25T18:28:49.056 INFO:teuthology.misc.health.smithi001.stderr: 2024-03-25T18:28:49.056 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:28:49.054+0000 7f801bfff640 4 set_mon_vals no callback set 2024-03-25T18:28:49.056 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:28:49.055+0000 7f801a7fc640 1 -- 172.21.15.1:0/1669196940 <== mon.0 v1:172.21.15.1:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 21787975 0 0) 0x7f801c019ae0 con 0x7f802413c480 2024-03-25T18:28:49.056 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:28:49.055+0000 7f8029ddf640 10 monclient: get_auth_request con 0x7f800806de20 auth_method 0 2024-03-25T18:28:49.056 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:28:49.055+0000 7f801a7fc640 10 client.4314.objecter ms_handle_connect 0x7f800806de20 2024-03-25T18:28:49.061 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:28:49.060+0000 7f801a7fc640 1 -- 172.21.15.1:0/1669196940 <== mon.0 v1:172.21.15.1:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 996254230 0 0) 0x7f801c053730 con 0x7f802413c480 2024-03-25T18:28:49.061 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:28:49.060+0000 7f801a7fc640 10 client.4314.objecter ms_dispatch 0x7f8024000b90 osd_map(10..10 src has 1..10) v4 2024-03-25T18:28:49.061 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:28:49.060+0000 7f801a7fc640 3 client.4314.objecter handle_osd_map got epochs [10,10] > 0 2024-03-25T18:28:49.061 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:28:49.060+0000 7f801a7fc640 3 client.4314.objecter handle_osd_map decoding full epoch 10 2024-03-25T18:28:49.061 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:28:49.060+0000 7f801a7fc640 20 client.4314.objecter dump_active .. 0 homeless 2024-03-25T18:28:49.062 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:28:49.060+0000 7f801a7fc640 1 -- 172.21.15.1:0/1669196940 <== mon.0 v1:172.21.15.1:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f801c04f240 con 0x7f802413c480 2024-03-25T18:28:49.062 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:28:49.060+0000 7f801a7fc640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:28:49.062 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:28:49.060+0000 7f801a7fc640 10 monclient: _finish_command 1 = system:0 2024-03-25T18:28:49.319 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:28:49.318+0000 7f802c043640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:28:49.319 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:28:49.319+0000 7f802c043640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:28:49.320 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:28:49.319+0000 7f802c043640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.1:6789/0 2024-03-25T18:28:49.320 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:28:49.319+0000 7f802c043640 1 -- 172.21.15.1:0/1669196940 --> v1:172.21.15.1:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f802413c240 con 0x7f802413c480 2024-03-25T18:28:49.361 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:28:49.360+0000 7f801a7fc640 1 -- 172.21.15.1:0/1669196940 <== mon.0 v1:172.21.15.1:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 4092213348) 0x7f801c051170 con 0x7f802413c480 2024-03-25T18:28:49.361 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:28:49.360+0000 7f801a7fc640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:28:49.361 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:28:49.360+0000 7f801a7fc640 10 monclient: _finish_command 2 = system:0 2024-03-25T18:28:49.361 INFO:teuthology.misc.health.smithi001.stdout: 2024-03-25T18:28:49.362 INFO:teuthology.misc.health.smithi001.stdout:{"epoch":10,"fsid":"c8ea72cf-9e99-4383-9b52-6c23d365fdfc","created":"2024-03-25T18:26:38.856638+0000","modified":"2024-03-25T18:26:44.224208+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T18:26:44.224208+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"b9f34922-3a84-49be-8426-807c1eceaefc","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"42bf306f-55e3-4634-9874-8b4b73ac6644","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"292cb23a-18a9-4207-a7cc-4d5727feaec0","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"45e0e9ab-898d-4673-9590-b4117bc5e84a","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"831a5a09-73ab-4a61-9872-088e76bdbc66","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"88cd903d-3b55-4840-9396-6e3cfe005282","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"89db7355-bc9d-46f4-8ab0-e8d9e744e026","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"cac47bee-0914-4f08-896f-348838882866","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T18:28:49.363 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:28:49.362+0000 7f802c043640 10 librados: watch_flush enter 2024-03-25T18:28:49.364 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:28:49.363+0000 7f802c043640 10 librados: watch_flush exit 2024-03-25T18:28:49.364 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:28:49.363+0000 7f802c043640 20 client.4314.objecter shutdown clearing up homeless session... 2024-03-25T18:28:49.364 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:28:49.363+0000 7f802c043640 10 client.4314.objecter successfully canceled tick 2024-03-25T18:28:49.364 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:28:49.363+0000 7f802c043640 1 -- 172.21.15.1:0/1669196940 >> v1:172.21.15.115:6800/4237379176 conn(0x7f800806de20 legacy=0x7f80080702e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:28:49.364 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:28:49.363+0000 7f802c043640 10 monclient: shutdown 2024-03-25T18:28:49.364 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:28:49.363+0000 7f802c043640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:28:49.364 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:28:49.363+0000 7f802c043640 1 -- 172.21.15.1:0/1669196940 >> v1:172.21.15.1:6789/0 conn(0x7f802413c480 legacy=0x7f80241457a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:28:49.364 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:28:49.363+0000 7f802c043640 1 -- 172.21.15.1:0/1669196940 shutdown_connections 2024-03-25T18:28:49.364 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:28:49.363+0000 7f802c043640 1 -- 172.21.15.1:0/1669196940 >> 172.21.15.1:0/1669196940 conn(0x7f80240617a0 msgr2=0x7f80240b8690 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:28:49.364 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:28:49.363+0000 7f802c043640 1 -- 172.21.15.1:0/1669196940 shutdown_connections 2024-03-25T18:28:49.364 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:28:49.363+0000 7f802c043640 1 -- 172.21.15.1:0/1669196940 wait complete. 2024-03-25T18:28:49.364 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:28:49.363+0000 7f802c043640 1 librados: shutdown 2024-03-25T18:28:49.375 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T18:28:56.182 DEBUG:teuthology.orchestra.run.smithi001:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T18:28:56.195 INFO:tasks.ceph.mon.a.smithi001.stderr:==00:00:02:34.753 34876== Warning: unimplemented fcntl command: 1036 2024-03-25T18:28:56.248 INFO:tasks.ceph.mon.a.smithi001.stderr:==00:00:02:34.807 34876== Warning: unimplemented fcntl command: 1036 2024-03-25T18:28:56.270 INFO:tasks.ceph.mon.a.smithi001.stderr:==00:00:02:34.829 34876== Warning: unimplemented fcntl command: 1036 2024-03-25T18:28:56.277 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:28:56.276+0000 7fe3cf8a8640 10 monclient: get_monmap_and_config 2024-03-25T18:28:56.277 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:28:56.276+0000 7fe3cf8a8640 10 monclient: build_initial_monmap 2024-03-25T18:28:56.278 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:28:56.276+0000 7fe3cf8a8640 10 monclient: monmap: 2024-03-25T18:28:56.278 INFO:teuthology.misc.health.smithi001.stderr:epoch 0 2024-03-25T18:28:56.278 INFO:teuthology.misc.health.smithi001.stderr:fsid c8ea72cf-9e99-4383-9b52-6c23d365fdfc 2024-03-25T18:28:56.278 INFO:teuthology.misc.health.smithi001.stderr:last_changed 2024-03-25T18:28:56.277868+0000 2024-03-25T18:28:56.278 INFO:teuthology.misc.health.smithi001.stderr:created 2024-03-25T18:28:56.277868+0000 2024-03-25T18:28:56.278 INFO:teuthology.misc.health.smithi001.stderr:min_mon_release 0 (unknown) 2024-03-25T18:28:56.278 INFO:teuthology.misc.health.smithi001.stderr:election_strategy: 3 2024-03-25T18:28:56.278 INFO:teuthology.misc.health.smithi001.stderr:0: v1:172.21.15.1:6789/0 mon.noname-a 2024-03-25T18:28:56.278 INFO:teuthology.misc.health.smithi001.stderr:1: v1:172.21.15.115:6789/0 mon.noname-c 2024-03-25T18:28:56.278 INFO:teuthology.misc.health.smithi001.stderr:2: v1:172.21.15.1:6790/0 mon.noname-b 2024-03-25T18:28:56.278 INFO:teuthology.misc.health.smithi001.stderr: 2024-03-25T18:28:56.279 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:28:56.276+0000 7fe3cf8a8640 1 Processor -- start 2024-03-25T18:28:56.280 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:28:56.277+0000 7fe3cf8a8640 1 -- start start 2024-03-25T18:28:56.280 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:28:56.277+0000 7fe3cf8a8640 10 monclient: init 2024-03-25T18:28:56.280 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:28:56.277+0000 7fe3cf8a8640 10 monclient: _reopen_session rank -1 2024-03-25T18:28:56.280 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:28:56.277+0000 7fe3cf8a8640 10 monclient: _add_conns ranks=[0,1,2] 2024-03-25T18:28:56.280 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:28:56.277+0000 7fe3cf8a8640 10 monclient(hunting): picked mon.noname-a con 0x7fe3c80b4e70 addr v1:172.21.15.1:6789/0 2024-03-25T18:28:56.280 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:28:56.277+0000 7fe3cf8a8640 10 monclient(hunting): picked mon.noname-c con 0x7fe3c80b44b0 addr v1:172.21.15.115:6789/0 2024-03-25T18:28:56.280 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:28:56.277+0000 7fe3cf8a8640 10 monclient(hunting): picked mon.noname-b con 0x7fe3c80b1b00 addr v1:172.21.15.1:6790/0 2024-03-25T18:28:56.280 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:28:56.277+0000 7fe3cf8a8640 1 -- --> v1:172.21.15.1:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe3c80a96f0 con 0x7fe3c80b4e70 2024-03-25T18:28:56.280 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:28:56.277+0000 7fe3cf8a8640 1 -- --> v1:172.21.15.115:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe3c80ab0e0 con 0x7fe3c80b44b0 2024-03-25T18:28:56.280 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:28:56.277+0000 7fe3cf8a8640 1 -- --> v1:172.21.15.1:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe3c8145f70 con 0x7fe3c80b1b00 2024-03-25T18:28:56.281 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:28:56.277+0000 7fe3cf8a8640 10 monclient(hunting): _renew_subs 2024-03-25T18:28:56.281 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:28:56.277+0000 7fe3cf8a8640 10 monclient(hunting): authenticate will time out at 7616.748361s 2024-03-25T18:28:56.296 INFO:tasks.ceph.mon.b.smithi115.stderr:==00:00:02:34.779 34836== Warning: unimplemented fcntl command: 1036 2024-03-25T18:28:56.298 INFO:tasks.ceph.mon.b.smithi115.stderr:==00:00:02:34.782 34836== Warning: unimplemented fcntl command: 1036 2024-03-25T18:28:56.309 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:28:56.307+0000 7fe3cce43640 1 --1- >> v1:172.21.15.115:6789/0 conn(0x7fe3c80b44b0 0x7fe3c8145860 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.115:6789/0 says I am v1:172.21.15.1:52266/0 (socket says 172.21.15.1:52266) 2024-03-25T18:28:56.309 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:28:56.307+0000 7fe3cce43640 1 -- 172.21.15.1:0/1784259784 learned_addr learned my addr 172.21.15.1:0/1784259784 (peer_addr_for_me v1:172.21.15.1:0/0) 2024-03-25T18:28:56.317 INFO:tasks.ceph.mon.c.smithi001.stderr:==00:00:02:34.875 34877== Warning: unimplemented fcntl command: 1036 2024-03-25T18:28:56.326 INFO:tasks.ceph.mon.b.smithi115.stderr:==00:00:02:34.809 34836== Warning: unimplemented fcntl command: 1036 2024-03-25T18:28:56.335 INFO:tasks.ceph.mon.c.smithi001.stderr:==00:00:02:34.894 34877== Warning: unimplemented fcntl command: 1036 2024-03-25T18:28:56.351 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:28:56.349+0000 7fe3bffff640 1 -- 172.21.15.1:0/1784259784 <== mon.0 v1:172.21.15.1:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2580509691 0 0) 0x7fe3c80a96f0 con 0x7fe3c80b4e70 2024-03-25T18:28:56.351 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:28:56.349+0000 7fe3bffff640 10 monclient(hunting): _negotiate 2024-03-25T18:28:56.351 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:28:56.349+0000 7fe3bffff640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:28:56.351 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:28:56.349+0000 7fe3bffff640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:28:56.351 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:28:56.349+0000 7fe3bffff640 10 monclient(hunting): my global_id is 4317 2024-03-25T18:28:56.351 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:28:56.349+0000 7fe3bffff640 1 -- 172.21.15.1:0/1784259784 --> v1:172.21.15.1:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fe3b4003520 con 0x7fe3c80b4e70 2024-03-25T18:28:56.354 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:28:56.353+0000 7fe3bffff640 1 -- 172.21.15.1:0/1784259784 <== mon.2 v1:172.21.15.1:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1971334048 0 0) 0x7fe3c8145f70 con 0x7fe3c80b1b00 2024-03-25T18:28:56.354 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:28:56.353+0000 7fe3bffff640 10 monclient(hunting): _negotiate 2024-03-25T18:28:56.354 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:28:56.353+0000 7fe3bffff640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:28:56.354 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:28:56.353+0000 7fe3bffff640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:28:56.354 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:28:56.353+0000 7fe3bffff640 10 monclient(hunting): my global_id is 4319 2024-03-25T18:28:56.355 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:28:56.353+0000 7fe3bffff640 1 -- 172.21.15.1:0/1784259784 --> v1:172.21.15.1:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fe3c80a96f0 con 0x7fe3c80b1b00 2024-03-25T18:28:56.362 INFO:tasks.ceph.mon.c.smithi001.stderr:==00:00:02:34.920 34877== Warning: unimplemented fcntl command: 1036 2024-03-25T18:28:56.364 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:28:56.363+0000 7fe3bffff640 1 -- 172.21.15.1:0/1784259784 <== mon.1 v1:172.21.15.115:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2012396770 0 0) 0x7fe3c80ab0e0 con 0x7fe3c80b44b0 2024-03-25T18:28:56.365 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:28:56.363+0000 7fe3bffff640 10 monclient(hunting): _negotiate 2024-03-25T18:28:56.365 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:28:56.363+0000 7fe3bffff640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:28:56.365 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:28:56.363+0000 7fe3bffff640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:28:56.365 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:28:56.363+0000 7fe3bffff640 10 monclient(hunting): my global_id is 4312 2024-03-25T18:28:56.365 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:28:56.363+0000 7fe3bffff640 1 -- 172.21.15.1:0/1784259784 --> v1:172.21.15.115:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fe3c8145f70 con 0x7fe3c80b44b0 2024-03-25T18:28:56.373 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:28:56.371+0000 7fe3bffff640 1 -- 172.21.15.1:0/1784259784 <== mon.2 v1:172.21.15.1:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3783607342 0 0) 0x7fe3c80a96f0 con 0x7fe3c80b1b00 2024-03-25T18:28:56.373 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:28:56.371+0000 7fe3bffff640 1 -- 172.21.15.1:0/1784259784 >> v1:172.21.15.115:6789/0 conn(0x7fe3c80b44b0 legacy=0x7fe3c8145860 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:28:56.373 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:28:56.371+0000 7fe3bffff640 1 -- 172.21.15.1:0/1784259784 >> v1:172.21.15.1:6789/0 conn(0x7fe3c80b4e70 legacy=0x7fe3c813f940 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:28:56.374 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:28:56.371+0000 7fe3bffff640 10 monclient: _finish_hunting 0 2024-03-25T18:28:56.374 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:28:56.371+0000 7fe3bffff640 1 monclient: found mon.noname-b 2024-03-25T18:28:56.374 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:28:56.371+0000 7fe3bffff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:28:56.374 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:28:56.371+0000 7fe3bffff640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.1:6790/0 2024-03-25T18:28:56.374 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:28:56.371+0000 7fe3bffff640 1 -- 172.21.15.1:0/1784259784 --> v1:172.21.15.1:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe3c80a9b80 con 0x7fe3c80b1b00 2024-03-25T18:28:56.374 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:28:56.371+0000 7fe3bffff640 10 monclient: _finish_auth 0 2024-03-25T18:28:56.374 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:28:56.371+0000 7fe3bffff640 10 monclient: _check_auth_tickets 2024-03-25T18:28:56.374 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:28:56.371+0000 7fe3bffff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:28:56.374 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:28:56.371+0000 7fe3bffff640 1 -- 172.21.15.1:0/1784259784 <== mon.2 v1:172.21.15.1:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1586965025 0 0) 0x7fe3c00019f0 con 0x7fe3c80b1b00 2024-03-25T18:28:56.374 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:28:56.371+0000 7fe3cf8a8640 5 monclient: authenticate success, global_id 4319 2024-03-25T18:28:56.374 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:28:56.371+0000 7fe3cf8a8640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T18:28:56.374 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:28:56.371+0000 7fe3bffff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:28:56.375 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:28:56.371+0000 7fe3bffff640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-03-25T18:28:56.375 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:28:56.371+0000 7fe3bffff640 10 monclient: dump: 2024-03-25T18:28:56.375 INFO:teuthology.misc.health.smithi001.stderr:epoch 1 2024-03-25T18:28:56.375 INFO:teuthology.misc.health.smithi001.stderr:fsid c8ea72cf-9e99-4383-9b52-6c23d365fdfc 2024-03-25T18:28:56.375 INFO:teuthology.misc.health.smithi001.stderr:last_changed 2024-03-25T18:25:50.510519+0000 2024-03-25T18:28:56.375 INFO:teuthology.misc.health.smithi001.stderr:created 2024-03-25T18:25:50.510519+0000 2024-03-25T18:28:56.375 INFO:teuthology.misc.health.smithi001.stderr:min_mon_release 19 (squid) 2024-03-25T18:28:56.375 INFO:teuthology.misc.health.smithi001.stderr:election_strategy: 3 2024-03-25T18:28:56.375 INFO:teuthology.misc.health.smithi001.stderr:0: v1:172.21.15.1:6789/0 mon.a 2024-03-25T18:28:56.375 INFO:teuthology.misc.health.smithi001.stderr:1: v1:172.21.15.115:6789/0 mon.b 2024-03-25T18:28:56.375 INFO:teuthology.misc.health.smithi001.stderr:2: v1:172.21.15.1:6790/0 mon.c 2024-03-25T18:28:56.376 INFO:teuthology.misc.health.smithi001.stderr: 2024-03-25T18:28:56.376 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:28:56.371+0000 7fe3cf8a8640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T18:28:56.394 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:28:56.392+0000 7fe3bffff640 1 -- 172.21.15.1:0/1784259784 <== mon.2 v1:172.21.15.1:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fe3c0001b50 con 0x7fe3c80b1b00 2024-03-25T18:28:56.394 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:28:56.392+0000 7fe3bffff640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:28:56.394 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:28:56.392+0000 7fe3bffff640 1 -- 172.21.15.1:0/1784259784 <== mon.2 v1:172.21.15.1:6790/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1586965025 0 0) 0x7fe3c0001f60 con 0x7fe3c80b1b00 2024-03-25T18:28:56.394 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:28:56.392+0000 7fe3bffff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:28:56.394 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:28:56.392+0000 7fe3bffff640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-25T18:28:56.395 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:28:56.392+0000 7fe3bffff640 10 monclient: dump: 2024-03-25T18:28:56.395 INFO:teuthology.misc.health.smithi001.stderr:epoch 1 2024-03-25T18:28:56.395 INFO:teuthology.misc.health.smithi001.stderr:fsid c8ea72cf-9e99-4383-9b52-6c23d365fdfc 2024-03-25T18:28:56.395 INFO:teuthology.misc.health.smithi001.stderr:last_changed 2024-03-25T18:25:50.510519+0000 2024-03-25T18:28:56.395 INFO:teuthology.misc.health.smithi001.stderr:created 2024-03-25T18:25:50.510519+0000 2024-03-25T18:28:56.395 INFO:teuthology.misc.health.smithi001.stderr:min_mon_release 19 (squid) 2024-03-25T18:28:56.395 INFO:teuthology.misc.health.smithi001.stderr:election_strategy: 3 2024-03-25T18:28:56.395 INFO:teuthology.misc.health.smithi001.stderr:0: v1:172.21.15.1:6789/0 mon.a 2024-03-25T18:28:56.395 INFO:teuthology.misc.health.smithi001.stderr:1: v1:172.21.15.115:6789/0 mon.b 2024-03-25T18:28:56.395 INFO:teuthology.misc.health.smithi001.stderr:2: v1:172.21.15.1:6790/0 mon.c 2024-03-25T18:28:56.395 INFO:teuthology.misc.health.smithi001.stderr: 2024-03-25T18:28:56.395 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:28:56.392+0000 7fe3cf8a8640 10 monclient: get_monmap_and_config success 2024-03-25T18:28:56.395 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:28:56.392+0000 7fe3cf8a8640 4 set_mon_vals no callback set 2024-03-25T18:28:56.397 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:28:56.393+0000 7fe3cf8a8640 10 monclient: shutdown 2024-03-25T18:28:56.397 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:28:56.393+0000 7fe3cf8a8640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:28:56.397 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:28:56.393+0000 7fe3cf8a8640 1 -- 172.21.15.1:0/1784259784 >> v1:172.21.15.1:6790/0 conn(0x7fe3c80b1b00 legacy=0x7fe3c814dc00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:28:56.397 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:28:56.393+0000 7fe3cf8a8640 1 -- 172.21.15.1:0/1784259784 shutdown_connections 2024-03-25T18:28:56.397 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:28:56.393+0000 7fe3cf8a8640 1 -- 172.21.15.1:0/1784259784 >> 172.21.15.1:0/1784259784 conn(0x7fe3c80617a0 msgr2=0x7fe3c8061bb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:28:56.397 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:28:56.393+0000 7fe3cf8a8640 1 -- 172.21.15.1:0/1784259784 shutdown_connections 2024-03-25T18:28:56.397 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:28:56.393+0000 7fe3cf8a8640 1 -- 172.21.15.1:0/1784259784 wait complete. 2024-03-25T18:28:56.397 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:28:56.394+0000 7fe3cf8a8640 10 monclient: build_initial_monmap 2024-03-25T18:28:56.397 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:28:56.394+0000 7fe3cf8a8640 10 monclient: monmap: 2024-03-25T18:28:56.397 INFO:teuthology.misc.health.smithi001.stderr:epoch 0 2024-03-25T18:28:56.397 INFO:teuthology.misc.health.smithi001.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T18:28:56.398 INFO:teuthology.misc.health.smithi001.stderr:last_changed 0.000000 2024-03-25T18:28:56.398 INFO:teuthology.misc.health.smithi001.stderr:created 0.000000 2024-03-25T18:28:56.398 INFO:teuthology.misc.health.smithi001.stderr:min_mon_release 0 (unknown) 2024-03-25T18:28:56.398 INFO:teuthology.misc.health.smithi001.stderr:election_strategy: 1 2024-03-25T18:28:56.398 INFO:teuthology.misc.health.smithi001.stderr:0: v1:172.21.15.1:6789/0 mon.noname-a 2024-03-25T18:28:56.398 INFO:teuthology.misc.health.smithi001.stderr:1: v1:172.21.15.115:6789/0 mon.noname-b 2024-03-25T18:28:56.398 INFO:teuthology.misc.health.smithi001.stderr:2: v1:172.21.15.1:6790/0 mon.noname-c 2024-03-25T18:28:56.398 INFO:teuthology.misc.health.smithi001.stderr: 2024-03-25T18:28:56.398 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:28:56.395+0000 7fe3cf8a8640 1 librados: starting msgr at 2024-03-25T18:28:56.398 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:28:56.395+0000 7fe3cf8a8640 1 librados: starting objecter 2024-03-25T18:28:56.398 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:28:56.395+0000 7fe3cf8a8640 1 Processor -- start 2024-03-25T18:28:56.398 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:28:56.395+0000 7fe3cf8a8640 1 -- start start 2024-03-25T18:28:56.398 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:28:56.395+0000 7fe3cf8a8640 1 librados: setting wanted keys 2024-03-25T18:28:56.398 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:28:56.395+0000 7fe3cf8a8640 1 librados: calling monclient init 2024-03-25T18:28:56.399 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:28:56.395+0000 7fe3cf8a8640 10 monclient: init 2024-03-25T18:28:56.400 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:28:56.396+0000 7fe3cf8a8640 10 monclient: _reopen_session rank -1 2024-03-25T18:28:56.400 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:28:56.396+0000 7fe3cf8a8640 10 monclient: _add_conns ranks=[1,0,2] 2024-03-25T18:28:56.400 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:28:56.396+0000 7fe3cf8a8640 10 monclient(hunting): picked mon.noname-b con 0x7fe3c80b1b00 addr v1:172.21.15.115:6789/0 2024-03-25T18:28:56.400 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:28:56.396+0000 7fe3cf8a8640 10 monclient(hunting): picked mon.noname-a con 0x7fe3c80b44b0 addr v1:172.21.15.1:6789/0 2024-03-25T18:28:56.400 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:28:56.396+0000 7fe3cf8a8640 10 monclient(hunting): picked mon.noname-c con 0x7fe3c80b4e70 addr v1:172.21.15.1:6790/0 2024-03-25T18:28:56.400 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:28:56.396+0000 7fe3cf8a8640 1 -- --> v1:172.21.15.1:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe3c8134950 con 0x7fe3c80b44b0 2024-03-25T18:28:56.400 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:28:56.396+0000 7fe3cf8a8640 1 -- --> v1:172.21.15.115:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe3c8137a80 con 0x7fe3c80b1b00 2024-03-25T18:28:56.400 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:28:56.396+0000 7fe3cf8a8640 1 -- --> v1:172.21.15.1:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe3c8137d60 con 0x7fe3c80b4e70 2024-03-25T18:28:56.400 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:28:56.396+0000 7fe3cf8a8640 10 monclient(hunting): _renew_subs 2024-03-25T18:28:56.400 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:28:56.396+0000 7fe3cf8a8640 10 monclient(hunting): authenticate will time out at 7616.867037s 2024-03-25T18:28:56.404 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:28:56.402+0000 7fe3cce43640 1 --1- >> v1:172.21.15.1:6789/0 conn(0x7fe3c80b44b0 0x7fe3c8133b30 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.1:6789/0 says I am v1:172.21.15.1:34462/0 (socket says 172.21.15.1:34462) 2024-03-25T18:28:56.404 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:28:56.402+0000 7fe3cce43640 1 -- 172.21.15.1:0/4041090277 learned_addr learned my addr 172.21.15.1:0/4041090277 (peer_addr_for_me v1:172.21.15.1:0/0) 2024-03-25T18:28:56.407 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:28:56.406+0000 7fe3bdffb640 10 client.?.objecter ms_handle_connect 0x7fe3c80b44b0 2024-03-25T18:28:56.407 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:28:56.406+0000 7fe3bdffb640 10 client.?.objecter resend_mon_ops 2024-03-25T18:28:56.418 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:28:56.416+0000 7fe3bdffb640 1 -- 172.21.15.1:0/4041090277 <== mon.0 v1:172.21.15.1:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2108657176 0 0) 0x7fe3c8134950 con 0x7fe3c80b44b0 2024-03-25T18:28:56.418 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:28:56.416+0000 7fe3bdffb640 10 monclient(hunting): _negotiate 2024-03-25T18:28:56.418 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:28:56.416+0000 7fe3bdffb640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:28:56.418 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:28:56.416+0000 7fe3bdffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:28:56.418 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:28:56.416+0000 7fe3bdffb640 10 monclient(hunting): my global_id is 4320 2024-03-25T18:28:56.418 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:28:56.416+0000 7fe3bdffb640 1 -- 172.21.15.1:0/4041090277 --> v1:172.21.15.1:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fe3ac004eb0 con 0x7fe3c80b44b0 2024-03-25T18:28:56.422 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:28:56.420+0000 7fe3bdffb640 10 client.?.objecter ms_handle_connect 0x7fe3c80b1b00 2024-03-25T18:28:56.422 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:28:56.420+0000 7fe3bdffb640 10 client.?.objecter resend_mon_ops 2024-03-25T18:28:56.429 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:28:56.428+0000 7fe3bdffb640 1 -- 172.21.15.1:0/4041090277 <== mon.1 v1:172.21.15.115:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 172627957 0 0) 0x7fe3c8137a80 con 0x7fe3c80b1b00 2024-03-25T18:28:56.430 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:28:56.428+0000 7fe3bdffb640 10 monclient(hunting): _negotiate 2024-03-25T18:28:56.430 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:28:56.428+0000 7fe3bdffb640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:28:56.430 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:28:56.428+0000 7fe3bdffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:28:56.430 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:28:56.428+0000 7fe3bdffb640 10 monclient(hunting): my global_id is 4315 2024-03-25T18:28:56.430 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:28:56.428+0000 7fe3bdffb640 1 -- 172.21.15.1:0/4041090277 --> v1:172.21.15.115:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fe3c8134950 con 0x7fe3c80b1b00 2024-03-25T18:28:56.432 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:28:56.430+0000 7fe3bdffb640 10 client.?.objecter ms_handle_connect 0x7fe3c80b4e70 2024-03-25T18:28:56.432 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:28:56.430+0000 7fe3bdffb640 10 client.?.objecter resend_mon_ops 2024-03-25T18:28:56.437 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:28:56.435+0000 7fe3bdffb640 1 -- 172.21.15.1:0/4041090277 <== mon.0 v1:172.21.15.1:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1842459133 0 0) 0x7fe3ac004eb0 con 0x7fe3c80b44b0 2024-03-25T18:28:56.437 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:28:56.435+0000 7fe3bdffb640 1 -- 172.21.15.1:0/4041090277 --> v1:172.21.15.1:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fe3c8137a80 con 0x7fe3c80b44b0 2024-03-25T18:28:56.437 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:28:56.435+0000 7fe3bdffb640 1 -- 172.21.15.1:0/4041090277 <== mon.0 v1:172.21.15.1:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1586965025 0 0) 0x7fe3b80019b0 con 0x7fe3c80b44b0 2024-03-25T18:28:56.437 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:28:56.435+0000 7fe3bdffb640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:28:56.437 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:28:56.435+0000 7fe3bdffb640 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2024-03-25T18:28:56.437 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:28:56.435+0000 7fe3bdffb640 10 monclient(hunting): dump: 2024-03-25T18:28:56.437 INFO:teuthology.misc.health.smithi001.stderr:epoch 1 2024-03-25T18:28:56.437 INFO:teuthology.misc.health.smithi001.stderr:fsid c8ea72cf-9e99-4383-9b52-6c23d365fdfc 2024-03-25T18:28:56.437 INFO:teuthology.misc.health.smithi001.stderr:last_changed 2024-03-25T18:25:50.510519+0000 2024-03-25T18:28:56.437 INFO:teuthology.misc.health.smithi001.stderr:created 2024-03-25T18:25:50.510519+0000 2024-03-25T18:28:56.437 INFO:teuthology.misc.health.smithi001.stderr:min_mon_release 19 (squid) 2024-03-25T18:28:56.437 INFO:teuthology.misc.health.smithi001.stderr:election_strategy: 3 2024-03-25T18:28:56.437 INFO:teuthology.misc.health.smithi001.stderr:0: v1:172.21.15.1:6789/0 mon.a 2024-03-25T18:28:56.438 INFO:teuthology.misc.health.smithi001.stderr:1: v1:172.21.15.115:6789/0 mon.b 2024-03-25T18:28:56.438 INFO:teuthology.misc.health.smithi001.stderr:2: v1:172.21.15.1:6790/0 mon.c 2024-03-25T18:28:56.438 INFO:teuthology.misc.health.smithi001.stderr: 2024-03-25T18:28:56.438 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:28:56.435+0000 7fe3bdffb640 10 monclient(hunting): _finish_auth 0 2024-03-25T18:28:56.440 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:28:56.438+0000 7fe3bdffb640 1 -- 172.21.15.1:0/4041090277 <== mon.1 v1:172.21.15.115:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2818193989 0 0) 0x7fe3c8134950 con 0x7fe3c80b1b00 2024-03-25T18:28:56.440 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:28:56.438+0000 7fe3bdffb640 1 -- 172.21.15.1:0/4041090277 --> v1:172.21.15.115:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fe3ac004eb0 con 0x7fe3c80b1b00 2024-03-25T18:28:56.440 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:28:56.438+0000 7fe3bdffb640 1 -- 172.21.15.1:0/4041090277 <== mon.1 v1:172.21.15.115:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1586965025 0 0) 0x7fe3c4001790 con 0x7fe3c80b1b00 2024-03-25T18:28:56.440 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:28:56.438+0000 7fe3bdffb640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:28:56.440 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:28:56.438+0000 7fe3bdffb640 10 monclient(hunting): got monmap 1 from mon.b (according to old e1) 2024-03-25T18:28:56.440 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:28:56.438+0000 7fe3bdffb640 10 monclient(hunting): dump: 2024-03-25T18:28:56.440 INFO:teuthology.misc.health.smithi001.stderr:epoch 1 2024-03-25T18:28:56.440 INFO:teuthology.misc.health.smithi001.stderr:fsid c8ea72cf-9e99-4383-9b52-6c23d365fdfc 2024-03-25T18:28:56.440 INFO:teuthology.misc.health.smithi001.stderr:last_changed 2024-03-25T18:25:50.510519+0000 2024-03-25T18:28:56.440 INFO:teuthology.misc.health.smithi001.stderr:created 2024-03-25T18:25:50.510519+0000 2024-03-25T18:28:56.441 INFO:teuthology.misc.health.smithi001.stderr:min_mon_release 19 (squid) 2024-03-25T18:28:56.441 INFO:teuthology.misc.health.smithi001.stderr:election_strategy: 3 2024-03-25T18:28:56.441 INFO:teuthology.misc.health.smithi001.stderr:0: v1:172.21.15.1:6789/0 mon.a 2024-03-25T18:28:56.441 INFO:teuthology.misc.health.smithi001.stderr:1: v1:172.21.15.115:6789/0 mon.b 2024-03-25T18:28:56.441 INFO:teuthology.misc.health.smithi001.stderr:2: v1:172.21.15.1:6790/0 mon.c 2024-03-25T18:28:56.441 INFO:teuthology.misc.health.smithi001.stderr: 2024-03-25T18:28:56.443 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:28:56.441+0000 7fe3bdffb640 1 -- 172.21.15.1:0/4041090277 <== mon.0 v1:172.21.15.1:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 2584307141 0 0) 0x7fe3c8137a80 con 0x7fe3c80b44b0 2024-03-25T18:28:56.443 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:28:56.441+0000 7fe3bdffb640 1 -- 172.21.15.1:0/4041090277 >> v1:172.21.15.1:6790/0 conn(0x7fe3c80b4e70 legacy=0x7fe3c8134240 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:28:56.443 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:28:56.441+0000 7fe3bdffb640 1 -- 172.21.15.1:0/4041090277 >> v1:172.21.15.115:6789/0 conn(0x7fe3c80b1b00 legacy=0x7fe3c81d7650 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:28:56.443 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:28:56.441+0000 7fe3bdffb640 10 monclient: _finish_hunting 0 2024-03-25T18:28:56.444 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:28:56.441+0000 7fe3bdffb640 1 monclient: found mon.a 2024-03-25T18:28:56.444 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:28:56.441+0000 7fe3bdffb640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:28:56.444 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:28:56.441+0000 7fe3bdffb640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.1:6789/0 2024-03-25T18:28:56.444 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:28:56.441+0000 7fe3bdffb640 1 -- 172.21.15.1:0/4041090277 --> v1:172.21.15.1:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe3c8136870 con 0x7fe3c80b44b0 2024-03-25T18:28:56.444 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:28:56.441+0000 7fe3bdffb640 10 monclient: _finish_auth 0 2024-03-25T18:28:56.444 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:28:56.441+0000 7fe3bdffb640 10 monclient: _check_auth_tickets 2024-03-25T18:28:56.444 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:28:56.441+0000 7fe3bdffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:28:56.444 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:28:56.441+0000 7fe3cf8a8640 5 monclient: authenticate success, global_id 4320 2024-03-25T18:28:56.444 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:28:56.441+0000 7fe3cf8a8640 10 monclient: _renew_subs 2024-03-25T18:28:56.444 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:28:56.441+0000 7fe3cf8a8640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.1:6789/0 2024-03-25T18:28:56.444 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:28:56.441+0000 7fe3cf8a8640 1 -- 172.21.15.1:0/4041090277 --> v1:172.21.15.1:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe3c8135970 con 0x7fe3c80b44b0 2024-03-25T18:28:56.444 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:28:56.441+0000 7fe3cf8a8640 10 client.4320.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T18:28:56.444 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:28:56.441+0000 7fe3cf8a8640 10 monclient: _renew_subs 2024-03-25T18:28:56.444 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:28:56.441+0000 7fe3cf8a8640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.1:6789/0 2024-03-25T18:28:56.445 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:28:56.441+0000 7fe3cf8a8640 1 -- 172.21.15.1:0/4041090277 --> v1:172.21.15.1:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7fe3c8136110 con 0x7fe3c80b44b0 2024-03-25T18:28:56.445 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:28:56.441+0000 7fe3cf8a8640 1 librados: init done 2024-03-25T18:28:56.445 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:28:56.442+0000 7fe3cf8a8640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T18:28:56.445 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:28:56.442+0000 7fe3cf8a8640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:28:56.445 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:28:56.442+0000 7fe3cf8a8640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.1:6789/0 2024-03-25T18:28:56.445 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:28:56.442+0000 7fe3cf8a8640 1 -- 172.21.15.1:0/4041090277 --> v1:172.21.15.1:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe3c804f730 con 0x7fe3c80b44b0 2024-03-25T18:28:56.648 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:28:56.647+0000 7fe3bdffb640 1 -- 172.21.15.1:0/4041090277 <== mon.0 v1:172.21.15.1:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fe3b80047e0 con 0x7fe3c80b44b0 2024-03-25T18:28:56.648 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:28:56.647+0000 7fe3bdffb640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:28:56.649 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:28:56.647+0000 7fe3bf7fe640 4 set_mon_vals no callback set 2024-03-25T18:28:56.756 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:28:56.754+0000 7fe3bdffb640 1 -- 172.21.15.1:0/4041090277 <== mon.0 v1:172.21.15.1:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1586965025 0 0) 0x7fe3b8004bf0 con 0x7fe3c80b44b0 2024-03-25T18:28:56.756 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:28:56.754+0000 7fe3bdffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:28:56.756 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:28:56.754+0000 7fe3bdffb640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-25T18:28:56.756 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:28:56.754+0000 7fe3bdffb640 10 monclient: dump: 2024-03-25T18:28:56.756 INFO:teuthology.misc.health.smithi001.stderr:epoch 1 2024-03-25T18:28:56.756 INFO:teuthology.misc.health.smithi001.stderr:fsid c8ea72cf-9e99-4383-9b52-6c23d365fdfc 2024-03-25T18:28:56.756 INFO:teuthology.misc.health.smithi001.stderr:last_changed 2024-03-25T18:25:50.510519+0000 2024-03-25T18:28:56.756 INFO:teuthology.misc.health.smithi001.stderr:created 2024-03-25T18:25:50.510519+0000 2024-03-25T18:28:56.756 INFO:teuthology.misc.health.smithi001.stderr:min_mon_release 19 (squid) 2024-03-25T18:28:56.757 INFO:teuthology.misc.health.smithi001.stderr:election_strategy: 3 2024-03-25T18:28:56.757 INFO:teuthology.misc.health.smithi001.stderr:0: v1:172.21.15.1:6789/0 mon.a 2024-03-25T18:28:56.757 INFO:teuthology.misc.health.smithi001.stderr:1: v1:172.21.15.115:6789/0 mon.b 2024-03-25T18:28:56.757 INFO:teuthology.misc.health.smithi001.stderr:2: v1:172.21.15.1:6790/0 mon.c 2024-03-25T18:28:56.757 INFO:teuthology.misc.health.smithi001.stderr: 2024-03-25T18:28:56.757 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:28:56.755+0000 7fe3bdffb640 1 -- 172.21.15.1:0/4041090277 <== mon.0 v1:172.21.15.1:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 21787975 0 0) 0x7fe3b8004dd0 con 0x7fe3c80b44b0 2024-03-25T18:28:56.758 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:28:56.755+0000 7fe3cd644640 10 monclient: get_auth_request con 0x7fe3ac06df20 auth_method 0 2024-03-25T18:28:56.758 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:28:56.756+0000 7fe3bdffb640 10 client.4320.objecter ms_handle_connect 0x7fe3ac06df20 2024-03-25T18:28:56.763 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:28:56.762+0000 7fe3bdffb640 1 -- 172.21.15.1:0/4041090277 <== mon.0 v1:172.21.15.1:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 996254230 0 0) 0x7fe3b8054880 con 0x7fe3c80b44b0 2024-03-25T18:28:56.763 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:28:56.762+0000 7fe3bdffb640 10 client.4320.objecter ms_dispatch 0x7fe3c8000b90 osd_map(10..10 src has 1..10) v4 2024-03-25T18:28:56.764 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:28:56.762+0000 7fe3bdffb640 3 client.4320.objecter handle_osd_map got epochs [10,10] > 0 2024-03-25T18:28:56.764 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:28:56.762+0000 7fe3bdffb640 3 client.4320.objecter handle_osd_map decoding full epoch 10 2024-03-25T18:28:56.764 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:28:56.762+0000 7fe3bdffb640 20 client.4320.objecter dump_active .. 0 homeless 2024-03-25T18:28:56.764 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:28:56.762+0000 7fe3bdffb640 1 -- 172.21.15.1:0/4041090277 <== mon.0 v1:172.21.15.1:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7fe3b8050390 con 0x7fe3c80b44b0 2024-03-25T18:28:56.764 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:28:56.762+0000 7fe3bdffb640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:28:56.764 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:28:56.762+0000 7fe3bdffb640 10 monclient: _finish_command 1 = system:0 2024-03-25T18:28:57.010 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:28:57.008+0000 7fe3cf8a8640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:28:57.010 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:28:57.008+0000 7fe3cf8a8640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:28:57.010 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:28:57.008+0000 7fe3cf8a8640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.1:6789/0 2024-03-25T18:28:57.010 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:28:57.008+0000 7fe3cf8a8640 1 -- 172.21.15.1:0/4041090277 --> v1:172.21.15.1:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fe3c80aa090 con 0x7fe3c80b44b0 2024-03-25T18:28:57.036 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:28:57.034+0000 7fe3bdffb640 1 -- 172.21.15.1:0/4041090277 <== mon.0 v1:172.21.15.1:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 4092213348) 0x7fe3b80522c0 con 0x7fe3c80b44b0 2024-03-25T18:28:57.036 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:28:57.034+0000 7fe3bdffb640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:28:57.036 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:28:57.034+0000 7fe3bdffb640 10 monclient: _finish_command 2 = system:0 2024-03-25T18:28:57.036 INFO:teuthology.misc.health.smithi001.stdout: 2024-03-25T18:28:57.036 INFO:teuthology.misc.health.smithi001.stdout:{"epoch":10,"fsid":"c8ea72cf-9e99-4383-9b52-6c23d365fdfc","created":"2024-03-25T18:26:38.856638+0000","modified":"2024-03-25T18:26:44.224208+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T18:26:44.224208+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"b9f34922-3a84-49be-8426-807c1eceaefc","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"42bf306f-55e3-4634-9874-8b4b73ac6644","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"292cb23a-18a9-4207-a7cc-4d5727feaec0","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"45e0e9ab-898d-4673-9590-b4117bc5e84a","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"831a5a09-73ab-4a61-9872-088e76bdbc66","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"88cd903d-3b55-4840-9396-6e3cfe005282","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"89db7355-bc9d-46f4-8ab0-e8d9e744e026","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"cac47bee-0914-4f08-896f-348838882866","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T18:28:57.039 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:28:57.037+0000 7fe3cf8a8640 10 librados: watch_flush enter 2024-03-25T18:28:57.039 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:28:57.037+0000 7fe3cf8a8640 10 librados: watch_flush exit 2024-03-25T18:28:57.039 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:28:57.037+0000 7fe3cf8a8640 20 client.4320.objecter shutdown clearing up homeless session... 2024-03-25T18:28:57.039 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:28:57.037+0000 7fe3cf8a8640 10 client.4320.objecter successfully canceled tick 2024-03-25T18:28:57.039 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:28:57.037+0000 7fe3cf8a8640 1 -- 172.21.15.1:0/4041090277 >> v1:172.21.15.115:6800/4237379176 conn(0x7fe3ac06df20 legacy=0x7fe3ac0703e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:28:57.040 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:28:57.037+0000 7fe3cf8a8640 10 monclient: shutdown 2024-03-25T18:28:57.040 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:28:57.037+0000 7fe3cf8a8640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:28:57.040 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:28:57.037+0000 7fe3cf8a8640 1 -- 172.21.15.1:0/4041090277 >> v1:172.21.15.1:6789/0 conn(0x7fe3c80b44b0 legacy=0x7fe3c8133b30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:28:57.040 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:28:57.037+0000 7fe3cf8a8640 1 -- 172.21.15.1:0/4041090277 shutdown_connections 2024-03-25T18:28:57.040 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:28:57.037+0000 7fe3cf8a8640 1 -- 172.21.15.1:0/4041090277 >> 172.21.15.1:0/4041090277 conn(0x7fe3c80617a0 msgr2=0x7fe3c80b2710 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:28:57.040 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:28:57.037+0000 7fe3cf8a8640 1 -- 172.21.15.1:0/4041090277 shutdown_connections 2024-03-25T18:28:57.040 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:28:57.037+0000 7fe3cf8a8640 1 -- 172.21.15.1:0/4041090277 wait complete. 2024-03-25T18:28:57.040 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:28:57.037+0000 7fe3cf8a8640 1 librados: shutdown 2024-03-25T18:28:57.050 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T18:29:03.858 DEBUG:teuthology.orchestra.run.smithi001:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T18:29:03.969 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:29:03.967+0000 7f198d4fb640 10 monclient: get_monmap_and_config 2024-03-25T18:29:03.969 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:29:03.967+0000 7f198d4fb640 10 monclient: build_initial_monmap 2024-03-25T18:29:03.969 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:29:03.967+0000 7f198d4fb640 10 monclient: monmap: 2024-03-25T18:29:03.969 INFO:teuthology.misc.health.smithi001.stderr:epoch 0 2024-03-25T18:29:03.969 INFO:teuthology.misc.health.smithi001.stderr:fsid c8ea72cf-9e99-4383-9b52-6c23d365fdfc 2024-03-25T18:29:03.970 INFO:teuthology.misc.health.smithi001.stderr:last_changed 2024-03-25T18:29:03.969069+0000 2024-03-25T18:29:03.970 INFO:teuthology.misc.health.smithi001.stderr:created 2024-03-25T18:29:03.969069+0000 2024-03-25T18:29:03.970 INFO:teuthology.misc.health.smithi001.stderr:min_mon_release 0 (unknown) 2024-03-25T18:29:03.970 INFO:teuthology.misc.health.smithi001.stderr:election_strategy: 3 2024-03-25T18:29:03.970 INFO:teuthology.misc.health.smithi001.stderr:0: v1:172.21.15.1:6789/0 mon.noname-a 2024-03-25T18:29:03.970 INFO:teuthology.misc.health.smithi001.stderr:1: v1:172.21.15.115:6789/0 mon.noname-c 2024-03-25T18:29:03.970 INFO:teuthology.misc.health.smithi001.stderr:2: v1:172.21.15.1:6790/0 mon.noname-b 2024-03-25T18:29:03.970 INFO:teuthology.misc.health.smithi001.stderr: 2024-03-25T18:29:03.972 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:29:03.968+0000 7f198d4fb640 1 Processor -- start 2024-03-25T18:29:03.972 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:29:03.968+0000 7f198d4fb640 1 -- start start 2024-03-25T18:29:03.972 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:29:03.968+0000 7f198d4fb640 10 monclient: init 2024-03-25T18:29:03.972 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:29:03.969+0000 7f198d4fb640 10 monclient: _reopen_session rank -1 2024-03-25T18:29:03.972 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:29:03.969+0000 7f198d4fb640 10 monclient: _add_conns ranks=[2,0,1] 2024-03-25T18:29:03.972 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:29:03.969+0000 7f198d4fb640 10 monclient(hunting): picked mon.noname-b con 0x7f19880b4e70 addr v1:172.21.15.1:6790/0 2024-03-25T18:29:03.973 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:29:03.969+0000 7f198d4fb640 10 monclient(hunting): picked mon.noname-a con 0x7f19880b44b0 addr v1:172.21.15.1:6789/0 2024-03-25T18:29:03.973 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:29:03.969+0000 7f198d4fb640 10 monclient(hunting): picked mon.noname-c con 0x7f19880b1b00 addr v1:172.21.15.115:6789/0 2024-03-25T18:29:03.973 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:29:03.969+0000 7f198d4fb640 1 -- --> v1:172.21.15.1:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f19880a96f0 con 0x7f19880b44b0 2024-03-25T18:29:03.973 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:29:03.969+0000 7f198d4fb640 1 -- --> v1:172.21.15.115:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f19880ab0e0 con 0x7f19880b1b00 2024-03-25T18:29:03.973 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:29:03.969+0000 7f198d4fb640 1 -- --> v1:172.21.15.1:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1988145f70 con 0x7f19880b4e70 2024-03-25T18:29:03.973 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:29:03.969+0000 7f198d4fb640 10 monclient(hunting): _renew_subs 2024-03-25T18:29:03.973 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:29:03.969+0000 7f198d4fb640 10 monclient(hunting): authenticate will time out at 7624.440380s 2024-03-25T18:29:03.973 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:29:03.971+0000 7f19867fc640 1 --1- >> v1:172.21.15.1:6789/0 conn(0x7f19880b44b0 0x7f1988145860 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.1:6789/0 says I am v1:172.21.15.1:41550/0 (socket says 172.21.15.1:41550) 2024-03-25T18:29:03.973 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:29:03.971+0000 7f19867fc640 1 -- 172.21.15.1:0/2846384540 learned_addr learned my addr 172.21.15.1:0/2846384540 (peer_addr_for_me v1:172.21.15.1:0/0) 2024-03-25T18:29:03.988 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:29:03.987+0000 7f1985ffb640 1 -- 172.21.15.1:0/2846384540 <== mon.0 v1:172.21.15.1:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3827912382 0 0) 0x7f19880a96f0 con 0x7f19880b44b0 2024-03-25T18:29:03.988 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:29:03.987+0000 7f1985ffb640 10 monclient(hunting): _negotiate 2024-03-25T18:29:03.988 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:29:03.987+0000 7f1985ffb640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:29:03.988 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:29:03.987+0000 7f1985ffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:29:03.989 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:29:03.987+0000 7f1985ffb640 10 monclient(hunting): my global_id is 4323 2024-03-25T18:29:03.989 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:29:03.987+0000 7f1985ffb640 1 -- 172.21.15.1:0/2846384540 --> v1:172.21.15.1:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f1970003520 con 0x7f19880b44b0 2024-03-25T18:29:04.002 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:29:03.998+0000 7f1985ffb640 1 -- 172.21.15.1:0/2846384540 <== mon.2 v1:172.21.15.1:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2740718981 0 0) 0x7f1988145f70 con 0x7f19880b4e70 2024-03-25T18:29:04.002 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:29:03.998+0000 7f1985ffb640 10 monclient(hunting): _negotiate 2024-03-25T18:29:04.002 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:29:03.998+0000 7f1985ffb640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:29:04.002 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:29:03.998+0000 7f1985ffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:29:04.002 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:29:03.998+0000 7f1985ffb640 10 monclient(hunting): my global_id is 4325 2024-03-25T18:29:04.002 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:29:03.998+0000 7f1985ffb640 1 -- 172.21.15.1:0/2846384540 --> v1:172.21.15.1:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f19880a96f0 con 0x7f19880b4e70 2024-03-25T18:29:04.003 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:29:03.998+0000 7f1985ffb640 1 -- 172.21.15.1:0/2846384540 <== mon.1 v1:172.21.15.115:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4239793192 0 0) 0x7f19880ab0e0 con 0x7f19880b1b00 2024-03-25T18:29:04.003 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:29:03.998+0000 7f1985ffb640 10 monclient(hunting): _negotiate 2024-03-25T18:29:04.003 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:29:03.998+0000 7f1985ffb640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:29:04.003 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:29:03.998+0000 7f1985ffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:29:04.003 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:29:03.998+0000 7f1985ffb640 10 monclient(hunting): my global_id is 4318 2024-03-25T18:29:04.003 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:29:03.998+0000 7f1985ffb640 1 -- 172.21.15.1:0/2846384540 --> v1:172.21.15.115:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f1988145f70 con 0x7f19880b1b00 2024-03-25T18:29:04.003 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:29:03.999+0000 7f1985ffb640 1 -- 172.21.15.1:0/2846384540 <== mon.0 v1:172.21.15.1:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 4200829707 0 0) 0x7f1970003520 con 0x7f19880b44b0 2024-03-25T18:29:04.003 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:29:03.999+0000 7f1985ffb640 1 -- 172.21.15.1:0/2846384540 >> v1:172.21.15.1:6790/0 conn(0x7f19880b4e70 legacy=0x7f198813f940 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:29:04.004 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:29:03.999+0000 7f1985ffb640 1 -- 172.21.15.1:0/2846384540 >> v1:172.21.15.115:6789/0 conn(0x7f19880b1b00 legacy=0x7f198814dc00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:29:04.004 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:29:03.999+0000 7f1985ffb640 10 monclient: _finish_hunting 0 2024-03-25T18:29:04.004 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:29:03.999+0000 7f1985ffb640 1 monclient: found mon.noname-a 2024-03-25T18:29:04.004 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:29:03.999+0000 7f1985ffb640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:29:04.004 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:29:03.999+0000 7f1985ffb640 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.1:6789/0 2024-03-25T18:29:04.004 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:29:03.999+0000 7f1985ffb640 1 -- 172.21.15.1:0/2846384540 --> v1:172.21.15.1:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f19880a9b80 con 0x7f19880b44b0 2024-03-25T18:29:04.004 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:29:03.999+0000 7f1985ffb640 10 monclient: _finish_auth 0 2024-03-25T18:29:04.004 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:29:03.999+0000 7f1985ffb640 10 monclient: _check_auth_tickets 2024-03-25T18:29:04.004 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:29:03.999+0000 7f1985ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:29:04.005 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:29:03.999+0000 7f1985ffb640 1 -- 172.21.15.1:0/2846384540 <== mon.0 v1:172.21.15.1:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1586965025 0 0) 0x7f197c001690 con 0x7f19880b44b0 2024-03-25T18:29:04.005 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:29:03.999+0000 7f1985ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:29:04.005 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:29:03.999+0000 7f1985ffb640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-03-25T18:29:04.005 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:29:03.999+0000 7f1985ffb640 10 monclient: dump: 2024-03-25T18:29:04.005 INFO:teuthology.misc.health.smithi001.stderr:epoch 1 2024-03-25T18:29:04.005 INFO:teuthology.misc.health.smithi001.stderr:fsid c8ea72cf-9e99-4383-9b52-6c23d365fdfc 2024-03-25T18:29:04.005 INFO:teuthology.misc.health.smithi001.stderr:last_changed 2024-03-25T18:25:50.510519+0000 2024-03-25T18:29:04.005 INFO:teuthology.misc.health.smithi001.stderr:created 2024-03-25T18:25:50.510519+0000 2024-03-25T18:29:04.005 INFO:teuthology.misc.health.smithi001.stderr:min_mon_release 19 (squid) 2024-03-25T18:29:04.006 INFO:teuthology.misc.health.smithi001.stderr:election_strategy: 3 2024-03-25T18:29:04.006 INFO:teuthology.misc.health.smithi001.stderr:0: v1:172.21.15.1:6789/0 mon.a 2024-03-25T18:29:04.006 INFO:teuthology.misc.health.smithi001.stderr:1: v1:172.21.15.115:6789/0 mon.b 2024-03-25T18:29:04.006 INFO:teuthology.misc.health.smithi001.stderr:2: v1:172.21.15.1:6790/0 mon.c 2024-03-25T18:29:04.006 INFO:teuthology.misc.health.smithi001.stderr: 2024-03-25T18:29:04.006 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:29:03.999+0000 7f198d4fb640 5 monclient: authenticate success, global_id 4323 2024-03-25T18:29:04.006 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:29:03.999+0000 7f198d4fb640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T18:29:04.010 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:29:04.008+0000 7f1985ffb640 1 -- 172.21.15.1:0/2846384540 <== mon.0 v1:172.21.15.1:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f197c0017f0 con 0x7f19880b44b0 2024-03-25T18:29:04.010 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:29:04.008+0000 7f1985ffb640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:29:04.010 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:29:04.008+0000 7f1985ffb640 1 -- 172.21.15.1:0/2846384540 <== mon.0 v1:172.21.15.1:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1586965025 0 0) 0x7f197c001c00 con 0x7f19880b44b0 2024-03-25T18:29:04.010 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:29:04.008+0000 7f198d4fb640 10 monclient: get_monmap_and_config success 2024-03-25T18:29:04.010 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:29:04.008+0000 7f198d4fb640 4 set_mon_vals no callback set 2024-03-25T18:29:04.010 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:29:04.008+0000 7f1985ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:29:04.010 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:29:04.008+0000 7f1985ffb640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-25T18:29:04.010 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:29:04.008+0000 7f1985ffb640 10 monclient: dump: 2024-03-25T18:29:04.010 INFO:teuthology.misc.health.smithi001.stderr:epoch 1 2024-03-25T18:29:04.011 INFO:teuthology.misc.health.smithi001.stderr:fsid c8ea72cf-9e99-4383-9b52-6c23d365fdfc 2024-03-25T18:29:04.011 INFO:teuthology.misc.health.smithi001.stderr:last_changed 2024-03-25T18:25:50.510519+0000 2024-03-25T18:29:04.011 INFO:teuthology.misc.health.smithi001.stderr:created 2024-03-25T18:25:50.510519+0000 2024-03-25T18:29:04.011 INFO:teuthology.misc.health.smithi001.stderr:min_mon_release 19 (squid) 2024-03-25T18:29:04.011 INFO:teuthology.misc.health.smithi001.stderr:election_strategy: 3 2024-03-25T18:29:04.011 INFO:teuthology.misc.health.smithi001.stderr:0: v1:172.21.15.1:6789/0 mon.a 2024-03-25T18:29:04.011 INFO:teuthology.misc.health.smithi001.stderr:1: v1:172.21.15.115:6789/0 mon.b 2024-03-25T18:29:04.011 INFO:teuthology.misc.health.smithi001.stderr:2: v1:172.21.15.1:6790/0 mon.c 2024-03-25T18:29:04.011 INFO:teuthology.misc.health.smithi001.stderr: 2024-03-25T18:29:04.013 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:29:04.009+0000 7f198d4fb640 10 monclient: shutdown 2024-03-25T18:29:04.013 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:29:04.009+0000 7f198d4fb640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:29:04.013 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:29:04.009+0000 7f198d4fb640 1 -- 172.21.15.1:0/2846384540 >> v1:172.21.15.1:6789/0 conn(0x7f19880b44b0 legacy=0x7f1988145860 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:29:04.013 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:29:04.009+0000 7f198d4fb640 1 -- 172.21.15.1:0/2846384540 shutdown_connections 2024-03-25T18:29:04.013 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:29:04.009+0000 7f198d4fb640 1 -- 172.21.15.1:0/2846384540 >> 172.21.15.1:0/2846384540 conn(0x7f19880617a0 msgr2=0x7f1988061bb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:29:04.014 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:29:04.009+0000 7f198d4fb640 1 -- 172.21.15.1:0/2846384540 shutdown_connections 2024-03-25T18:29:04.014 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:29:04.009+0000 7f198d4fb640 1 -- 172.21.15.1:0/2846384540 wait complete. 2024-03-25T18:29:04.014 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:29:04.010+0000 7f198d4fb640 10 monclient: build_initial_monmap 2024-03-25T18:29:04.014 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:29:04.010+0000 7f198d4fb640 10 monclient: monmap: 2024-03-25T18:29:04.014 INFO:teuthology.misc.health.smithi001.stderr:epoch 0 2024-03-25T18:29:04.014 INFO:teuthology.misc.health.smithi001.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T18:29:04.014 INFO:teuthology.misc.health.smithi001.stderr:last_changed 0.000000 2024-03-25T18:29:04.014 INFO:teuthology.misc.health.smithi001.stderr:created 0.000000 2024-03-25T18:29:04.014 INFO:teuthology.misc.health.smithi001.stderr:min_mon_release 0 (unknown) 2024-03-25T18:29:04.014 INFO:teuthology.misc.health.smithi001.stderr:election_strategy: 1 2024-03-25T18:29:04.014 INFO:teuthology.misc.health.smithi001.stderr:0: v1:172.21.15.1:6789/0 mon.noname-a 2024-03-25T18:29:04.014 INFO:teuthology.misc.health.smithi001.stderr:1: v1:172.21.15.115:6789/0 mon.noname-b 2024-03-25T18:29:04.014 INFO:teuthology.misc.health.smithi001.stderr:2: v1:172.21.15.1:6790/0 mon.noname-c 2024-03-25T18:29:04.014 INFO:teuthology.misc.health.smithi001.stderr: 2024-03-25T18:29:04.015 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:29:04.010+0000 7f198d4fb640 1 librados: starting msgr at 2024-03-25T18:29:04.015 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:29:04.010+0000 7f198d4fb640 1 librados: starting objecter 2024-03-25T18:29:04.015 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:29:04.010+0000 7f198d4fb640 1 Processor -- start 2024-03-25T18:29:04.015 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:29:04.011+0000 7f198d4fb640 1 -- start start 2024-03-25T18:29:04.015 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:29:04.011+0000 7f198d4fb640 1 librados: setting wanted keys 2024-03-25T18:29:04.015 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:29:04.011+0000 7f198d4fb640 1 librados: calling monclient init 2024-03-25T18:29:04.015 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:29:04.011+0000 7f198d4fb640 10 monclient: init 2024-03-25T18:29:04.015 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:29:04.011+0000 7f198d4fb640 10 monclient: _reopen_session rank -1 2024-03-25T18:29:04.015 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:29:04.011+0000 7f198d4fb640 10 monclient: _add_conns ranks=[1,0,2] 2024-03-25T18:29:04.015 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:29:04.011+0000 7f198d4fb640 10 monclient(hunting): picked mon.noname-b con 0x7f19880b1b00 addr v1:172.21.15.115:6789/0 2024-03-25T18:29:04.015 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:29:04.011+0000 7f198d4fb640 10 monclient(hunting): picked mon.noname-a con 0x7f19880b44b0 addr v1:172.21.15.1:6789/0 2024-03-25T18:29:04.015 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:29:04.011+0000 7f198d4fb640 10 monclient(hunting): picked mon.noname-c con 0x7f19880b4e70 addr v1:172.21.15.1:6790/0 2024-03-25T18:29:04.015 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:29:04.011+0000 7f198d4fb640 1 -- --> v1:172.21.15.1:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1988137f00 con 0x7f19880b44b0 2024-03-25T18:29:04.015 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:29:04.011+0000 7f198d4fb640 1 -- --> v1:172.21.15.115:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1988136b00 con 0x7f19880b1b00 2024-03-25T18:29:04.015 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:29:04.011+0000 7f198d4fb640 1 -- --> v1:172.21.15.1:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f19881d7b70 con 0x7f19880b4e70 2024-03-25T18:29:04.016 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:29:04.011+0000 7f198d4fb640 10 monclient(hunting): _renew_subs 2024-03-25T18:29:04.016 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:29:04.011+0000 7f198d4fb640 10 monclient(hunting): authenticate will time out at 7624.482364s 2024-03-25T18:29:04.020 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:29:04.018+0000 7f1986ffd640 1 --1- >> v1:172.21.15.115:6789/0 conn(0x7f19880b1b00 0x7f19881363f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.115:6789/0 says I am v1:172.21.15.1:40280/0 (socket says 172.21.15.1:40280) 2024-03-25T18:29:04.020 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:29:04.019+0000 7f1986ffd640 1 -- 172.21.15.1:0/318738530 learned_addr learned my addr 172.21.15.1:0/318738530 (peer_addr_for_me v1:172.21.15.1:0/0) 2024-03-25T18:29:04.023 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:29:04.021+0000 7f196f7fe640 10 client.?.objecter ms_handle_connect 0x7f19880b1b00 2024-03-25T18:29:04.023 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:29:04.021+0000 7f196f7fe640 10 client.?.objecter resend_mon_ops 2024-03-25T18:29:04.030 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:29:04.028+0000 7f196f7fe640 1 -- 172.21.15.1:0/318738530 <== mon.1 v1:172.21.15.115:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2344862787 0 0) 0x7f1988136b00 con 0x7f19880b1b00 2024-03-25T18:29:04.030 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:29:04.028+0000 7f196f7fe640 10 monclient(hunting): _negotiate 2024-03-25T18:29:04.030 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:29:04.028+0000 7f196f7fe640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:29:04.030 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:29:04.028+0000 7f196f7fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:29:04.030 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:29:04.028+0000 7f196f7fe640 10 monclient(hunting): my global_id is 4321 2024-03-25T18:29:04.030 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:29:04.028+0000 7f196f7fe640 1 -- 172.21.15.1:0/318738530 --> v1:172.21.15.115:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f1974004eb0 con 0x7f19880b1b00 2024-03-25T18:29:04.030 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:29:04.028+0000 7f196f7fe640 10 client.?.objecter ms_handle_connect 0x7f19880b4e70 2024-03-25T18:29:04.030 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:29:04.028+0000 7f196f7fe640 10 client.?.objecter resend_mon_ops 2024-03-25T18:29:04.035 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:29:04.033+0000 7f196f7fe640 10 client.?.objecter ms_handle_connect 0x7f19880b44b0 2024-03-25T18:29:04.035 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:29:04.033+0000 7f196f7fe640 10 client.?.objecter resend_mon_ops 2024-03-25T18:29:04.043 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:29:04.041+0000 7f196f7fe640 1 -- 172.21.15.1:0/318738530 <== mon.2 v1:172.21.15.1:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3320682638 0 0) 0x7f19881d7b70 con 0x7f19880b4e70 2024-03-25T18:29:04.043 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:29:04.041+0000 7f196f7fe640 10 monclient(hunting): _negotiate 2024-03-25T18:29:04.043 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:29:04.041+0000 7f196f7fe640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:29:04.044 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:29:04.041+0000 7f196f7fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:29:04.044 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:29:04.041+0000 7f196f7fe640 10 monclient(hunting): my global_id is 4328 2024-03-25T18:29:04.044 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:29:04.041+0000 7f196f7fe640 1 -- 172.21.15.1:0/318738530 --> v1:172.21.15.1:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f1988136b00 con 0x7f19880b4e70 2024-03-25T18:29:04.044 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:29:04.042+0000 7f196f7fe640 1 -- 172.21.15.1:0/318738530 <== mon.1 v1:172.21.15.115:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3631001917 0 0) 0x7f1974004eb0 con 0x7f19880b1b00 2024-03-25T18:29:04.044 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:29:04.042+0000 7f196f7fe640 1 -- 172.21.15.1:0/318738530 --> v1:172.21.15.115:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f19881d7b70 con 0x7f19880b1b00 2024-03-25T18:29:04.044 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:29:04.042+0000 7f196f7fe640 1 -- 172.21.15.1:0/318738530 <== mon.1 v1:172.21.15.115:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1586965025 0 0) 0x7f1978001990 con 0x7f19880b1b00 2024-03-25T18:29:04.044 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:29:04.042+0000 7f196f7fe640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:29:04.044 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:29:04.042+0000 7f196f7fe640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-03-25T18:29:04.044 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:29:04.042+0000 7f196f7fe640 10 monclient(hunting): dump: 2024-03-25T18:29:04.044 INFO:teuthology.misc.health.smithi001.stderr:epoch 1 2024-03-25T18:29:04.044 INFO:teuthology.misc.health.smithi001.stderr:fsid c8ea72cf-9e99-4383-9b52-6c23d365fdfc 2024-03-25T18:29:04.044 INFO:teuthology.misc.health.smithi001.stderr:last_changed 2024-03-25T18:25:50.510519+0000 2024-03-25T18:29:04.044 INFO:teuthology.misc.health.smithi001.stderr:created 2024-03-25T18:25:50.510519+0000 2024-03-25T18:29:04.044 INFO:teuthology.misc.health.smithi001.stderr:min_mon_release 19 (squid) 2024-03-25T18:29:04.045 INFO:teuthology.misc.health.smithi001.stderr:election_strategy: 3 2024-03-25T18:29:04.045 INFO:teuthology.misc.health.smithi001.stderr:0: v1:172.21.15.1:6789/0 mon.a 2024-03-25T18:29:04.045 INFO:teuthology.misc.health.smithi001.stderr:1: v1:172.21.15.115:6789/0 mon.b 2024-03-25T18:29:04.045 INFO:teuthology.misc.health.smithi001.stderr:2: v1:172.21.15.1:6790/0 mon.c 2024-03-25T18:29:04.045 INFO:teuthology.misc.health.smithi001.stderr: 2024-03-25T18:29:04.045 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:29:04.042+0000 7f196f7fe640 10 monclient(hunting): _finish_auth 0 2024-03-25T18:29:04.045 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:29:04.042+0000 7f196f7fe640 1 -- 172.21.15.1:0/318738530 <== mon.0 v1:172.21.15.1:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2010352347 0 0) 0x7f1988137f00 con 0x7f19880b44b0 2024-03-25T18:29:04.045 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:29:04.042+0000 7f196f7fe640 10 monclient(hunting): _negotiate 2024-03-25T18:29:04.045 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:29:04.042+0000 7f196f7fe640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:29:04.045 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:29:04.042+0000 7f196f7fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:29:04.046 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:29:04.042+0000 7f196f7fe640 10 monclient(hunting): my global_id is 4326 2024-03-25T18:29:04.046 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:29:04.042+0000 7f196f7fe640 1 -- 172.21.15.1:0/318738530 --> v1:172.21.15.1:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f1974004eb0 con 0x7f19880b44b0 2024-03-25T18:29:04.058 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:29:04.056+0000 7f196f7fe640 1 -- 172.21.15.1:0/318738530 <== mon.1 v1:172.21.15.115:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 88108079 0 0) 0x7f19881d7b70 con 0x7f19880b1b00 2024-03-25T18:29:04.058 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:29:04.056+0000 7f196f7fe640 1 -- 172.21.15.1:0/318738530 >> v1:172.21.15.1:6790/0 conn(0x7f19880b4e70 legacy=0x7f19881d4c50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:29:04.059 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:29:04.056+0000 7f196f7fe640 1 -- 172.21.15.1:0/318738530 >> v1:172.21.15.1:6789/0 conn(0x7f19880b44b0 legacy=0x7f19881d8090 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:29:04.059 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:29:04.056+0000 7f196f7fe640 10 monclient: _finish_hunting 0 2024-03-25T18:29:04.059 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:29:04.056+0000 7f196f7fe640 1 monclient: found mon.b 2024-03-25T18:29:04.059 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:29:04.056+0000 7f196f7fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:29:04.059 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:29:04.056+0000 7f196f7fe640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.115:6789/0 2024-03-25T18:29:04.059 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:29:04.056+0000 7f196f7fe640 1 -- 172.21.15.1:0/318738530 --> v1:172.21.15.115:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f19881d87a0 con 0x7f19880b1b00 2024-03-25T18:29:04.059 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:29:04.056+0000 7f196f7fe640 10 monclient: _finish_auth 0 2024-03-25T18:29:04.059 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:29:04.056+0000 7f196f7fe640 10 monclient: _check_auth_tickets 2024-03-25T18:29:04.059 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:29:04.056+0000 7f196f7fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:29:04.059 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:29:04.056+0000 7f198d4fb640 5 monclient: authenticate success, global_id 4321 2024-03-25T18:29:04.059 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:29:04.056+0000 7f198d4fb640 10 monclient: _renew_subs 2024-03-25T18:29:04.059 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:29:04.056+0000 7f198d4fb640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.115:6789/0 2024-03-25T18:29:04.059 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:29:04.056+0000 7f198d4fb640 1 -- 172.21.15.1:0/318738530 --> v1:172.21.15.115:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f19881d5360 con 0x7f19880b1b00 2024-03-25T18:29:04.059 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:29:04.057+0000 7f198d4fb640 10 client.4321.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T18:29:04.059 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:29:04.057+0000 7f198d4fb640 10 monclient: _renew_subs 2024-03-25T18:29:04.060 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:29:04.057+0000 7f198d4fb640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.115:6789/0 2024-03-25T18:29:04.060 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:29:04.057+0000 7f198d4fb640 1 -- 172.21.15.1:0/318738530 --> v1:172.21.15.115:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f19881d4720 con 0x7f19880b1b00 2024-03-25T18:29:04.060 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:29:04.057+0000 7f198d4fb640 1 librados: init done 2024-03-25T18:29:04.060 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:29:04.057+0000 7f198d4fb640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T18:29:04.060 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:29:04.058+0000 7f198d4fb640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:29:04.060 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:29:04.058+0000 7f198d4fb640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.115:6789/0 2024-03-25T18:29:04.060 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:29:04.058+0000 7f198d4fb640 1 -- 172.21.15.1:0/318738530 --> v1:172.21.15.115:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f19881d79d0 con 0x7f19880b1b00 2024-03-25T18:29:04.062 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:29:04.060+0000 7f196f7fe640 1 -- 172.21.15.1:0/318738530 <== mon.1 v1:172.21.15.115:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f19780035f0 con 0x7f19880b1b00 2024-03-25T18:29:04.062 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:29:04.060+0000 7f196f7fe640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:29:04.062 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:29:04.060+0000 7f19857fa640 4 set_mon_vals no callback set 2024-03-25T18:29:04.335 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:29:04.333+0000 7f196f7fe640 1 -- 172.21.15.1:0/318738530 <== mon.1 v1:172.21.15.115:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1586965025 0 0) 0x7f1978003820 con 0x7f19880b1b00 2024-03-25T18:29:04.335 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:29:04.333+0000 7f196f7fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:29:04.335 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:29:04.333+0000 7f196f7fe640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-25T18:29:04.335 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:29:04.333+0000 7f196f7fe640 10 monclient: dump: 2024-03-25T18:29:04.335 INFO:teuthology.misc.health.smithi001.stderr:epoch 1 2024-03-25T18:29:04.335 INFO:teuthology.misc.health.smithi001.stderr:fsid c8ea72cf-9e99-4383-9b52-6c23d365fdfc 2024-03-25T18:29:04.335 INFO:teuthology.misc.health.smithi001.stderr:last_changed 2024-03-25T18:25:50.510519+0000 2024-03-25T18:29:04.335 INFO:teuthology.misc.health.smithi001.stderr:created 2024-03-25T18:25:50.510519+0000 2024-03-25T18:29:04.335 INFO:teuthology.misc.health.smithi001.stderr:min_mon_release 19 (squid) 2024-03-25T18:29:04.336 INFO:teuthology.misc.health.smithi001.stderr:election_strategy: 3 2024-03-25T18:29:04.336 INFO:teuthology.misc.health.smithi001.stderr:0: v1:172.21.15.1:6789/0 mon.a 2024-03-25T18:29:04.336 INFO:teuthology.misc.health.smithi001.stderr:1: v1:172.21.15.115:6789/0 mon.b 2024-03-25T18:29:04.336 INFO:teuthology.misc.health.smithi001.stderr:2: v1:172.21.15.1:6790/0 mon.c 2024-03-25T18:29:04.336 INFO:teuthology.misc.health.smithi001.stderr: 2024-03-25T18:29:04.337 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:29:04.334+0000 7f196f7fe640 1 -- 172.21.15.1:0/318738530 <== mon.1 v1:172.21.15.115:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 21787975 0 0) 0x7f1978003a00 con 0x7f19880b1b00 2024-03-25T18:29:04.337 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:29:04.334+0000 7f19867fc640 10 monclient: get_auth_request con 0x7f197406de20 auth_method 0 2024-03-25T18:29:04.337 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:29:04.335+0000 7f196f7fe640 10 client.4321.objecter ms_handle_connect 0x7f197406de20 2024-03-25T18:29:04.339 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:29:04.337+0000 7f196f7fe640 1 -- 172.21.15.1:0/318738530 <== mon.1 v1:172.21.15.115:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 996254230 0 0) 0x7f19780546c0 con 0x7f19880b1b00 2024-03-25T18:29:04.339 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:29:04.337+0000 7f196f7fe640 10 client.4321.objecter ms_dispatch 0x7f1988000b90 osd_map(10..10 src has 1..10) v4 2024-03-25T18:29:04.339 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:29:04.337+0000 7f196f7fe640 3 client.4321.objecter handle_osd_map got epochs [10,10] > 0 2024-03-25T18:29:04.339 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:29:04.337+0000 7f196f7fe640 3 client.4321.objecter handle_osd_map decoding full epoch 10 2024-03-25T18:29:04.340 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:29:04.338+0000 7f196f7fe640 20 client.4321.objecter dump_active .. 0 homeless 2024-03-25T18:29:04.342 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:29:04.340+0000 7f196f7fe640 1 -- 172.21.15.1:0/318738530 <== mon.1 v1:172.21.15.115:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f19780501d0 con 0x7f19880b1b00 2024-03-25T18:29:04.342 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:29:04.341+0000 7f196f7fe640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:29:04.342 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:29:04.341+0000 7f196f7fe640 10 monclient: _finish_command 1 = system:0 2024-03-25T18:29:04.594 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:29:04.592+0000 7f198d4fb640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:29:04.594 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:29:04.592+0000 7f198d4fb640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:29:04.594 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:29:04.592+0000 7f198d4fb640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.115:6789/0 2024-03-25T18:29:04.594 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:29:04.592+0000 7f198d4fb640 1 -- 172.21.15.1:0/318738530 --> v1:172.21.15.115:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f19881d7d00 con 0x7f19880b1b00 2024-03-25T18:29:04.736 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:29:04.734+0000 7f196f7fe640 1 -- 172.21.15.1:0/318738530 <== mon.1 v1:172.21.15.115:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 4092213348) 0x7f1978052100 con 0x7f19880b1b00 2024-03-25T18:29:04.753 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:29:04.734+0000 7f196f7fe640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:29:04.753 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:29:04.734+0000 7f196f7fe640 10 monclient: _finish_command 2 = system:0 2024-03-25T18:29:04.754 INFO:teuthology.misc.health.smithi001.stdout: 2024-03-25T18:29:04.754 INFO:teuthology.misc.health.smithi001.stdout:{"epoch":10,"fsid":"c8ea72cf-9e99-4383-9b52-6c23d365fdfc","created":"2024-03-25T18:26:38.856638+0000","modified":"2024-03-25T18:26:44.224208+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T18:26:44.224208+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"b9f34922-3a84-49be-8426-807c1eceaefc","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"42bf306f-55e3-4634-9874-8b4b73ac6644","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"292cb23a-18a9-4207-a7cc-4d5727feaec0","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"45e0e9ab-898d-4673-9590-b4117bc5e84a","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"831a5a09-73ab-4a61-9872-088e76bdbc66","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"88cd903d-3b55-4840-9396-6e3cfe005282","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"89db7355-bc9d-46f4-8ab0-e8d9e744e026","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"cac47bee-0914-4f08-896f-348838882866","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T18:29:04.755 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:29:04.736+0000 7f198d4fb640 10 librados: watch_flush enter 2024-03-25T18:29:04.755 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:29:04.737+0000 7f198d4fb640 10 librados: watch_flush exit 2024-03-25T18:29:04.755 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:29:04.737+0000 7f198d4fb640 20 client.4321.objecter shutdown clearing up homeless session... 2024-03-25T18:29:04.755 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:29:04.737+0000 7f198d4fb640 10 client.4321.objecter successfully canceled tick 2024-03-25T18:29:04.756 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:29:04.737+0000 7f198d4fb640 1 -- 172.21.15.1:0/318738530 >> v1:172.21.15.115:6800/4237379176 conn(0x7f197406de20 legacy=0x7f19740702e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:29:04.756 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:29:04.737+0000 7f198d4fb640 10 monclient: shutdown 2024-03-25T18:29:04.756 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:29:04.737+0000 7f198d4fb640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:29:04.756 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:29:04.737+0000 7f198d4fb640 1 -- 172.21.15.1:0/318738530 >> v1:172.21.15.115:6789/0 conn(0x7f19880b1b00 legacy=0x7f19881363f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:29:04.756 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:29:04.737+0000 7f198d4fb640 1 -- 172.21.15.1:0/318738530 shutdown_connections 2024-03-25T18:29:04.756 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:29:04.737+0000 7f198d4fb640 1 -- 172.21.15.1:0/318738530 >> 172.21.15.1:0/318738530 conn(0x7f19880617a0 msgr2=0x7f19880b30a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:29:04.756 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:29:04.737+0000 7f198d4fb640 1 -- 172.21.15.1:0/318738530 shutdown_connections 2024-03-25T18:29:04.756 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:29:04.737+0000 7f198d4fb640 1 -- 172.21.15.1:0/318738530 wait complete. 2024-03-25T18:29:04.756 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:29:04.737+0000 7f198d4fb640 1 librados: shutdown 2024-03-25T18:29:04.757 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T18:29:11.563 DEBUG:teuthology.orchestra.run.smithi001:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T18:29:11.680 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:29:11.678+0000 7f3d3e0b3640 10 monclient: get_monmap_and_config 2024-03-25T18:29:11.680 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:29:11.678+0000 7f3d3e0b3640 10 monclient: build_initial_monmap 2024-03-25T18:29:11.680 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:29:11.678+0000 7f3d3e0b3640 10 monclient: monmap: 2024-03-25T18:29:11.680 INFO:teuthology.misc.health.smithi001.stderr:epoch 0 2024-03-25T18:29:11.680 INFO:teuthology.misc.health.smithi001.stderr:fsid c8ea72cf-9e99-4383-9b52-6c23d365fdfc 2024-03-25T18:29:11.681 INFO:teuthology.misc.health.smithi001.stderr:last_changed 2024-03-25T18:29:11.680164+0000 2024-03-25T18:29:11.681 INFO:teuthology.misc.health.smithi001.stderr:created 2024-03-25T18:29:11.680164+0000 2024-03-25T18:29:11.681 INFO:teuthology.misc.health.smithi001.stderr:min_mon_release 0 (unknown) 2024-03-25T18:29:11.681 INFO:teuthology.misc.health.smithi001.stderr:election_strategy: 3 2024-03-25T18:29:11.681 INFO:teuthology.misc.health.smithi001.stderr:0: v1:172.21.15.1:6789/0 mon.noname-a 2024-03-25T18:29:11.681 INFO:teuthology.misc.health.smithi001.stderr:1: v1:172.21.15.115:6789/0 mon.noname-c 2024-03-25T18:29:11.681 INFO:teuthology.misc.health.smithi001.stderr:2: v1:172.21.15.1:6790/0 mon.noname-b 2024-03-25T18:29:11.681 INFO:teuthology.misc.health.smithi001.stderr: 2024-03-25T18:29:11.683 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:29:11.679+0000 7f3d3e0b3640 1 Processor -- start 2024-03-25T18:29:11.683 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:29:11.679+0000 7f3d3e0b3640 1 -- start start 2024-03-25T18:29:11.683 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:29:11.679+0000 7f3d3e0b3640 10 monclient: init 2024-03-25T18:29:11.683 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:29:11.680+0000 7f3d3e0b3640 10 monclient: _reopen_session rank -1 2024-03-25T18:29:11.683 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:29:11.680+0000 7f3d3e0b3640 10 monclient: _add_conns ranks=[2,0,1] 2024-03-25T18:29:11.683 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:29:11.680+0000 7f3d3e0b3640 10 monclient(hunting): picked mon.noname-b con 0x7f3d380b4e70 addr v1:172.21.15.1:6790/0 2024-03-25T18:29:11.684 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:29:11.680+0000 7f3d3e0b3640 10 monclient(hunting): picked mon.noname-a con 0x7f3d380b44b0 addr v1:172.21.15.1:6789/0 2024-03-25T18:29:11.684 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:29:11.680+0000 7f3d3e0b3640 10 monclient(hunting): picked mon.noname-c con 0x7f3d380b1b00 addr v1:172.21.15.115:6789/0 2024-03-25T18:29:11.684 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:29:11.680+0000 7f3d3e0b3640 1 -- --> v1:172.21.15.1:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3d380a96f0 con 0x7f3d380b44b0 2024-03-25T18:29:11.684 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:29:11.680+0000 7f3d3e0b3640 1 -- --> v1:172.21.15.115:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3d380ab0e0 con 0x7f3d380b1b00 2024-03-25T18:29:11.684 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:29:11.680+0000 7f3d3e0b3640 1 -- --> v1:172.21.15.1:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3d38145f70 con 0x7f3d380b4e70 2024-03-25T18:29:11.684 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:29:11.680+0000 7f3d3e0b3640 10 monclient(hunting): _renew_subs 2024-03-25T18:29:11.684 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:29:11.680+0000 7f3d3e0b3640 10 monclient(hunting): authenticate will time out at 7632.150848s 2024-03-25T18:29:11.693 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:29:11.692+0000 7f3d377fe640 1 --1- >> v1:172.21.15.1:6790/0 conn(0x7f3d380b4e70 0x7f3d3813f940 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.1:6790/0 says I am v1:172.21.15.1:33410/0 (socket says 172.21.15.1:33410) 2024-03-25T18:29:11.693 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:29:11.692+0000 7f3d377fe640 1 -- 172.21.15.1:0/1544605118 learned_addr learned my addr 172.21.15.1:0/1544605118 (peer_addr_for_me v1:172.21.15.1:0/0) 2024-03-25T18:29:11.700 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:29:11.698+0000 7f3d367fc640 1 -- 172.21.15.1:0/1544605118 <== mon.2 v1:172.21.15.1:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 988182533 0 0) 0x7f3d38145f70 con 0x7f3d380b4e70 2024-03-25T18:29:11.700 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:29:11.698+0000 7f3d367fc640 10 monclient(hunting): _negotiate 2024-03-25T18:29:11.700 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:29:11.698+0000 7f3d367fc640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:29:11.700 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:29:11.698+0000 7f3d367fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:29:11.700 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:29:11.698+0000 7f3d367fc640 10 monclient(hunting): my global_id is 4331 2024-03-25T18:29:11.700 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:29:11.698+0000 7f3d367fc640 1 -- 172.21.15.1:0/1544605118 --> v1:172.21.15.1:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f3d20003520 con 0x7f3d380b4e70 2024-03-25T18:29:11.708 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:29:11.707+0000 7f3d367fc640 1 -- 172.21.15.1:0/1544605118 <== mon.1 v1:172.21.15.115:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1268821542 0 0) 0x7f3d380ab0e0 con 0x7f3d380b1b00 2024-03-25T18:29:11.708 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:29:11.707+0000 7f3d367fc640 10 monclient(hunting): _negotiate 2024-03-25T18:29:11.708 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:29:11.707+0000 7f3d367fc640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:29:11.709 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:29:11.707+0000 7f3d367fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:29:11.709 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:29:11.707+0000 7f3d367fc640 10 monclient(hunting): my global_id is 4324 2024-03-25T18:29:11.709 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:29:11.707+0000 7f3d367fc640 1 -- 172.21.15.1:0/1544605118 --> v1:172.21.15.115:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f3d38145f70 con 0x7f3d380b1b00 2024-03-25T18:29:11.713 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:29:11.711+0000 7f3d367fc640 1 -- 172.21.15.1:0/1544605118 <== mon.2 v1:172.21.15.1:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3425137835 0 0) 0x7f3d20003520 con 0x7f3d380b4e70 2024-03-25T18:29:11.713 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:29:11.711+0000 7f3d367fc640 1 -- 172.21.15.1:0/1544605118 >> v1:172.21.15.115:6789/0 conn(0x7f3d380b1b00 legacy=0x7f3d3814dc00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:29:11.713 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:29:11.711+0000 7f3d367fc640 1 -- 172.21.15.1:0/1544605118 >> v1:172.21.15.1:6789/0 conn(0x7f3d380b44b0 legacy=0x7f3d38145860 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:29:11.713 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:29:11.711+0000 7f3d367fc640 10 monclient: _finish_hunting 0 2024-03-25T18:29:11.713 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:29:11.711+0000 7f3d367fc640 1 monclient: found mon.noname-b 2024-03-25T18:29:11.713 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:29:11.711+0000 7f3d367fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:29:11.713 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:29:11.711+0000 7f3d367fc640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.1:6790/0 2024-03-25T18:29:11.713 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:29:11.711+0000 7f3d367fc640 1 -- 172.21.15.1:0/1544605118 --> v1:172.21.15.1:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3d380a9b80 con 0x7f3d380b4e70 2024-03-25T18:29:11.713 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:29:11.711+0000 7f3d367fc640 10 monclient: _finish_auth 0 2024-03-25T18:29:11.713 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:29:11.711+0000 7f3d367fc640 10 monclient: _check_auth_tickets 2024-03-25T18:29:11.714 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:29:11.711+0000 7f3d367fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:29:11.714 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:29:11.711+0000 7f3d367fc640 1 -- 172.21.15.1:0/1544605118 <== mon.2 v1:172.21.15.1:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1586965025 0 0) 0x7f3d28001690 con 0x7f3d380b4e70 2024-03-25T18:29:11.714 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:29:11.711+0000 7f3d3e0b3640 5 monclient: authenticate success, global_id 4331 2024-03-25T18:29:11.714 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:29:11.711+0000 7f3d3e0b3640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T18:29:11.714 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:29:11.711+0000 7f3d367fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:29:11.714 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:29:11.711+0000 7f3d367fc640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-03-25T18:29:11.714 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:29:11.711+0000 7f3d367fc640 10 monclient: dump: 2024-03-25T18:29:11.714 INFO:teuthology.misc.health.smithi001.stderr:epoch 1 2024-03-25T18:29:11.714 INFO:teuthology.misc.health.smithi001.stderr:fsid c8ea72cf-9e99-4383-9b52-6c23d365fdfc 2024-03-25T18:29:11.714 INFO:teuthology.misc.health.smithi001.stderr:last_changed 2024-03-25T18:25:50.510519+0000 2024-03-25T18:29:11.714 INFO:teuthology.misc.health.smithi001.stderr:created 2024-03-25T18:25:50.510519+0000 2024-03-25T18:29:11.714 INFO:teuthology.misc.health.smithi001.stderr:min_mon_release 19 (squid) 2024-03-25T18:29:11.714 INFO:teuthology.misc.health.smithi001.stderr:election_strategy: 3 2024-03-25T18:29:11.715 INFO:teuthology.misc.health.smithi001.stderr:0: v1:172.21.15.1:6789/0 mon.a 2024-03-25T18:29:11.715 INFO:teuthology.misc.health.smithi001.stderr:1: v1:172.21.15.115:6789/0 mon.b 2024-03-25T18:29:11.715 INFO:teuthology.misc.health.smithi001.stderr:2: v1:172.21.15.1:6790/0 mon.c 2024-03-25T18:29:11.715 INFO:teuthology.misc.health.smithi001.stderr: 2024-03-25T18:29:11.715 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:29:11.711+0000 7f3d3e0b3640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T18:29:11.717 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:29:11.715+0000 7f3d367fc640 1 -- 172.21.15.1:0/1544605118 <== mon.2 v1:172.21.15.1:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f3d280017f0 con 0x7f3d380b4e70 2024-03-25T18:29:11.717 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:29:11.715+0000 7f3d367fc640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:29:11.717 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:29:11.715+0000 7f3d367fc640 1 -- 172.21.15.1:0/1544605118 <== mon.2 v1:172.21.15.1:6790/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1586965025 0 0) 0x7f3d28001c00 con 0x7f3d380b4e70 2024-03-25T18:29:11.717 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:29:11.715+0000 7f3d367fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:29:11.717 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:29:11.715+0000 7f3d367fc640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-25T18:29:11.717 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:29:11.715+0000 7f3d367fc640 10 monclient: dump: 2024-03-25T18:29:11.717 INFO:teuthology.misc.health.smithi001.stderr:epoch 1 2024-03-25T18:29:11.717 INFO:teuthology.misc.health.smithi001.stderr:fsid c8ea72cf-9e99-4383-9b52-6c23d365fdfc 2024-03-25T18:29:11.718 INFO:teuthology.misc.health.smithi001.stderr:last_changed 2024-03-25T18:25:50.510519+0000 2024-03-25T18:29:11.718 INFO:teuthology.misc.health.smithi001.stderr:created 2024-03-25T18:25:50.510519+0000 2024-03-25T18:29:11.718 INFO:teuthology.misc.health.smithi001.stderr:min_mon_release 19 (squid) 2024-03-25T18:29:11.718 INFO:teuthology.misc.health.smithi001.stderr:election_strategy: 3 2024-03-25T18:29:11.718 INFO:teuthology.misc.health.smithi001.stderr:0: v1:172.21.15.1:6789/0 mon.a 2024-03-25T18:29:11.718 INFO:teuthology.misc.health.smithi001.stderr:1: v1:172.21.15.115:6789/0 mon.b 2024-03-25T18:29:11.718 INFO:teuthology.misc.health.smithi001.stderr:2: v1:172.21.15.1:6790/0 mon.c 2024-03-25T18:29:11.718 INFO:teuthology.misc.health.smithi001.stderr: 2024-03-25T18:29:11.718 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:29:11.715+0000 7f3d3e0b3640 10 monclient: get_monmap_and_config success 2024-03-25T18:29:11.718 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:29:11.715+0000 7f3d3e0b3640 4 set_mon_vals no callback set 2024-03-25T18:29:11.719 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:29:11.716+0000 7f3d3e0b3640 10 monclient: shutdown 2024-03-25T18:29:11.720 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:29:11.716+0000 7f3d3e0b3640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:29:11.720 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:29:11.716+0000 7f3d3e0b3640 1 -- 172.21.15.1:0/1544605118 >> v1:172.21.15.1:6790/0 conn(0x7f3d380b4e70 legacy=0x7f3d3813f940 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:29:11.720 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:29:11.716+0000 7f3d3e0b3640 1 -- 172.21.15.1:0/1544605118 shutdown_connections 2024-03-25T18:29:11.720 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:29:11.716+0000 7f3d3e0b3640 1 -- 172.21.15.1:0/1544605118 >> 172.21.15.1:0/1544605118 conn(0x7f3d380617a0 msgr2=0x7f3d38061bb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:29:11.720 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:29:11.716+0000 7f3d3e0b3640 1 -- 172.21.15.1:0/1544605118 shutdown_connections 2024-03-25T18:29:11.720 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:29:11.716+0000 7f3d3e0b3640 1 -- 172.21.15.1:0/1544605118 wait complete. 2024-03-25T18:29:11.720 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:29:11.717+0000 7f3d3e0b3640 10 monclient: build_initial_monmap 2024-03-25T18:29:11.720 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:29:11.717+0000 7f3d3e0b3640 10 monclient: monmap: 2024-03-25T18:29:11.720 INFO:teuthology.misc.health.smithi001.stderr:epoch 0 2024-03-25T18:29:11.720 INFO:teuthology.misc.health.smithi001.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T18:29:11.720 INFO:teuthology.misc.health.smithi001.stderr:last_changed 0.000000 2024-03-25T18:29:11.720 INFO:teuthology.misc.health.smithi001.stderr:created 0.000000 2024-03-25T18:29:11.720 INFO:teuthology.misc.health.smithi001.stderr:min_mon_release 0 (unknown) 2024-03-25T18:29:11.721 INFO:teuthology.misc.health.smithi001.stderr:election_strategy: 1 2024-03-25T18:29:11.721 INFO:teuthology.misc.health.smithi001.stderr:0: v1:172.21.15.1:6789/0 mon.noname-a 2024-03-25T18:29:11.721 INFO:teuthology.misc.health.smithi001.stderr:1: v1:172.21.15.115:6789/0 mon.noname-b 2024-03-25T18:29:11.721 INFO:teuthology.misc.health.smithi001.stderr:2: v1:172.21.15.1:6790/0 mon.noname-c 2024-03-25T18:29:11.721 INFO:teuthology.misc.health.smithi001.stderr: 2024-03-25T18:29:11.721 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:29:11.718+0000 7f3d3e0b3640 1 librados: starting msgr at 2024-03-25T18:29:11.721 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:29:11.718+0000 7f3d3e0b3640 1 librados: starting objecter 2024-03-25T18:29:11.721 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:29:11.718+0000 7f3d3e0b3640 1 Processor -- start 2024-03-25T18:29:11.721 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:29:11.718+0000 7f3d3e0b3640 1 -- start start 2024-03-25T18:29:11.721 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:29:11.718+0000 7f3d3e0b3640 1 librados: setting wanted keys 2024-03-25T18:29:11.721 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:29:11.718+0000 7f3d3e0b3640 1 librados: calling monclient init 2024-03-25T18:29:11.721 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:29:11.718+0000 7f3d3e0b3640 10 monclient: init 2024-03-25T18:29:11.723 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:29:11.719+0000 7f3d3e0b3640 10 monclient: _reopen_session rank -1 2024-03-25T18:29:11.723 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:29:11.719+0000 7f3d3e0b3640 10 monclient: _add_conns ranks=[1,2,0] 2024-03-25T18:29:11.723 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:29:11.719+0000 7f3d3e0b3640 10 monclient(hunting): picked mon.noname-b con 0x7f3d380b1b00 addr v1:172.21.15.115:6789/0 2024-03-25T18:29:11.723 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:29:11.719+0000 7f3d3e0b3640 10 monclient(hunting): picked mon.noname-c con 0x7f3d380b44b0 addr v1:172.21.15.1:6790/0 2024-03-25T18:29:11.723 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:29:11.719+0000 7f3d3e0b3640 10 monclient(hunting): picked mon.noname-a con 0x7f3d380b4e70 addr v1:172.21.15.1:6789/0 2024-03-25T18:29:11.723 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:29:11.719+0000 7f3d3e0b3640 1 -- --> v1:172.21.15.1:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3d381369d0 con 0x7f3d380b4e70 2024-03-25T18:29:11.723 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:29:11.720+0000 7f3d3e0b3640 1 -- --> v1:172.21.15.115:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3d38136cb0 con 0x7f3d380b1b00 2024-03-25T18:29:11.723 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:29:11.720+0000 7f3d3e0b3640 1 -- --> v1:172.21.15.1:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3d38136f90 con 0x7f3d380b44b0 2024-03-25T18:29:11.723 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:29:11.720+0000 7f3d3e0b3640 10 monclient(hunting): _renew_subs 2024-03-25T18:29:11.723 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:29:11.720+0000 7f3d3e0b3640 10 monclient(hunting): authenticate will time out at 7632.190942s 2024-03-25T18:29:11.726 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:29:11.724+0000 7f3d37fff640 1 --1- >> v1:172.21.15.1:6789/0 conn(0x7f3d380b4e70 0x7f3d381e2800 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.1:6789/0 says I am v1:172.21.15.1:46272/0 (socket says 172.21.15.1:46272) 2024-03-25T18:29:11.726 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:29:11.724+0000 7f3d37fff640 1 -- 172.21.15.1:0/1253424694 learned_addr learned my addr 172.21.15.1:0/1253424694 (peer_addr_for_me v1:172.21.15.1:0/0) 2024-03-25T18:29:11.728 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:29:11.726+0000 7f3d1ffff640 10 client.?.objecter ms_handle_connect 0x7f3d380b4e70 2024-03-25T18:29:11.728 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:29:11.726+0000 7f3d1ffff640 10 client.?.objecter resend_mon_ops 2024-03-25T18:29:11.734 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:29:11.733+0000 7f3d1ffff640 10 client.?.objecter ms_handle_connect 0x7f3d380b44b0 2024-03-25T18:29:11.734 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:29:11.733+0000 7f3d1ffff640 10 client.?.objecter resend_mon_ops 2024-03-25T18:29:11.739 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:29:11.738+0000 7f3d1ffff640 10 client.?.objecter ms_handle_connect 0x7f3d380b1b00 2024-03-25T18:29:11.739 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:29:11.738+0000 7f3d1ffff640 10 client.?.objecter resend_mon_ops 2024-03-25T18:29:11.740 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:29:11.739+0000 7f3d1ffff640 1 -- 172.21.15.1:0/1253424694 <== mon.0 v1:172.21.15.1:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2295157094 0 0) 0x7f3d381369d0 con 0x7f3d380b4e70 2024-03-25T18:29:11.740 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:29:11.739+0000 7f3d1ffff640 10 monclient(hunting): _negotiate 2024-03-25T18:29:11.740 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:29:11.739+0000 7f3d1ffff640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:29:11.741 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:29:11.739+0000 7f3d1ffff640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:29:11.741 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:29:11.739+0000 7f3d1ffff640 10 monclient(hunting): my global_id is 4332 2024-03-25T18:29:11.741 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:29:11.739+0000 7f3d1ffff640 1 -- 172.21.15.1:0/1253424694 --> v1:172.21.15.1:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f3d24004eb0 con 0x7f3d380b4e70 2024-03-25T18:29:11.743 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:29:11.742+0000 7f3d1ffff640 1 -- 172.21.15.1:0/1253424694 <== mon.2 v1:172.21.15.1:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 271819705 0 0) 0x7f3d38136f90 con 0x7f3d380b44b0 2024-03-25T18:29:11.743 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:29:11.742+0000 7f3d1ffff640 10 monclient(hunting): _negotiate 2024-03-25T18:29:11.744 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:29:11.742+0000 7f3d1ffff640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:29:11.744 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:29:11.742+0000 7f3d1ffff640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:29:11.744 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:29:11.742+0000 7f3d1ffff640 10 monclient(hunting): my global_id is 4334 2024-03-25T18:29:11.744 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:29:11.742+0000 7f3d1ffff640 1 -- 172.21.15.1:0/1253424694 --> v1:172.21.15.1:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f3d381369d0 con 0x7f3d380b44b0 2024-03-25T18:29:11.753 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:29:11.751+0000 7f3d1ffff640 1 -- 172.21.15.1:0/1253424694 <== mon.1 v1:172.21.15.115:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3729868319 0 0) 0x7f3d38136cb0 con 0x7f3d380b1b00 2024-03-25T18:29:11.753 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:29:11.751+0000 7f3d1ffff640 10 monclient(hunting): _negotiate 2024-03-25T18:29:11.753 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:29:11.751+0000 7f3d1ffff640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:29:11.753 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:29:11.751+0000 7f3d1ffff640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:29:11.753 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:29:11.751+0000 7f3d1ffff640 10 monclient(hunting): my global_id is 4327 2024-03-25T18:29:11.753 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:29:11.751+0000 7f3d1ffff640 1 -- 172.21.15.1:0/1253424694 --> v1:172.21.15.115:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f3d38136f90 con 0x7f3d380b1b00 2024-03-25T18:29:11.763 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:29:11.761+0000 7f3d1ffff640 1 -- 172.21.15.1:0/1253424694 <== mon.0 v1:172.21.15.1:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2308710558 0 0) 0x7f3d24004eb0 con 0x7f3d380b4e70 2024-03-25T18:29:11.763 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:29:11.761+0000 7f3d1ffff640 1 -- 172.21.15.1:0/1253424694 --> v1:172.21.15.1:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f3d38136cb0 con 0x7f3d380b4e70 2024-03-25T18:29:11.763 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:29:11.761+0000 7f3d1ffff640 1 -- 172.21.15.1:0/1253424694 <== mon.0 v1:172.21.15.1:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1586965025 0 0) 0x7f3d30001d10 con 0x7f3d380b4e70 2024-03-25T18:29:11.763 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:29:11.761+0000 7f3d1ffff640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:29:11.763 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:29:11.761+0000 7f3d1ffff640 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2024-03-25T18:29:11.764 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:29:11.761+0000 7f3d1ffff640 10 monclient(hunting): dump: 2024-03-25T18:29:11.764 INFO:teuthology.misc.health.smithi001.stderr:epoch 1 2024-03-25T18:29:11.764 INFO:teuthology.misc.health.smithi001.stderr:fsid c8ea72cf-9e99-4383-9b52-6c23d365fdfc 2024-03-25T18:29:11.764 INFO:teuthology.misc.health.smithi001.stderr:last_changed 2024-03-25T18:25:50.510519+0000 2024-03-25T18:29:11.764 INFO:teuthology.misc.health.smithi001.stderr:created 2024-03-25T18:25:50.510519+0000 2024-03-25T18:29:11.764 INFO:teuthology.misc.health.smithi001.stderr:min_mon_release 19 (squid) 2024-03-25T18:29:11.764 INFO:teuthology.misc.health.smithi001.stderr:election_strategy: 3 2024-03-25T18:29:11.764 INFO:teuthology.misc.health.smithi001.stderr:0: v1:172.21.15.1:6789/0 mon.a 2024-03-25T18:29:11.764 INFO:teuthology.misc.health.smithi001.stderr:1: v1:172.21.15.115:6789/0 mon.b 2024-03-25T18:29:11.764 INFO:teuthology.misc.health.smithi001.stderr:2: v1:172.21.15.1:6790/0 mon.c 2024-03-25T18:29:11.764 INFO:teuthology.misc.health.smithi001.stderr: 2024-03-25T18:29:11.764 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:29:11.761+0000 7f3d1ffff640 10 monclient(hunting): _finish_auth 0 2024-03-25T18:29:11.770 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:29:11.768+0000 7f3d1ffff640 1 -- 172.21.15.1:0/1253424694 <== mon.2 v1:172.21.15.1:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1076531692 0 0) 0x7f3d381369d0 con 0x7f3d380b44b0 2024-03-25T18:29:11.770 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:29:11.768+0000 7f3d1ffff640 1 -- 172.21.15.1:0/1253424694 --> v1:172.21.15.1:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f3d24004eb0 con 0x7f3d380b44b0 2024-03-25T18:29:11.770 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:29:11.768+0000 7f3d1ffff640 1 -- 172.21.15.1:0/1253424694 <== mon.2 v1:172.21.15.1:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1586965025 0 0) 0x7f3d2c0018b0 con 0x7f3d380b44b0 2024-03-25T18:29:11.770 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:29:11.768+0000 7f3d1ffff640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:29:11.770 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:29:11.768+0000 7f3d1ffff640 10 monclient(hunting): got monmap 1 from mon.c (according to old e1) 2024-03-25T18:29:11.770 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:29:11.768+0000 7f3d1ffff640 10 monclient(hunting): dump: 2024-03-25T18:29:11.770 INFO:teuthology.misc.health.smithi001.stderr:epoch 1 2024-03-25T18:29:11.770 INFO:teuthology.misc.health.smithi001.stderr:fsid c8ea72cf-9e99-4383-9b52-6c23d365fdfc 2024-03-25T18:29:11.770 INFO:teuthology.misc.health.smithi001.stderr:last_changed 2024-03-25T18:25:50.510519+0000 2024-03-25T18:29:11.770 INFO:teuthology.misc.health.smithi001.stderr:created 2024-03-25T18:25:50.510519+0000 2024-03-25T18:29:11.771 INFO:teuthology.misc.health.smithi001.stderr:min_mon_release 19 (squid) 2024-03-25T18:29:11.771 INFO:teuthology.misc.health.smithi001.stderr:election_strategy: 3 2024-03-25T18:29:11.771 INFO:teuthology.misc.health.smithi001.stderr:0: v1:172.21.15.1:6789/0 mon.a 2024-03-25T18:29:11.771 INFO:teuthology.misc.health.smithi001.stderr:1: v1:172.21.15.115:6789/0 mon.b 2024-03-25T18:29:11.771 INFO:teuthology.misc.health.smithi001.stderr:2: v1:172.21.15.1:6790/0 mon.c 2024-03-25T18:29:11.771 INFO:teuthology.misc.health.smithi001.stderr: 2024-03-25T18:29:11.772 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:29:11.769+0000 7f3d1ffff640 1 -- 172.21.15.1:0/1253424694 <== mon.0 v1:172.21.15.1:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 2251578787 0 0) 0x7f3d38136cb0 con 0x7f3d380b4e70 2024-03-25T18:29:11.772 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:29:11.769+0000 7f3d1ffff640 1 -- 172.21.15.1:0/1253424694 >> v1:172.21.15.1:6790/0 conn(0x7f3d380b44b0 legacy=0x7f3d38137770 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:29:11.772 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:29:11.769+0000 7f3d1ffff640 1 -- 172.21.15.1:0/1253424694 >> v1:172.21.15.115:6789/0 conn(0x7f3d380b1b00 legacy=0x7f3d381350e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:29:11.772 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:29:11.769+0000 7f3d1ffff640 10 monclient: _finish_hunting 0 2024-03-25T18:29:11.773 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:29:11.769+0000 7f3d1ffff640 1 monclient: found mon.a 2024-03-25T18:29:11.773 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:29:11.769+0000 7f3d1ffff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:29:11.773 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:29:11.769+0000 7f3d1ffff640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.1:6789/0 2024-03-25T18:29:11.773 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:29:11.769+0000 7f3d1ffff640 1 -- 172.21.15.1:0/1253424694 --> v1:172.21.15.1:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3d38137e80 con 0x7f3d380b4e70 2024-03-25T18:29:11.773 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:29:11.769+0000 7f3d1ffff640 10 monclient: _finish_auth 0 2024-03-25T18:29:11.773 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:29:11.769+0000 7f3d1ffff640 10 monclient: _check_auth_tickets 2024-03-25T18:29:11.773 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:29:11.769+0000 7f3d1ffff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:29:11.773 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:29:11.769+0000 7f3d3e0b3640 5 monclient: authenticate success, global_id 4332 2024-03-25T18:29:11.773 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:29:11.769+0000 7f3d3e0b3640 10 monclient: _renew_subs 2024-03-25T18:29:11.773 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:29:11.769+0000 7f3d3e0b3640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.1:6789/0 2024-03-25T18:29:11.773 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:29:11.769+0000 7f3d3e0b3640 1 -- 172.21.15.1:0/1253424694 --> v1:172.21.15.1:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3d381447c0 con 0x7f3d380b4e70 2024-03-25T18:29:11.773 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:29:11.769+0000 7f3d3e0b3640 10 client.4332.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T18:29:11.773 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:29:11.769+0000 7f3d3e0b3640 10 monclient: _renew_subs 2024-03-25T18:29:11.774 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:29:11.769+0000 7f3d3e0b3640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.1:6789/0 2024-03-25T18:29:11.774 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:29:11.769+0000 7f3d3e0b3640 1 -- 172.21.15.1:0/1253424694 --> v1:172.21.15.1:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f3d38143320 con 0x7f3d380b4e70 2024-03-25T18:29:11.774 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:29:11.769+0000 7f3d3e0b3640 1 librados: init done 2024-03-25T18:29:11.774 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:29:11.770+0000 7f3d3e0b3640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T18:29:11.774 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:29:11.770+0000 7f3d3e0b3640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:29:11.774 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:29:11.770+0000 7f3d3e0b3640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.1:6789/0 2024-03-25T18:29:11.774 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:29:11.770+0000 7f3d3e0b3640 1 -- 172.21.15.1:0/1253424694 --> v1:172.21.15.1:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f3d3804f730 con 0x7f3d380b4e70 2024-03-25T18:29:12.089 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:29:12.087+0000 7f3d1ffff640 1 -- 172.21.15.1:0/1253424694 <== mon.0 v1:172.21.15.1:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f3d30003990 con 0x7f3d380b4e70 2024-03-25T18:29:12.089 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:29:12.087+0000 7f3d1ffff640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:29:12.089 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:29:12.087+0000 7f3d1ffff640 1 -- 172.21.15.1:0/1253424694 <== mon.0 v1:172.21.15.1:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1586965025 0 0) 0x7f3d30003ce0 con 0x7f3d380b4e70 2024-03-25T18:29:12.089 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:29:12.087+0000 7f3d1ffff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:29:12.089 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:29:12.087+0000 7f3d1ffff640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-25T18:29:12.089 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:29:12.087+0000 7f3d1ffff640 10 monclient: dump: 2024-03-25T18:29:12.089 INFO:teuthology.misc.health.smithi001.stderr:epoch 1 2024-03-25T18:29:12.089 INFO:teuthology.misc.health.smithi001.stderr:fsid c8ea72cf-9e99-4383-9b52-6c23d365fdfc 2024-03-25T18:29:12.089 INFO:teuthology.misc.health.smithi001.stderr:last_changed 2024-03-25T18:25:50.510519+0000 2024-03-25T18:29:12.089 INFO:teuthology.misc.health.smithi001.stderr:created 2024-03-25T18:25:50.510519+0000 2024-03-25T18:29:12.090 INFO:teuthology.misc.health.smithi001.stderr:min_mon_release 19 (squid) 2024-03-25T18:29:12.090 INFO:teuthology.misc.health.smithi001.stderr:election_strategy: 3 2024-03-25T18:29:12.090 INFO:teuthology.misc.health.smithi001.stderr:0: v1:172.21.15.1:6789/0 mon.a 2024-03-25T18:29:12.090 INFO:teuthology.misc.health.smithi001.stderr:1: v1:172.21.15.115:6789/0 mon.b 2024-03-25T18:29:12.090 INFO:teuthology.misc.health.smithi001.stderr:2: v1:172.21.15.1:6790/0 mon.c 2024-03-25T18:29:12.090 INFO:teuthology.misc.health.smithi001.stderr: 2024-03-25T18:29:12.090 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:29:12.087+0000 7f3d35ffb640 4 set_mon_vals no callback set 2024-03-25T18:29:12.091 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:29:12.088+0000 7f3d1ffff640 1 -- 172.21.15.1:0/1253424694 <== mon.0 v1:172.21.15.1:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 21787975 0 0) 0x7f3d30003f00 con 0x7f3d380b4e70 2024-03-25T18:29:12.091 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:29:12.088+0000 7f3d377fe640 10 monclient: get_auth_request con 0x7f3d2406df70 auth_method 0 2024-03-25T18:29:12.091 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:29:12.088+0000 7f3d1ffff640 10 client.4332.objecter ms_handle_connect 0x7f3d2406df70 2024-03-25T18:29:12.096 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:29:12.094+0000 7f3d1ffff640 1 -- 172.21.15.1:0/1253424694 <== mon.0 v1:172.21.15.1:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 996254230 0 0) 0x7f3d30054af0 con 0x7f3d380b4e70 2024-03-25T18:29:12.096 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:29:12.094+0000 7f3d1ffff640 10 client.4332.objecter ms_dispatch 0x7f3d38000b90 osd_map(10..10 src has 1..10) v4 2024-03-25T18:29:12.096 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:29:12.094+0000 7f3d1ffff640 3 client.4332.objecter handle_osd_map got epochs [10,10] > 0 2024-03-25T18:29:12.096 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:29:12.094+0000 7f3d1ffff640 3 client.4332.objecter handle_osd_map decoding full epoch 10 2024-03-25T18:29:12.096 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:29:12.094+0000 7f3d1ffff640 20 client.4332.objecter dump_active .. 0 homeless 2024-03-25T18:29:12.096 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:29:12.094+0000 7f3d1ffff640 1 -- 172.21.15.1:0/1253424694 <== mon.0 v1:172.21.15.1:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f3d30050600 con 0x7f3d380b4e70 2024-03-25T18:29:12.096 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:29:12.094+0000 7f3d1ffff640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:29:12.096 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:29:12.094+0000 7f3d1ffff640 10 monclient: _finish_command 1 = system:0 2024-03-25T18:29:12.353 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:29:12.352+0000 7f3d3e0b3640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:29:12.354 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:29:12.352+0000 7f3d3e0b3640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:29:12.354 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:29:12.352+0000 7f3d3e0b3640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.1:6789/0 2024-03-25T18:29:12.354 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:29:12.352+0000 7f3d3e0b3640 1 -- 172.21.15.1:0/1253424694 --> v1:172.21.15.1:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f3d38136080 con 0x7f3d380b4e70 2024-03-25T18:29:12.376 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:29:12.375+0000 7f3d1ffff640 1 -- 172.21.15.1:0/1253424694 <== mon.0 v1:172.21.15.1:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 4092213348) 0x7f3d30052530 con 0x7f3d380b4e70 2024-03-25T18:29:12.377 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:29:12.375+0000 7f3d1ffff640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:29:12.377 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:29:12.375+0000 7f3d1ffff640 10 monclient: _finish_command 2 = system:0 2024-03-25T18:29:12.377 INFO:teuthology.misc.health.smithi001.stdout: 2024-03-25T18:29:12.377 INFO:teuthology.misc.health.smithi001.stdout:{"epoch":10,"fsid":"c8ea72cf-9e99-4383-9b52-6c23d365fdfc","created":"2024-03-25T18:26:38.856638+0000","modified":"2024-03-25T18:26:44.224208+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T18:26:44.224208+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"b9f34922-3a84-49be-8426-807c1eceaefc","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"42bf306f-55e3-4634-9874-8b4b73ac6644","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"292cb23a-18a9-4207-a7cc-4d5727feaec0","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"45e0e9ab-898d-4673-9590-b4117bc5e84a","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"831a5a09-73ab-4a61-9872-088e76bdbc66","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"88cd903d-3b55-4840-9396-6e3cfe005282","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"89db7355-bc9d-46f4-8ab0-e8d9e744e026","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"cac47bee-0914-4f08-896f-348838882866","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T18:29:12.380 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:29:12.377+0000 7f3d3e0b3640 10 librados: watch_flush enter 2024-03-25T18:29:12.380 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:29:12.377+0000 7f3d3e0b3640 10 librados: watch_flush exit 2024-03-25T18:29:12.380 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:29:12.377+0000 7f3d3e0b3640 20 client.4332.objecter shutdown clearing up homeless session... 2024-03-25T18:29:12.380 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:29:12.377+0000 7f3d3e0b3640 10 client.4332.objecter successfully canceled tick 2024-03-25T18:29:12.380 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:29:12.378+0000 7f3d3e0b3640 1 -- 172.21.15.1:0/1253424694 >> v1:172.21.15.115:6800/4237379176 conn(0x7f3d2406df70 legacy=0x7f3d24070430 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:29:12.380 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:29:12.378+0000 7f3d3e0b3640 10 monclient: shutdown 2024-03-25T18:29:12.380 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:29:12.378+0000 7f3d3e0b3640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:29:12.380 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:29:12.378+0000 7f3d3e0b3640 1 -- 172.21.15.1:0/1253424694 >> v1:172.21.15.1:6789/0 conn(0x7f3d380b4e70 legacy=0x7f3d381e2800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:29:12.380 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:29:12.378+0000 7f3d3e0b3640 1 -- 172.21.15.1:0/1253424694 shutdown_connections 2024-03-25T18:29:12.380 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:29:12.378+0000 7f3d3e0b3640 1 -- 172.21.15.1:0/1253424694 >> 172.21.15.1:0/1253424694 conn(0x7f3d380617a0 msgr2=0x7f3d38133ba0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:29:12.380 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:29:12.378+0000 7f3d3e0b3640 1 -- 172.21.15.1:0/1253424694 shutdown_connections 2024-03-25T18:29:12.380 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:29:12.378+0000 7f3d3e0b3640 1 -- 172.21.15.1:0/1253424694 wait complete. 2024-03-25T18:29:12.381 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:29:12.378+0000 7f3d3e0b3640 1 librados: shutdown 2024-03-25T18:29:12.392 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T18:29:16.258 INFO:tasks.ceph.mon.a.smithi001.stderr:==00:00:02:54.816 34876== Warning: unimplemented fcntl command: 1036 2024-03-25T18:29:16.305 INFO:tasks.ceph.mon.a.smithi001.stderr:==00:00:02:54.863 34876== Warning: unimplemented fcntl command: 1036 2024-03-25T18:29:16.324 INFO:tasks.ceph.mon.a.smithi001.stderr:==00:00:02:54.882 34876== Warning: unimplemented fcntl command: 1036 2024-03-25T18:29:16.356 INFO:tasks.ceph.mon.b.smithi115.stderr:==00:00:02:54.839 34836== Warning: unimplemented fcntl command: 1036 2024-03-25T18:29:16.361 INFO:tasks.ceph.mon.b.smithi115.stderr:==00:00:02:54.844 34836== Warning: unimplemented fcntl command: 1036 2024-03-25T18:29:16.364 INFO:tasks.ceph.mon.c.smithi001.stderr:==00:00:02:54.923 34877== Warning: unimplemented fcntl command: 1036 2024-03-25T18:29:16.369 INFO:tasks.ceph.mon.c.smithi001.stderr:==00:00:02:54.927 34877== Warning: unimplemented fcntl command: 1036 2024-03-25T18:29:16.395 INFO:tasks.ceph.mon.b.smithi115.stderr:==00:00:02:54.879 34836== Warning: unimplemented fcntl command: 1036 2024-03-25T18:29:16.409 INFO:tasks.ceph.mon.c.smithi001.stderr:==00:00:02:54.967 34877== Warning: unimplemented fcntl command: 1036 2024-03-25T18:29:19.200 DEBUG:teuthology.orchestra.run.smithi001:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T18:29:19.313 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:29:19.311+0000 7fe8b7063640 10 monclient: get_monmap_and_config 2024-03-25T18:29:19.313 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:29:19.311+0000 7fe8b7063640 10 monclient: build_initial_monmap 2024-03-25T18:29:19.313 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:29:19.311+0000 7fe8b7063640 10 monclient: monmap: 2024-03-25T18:29:19.313 INFO:teuthology.misc.health.smithi001.stderr:epoch 0 2024-03-25T18:29:19.313 INFO:teuthology.misc.health.smithi001.stderr:fsid c8ea72cf-9e99-4383-9b52-6c23d365fdfc 2024-03-25T18:29:19.313 INFO:teuthology.misc.health.smithi001.stderr:last_changed 2024-03-25T18:29:19.312741+0000 2024-03-25T18:29:19.313 INFO:teuthology.misc.health.smithi001.stderr:created 2024-03-25T18:29:19.312741+0000 2024-03-25T18:29:19.313 INFO:teuthology.misc.health.smithi001.stderr:min_mon_release 0 (unknown) 2024-03-25T18:29:19.313 INFO:teuthology.misc.health.smithi001.stderr:election_strategy: 3 2024-03-25T18:29:19.313 INFO:teuthology.misc.health.smithi001.stderr:0: v1:172.21.15.1:6789/0 mon.noname-a 2024-03-25T18:29:19.314 INFO:teuthology.misc.health.smithi001.stderr:1: v1:172.21.15.115:6789/0 mon.noname-c 2024-03-25T18:29:19.314 INFO:teuthology.misc.health.smithi001.stderr:2: v1:172.21.15.1:6790/0 mon.noname-b 2024-03-25T18:29:19.314 INFO:teuthology.misc.health.smithi001.stderr: 2024-03-25T18:29:19.315 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:29:19.312+0000 7fe8b7063640 1 Processor -- start 2024-03-25T18:29:19.315 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:29:19.312+0000 7fe8b7063640 1 -- start start 2024-03-25T18:29:19.315 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:29:19.312+0000 7fe8b7063640 10 monclient: init 2024-03-25T18:29:19.315 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:29:19.313+0000 7fe8b7063640 10 monclient: _reopen_session rank -1 2024-03-25T18:29:19.315 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:29:19.313+0000 7fe8b7063640 10 monclient: _add_conns ranks=[2,0,1] 2024-03-25T18:29:19.315 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:29:19.313+0000 7fe8b7063640 10 monclient(hunting): picked mon.noname-b con 0x7fe8b00b4e70 addr v1:172.21.15.1:6790/0 2024-03-25T18:29:19.315 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:29:19.313+0000 7fe8b7063640 10 monclient(hunting): picked mon.noname-a con 0x7fe8b00b44b0 addr v1:172.21.15.1:6789/0 2024-03-25T18:29:19.315 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:29:19.313+0000 7fe8b7063640 10 monclient(hunting): picked mon.noname-c con 0x7fe8b00b1b00 addr v1:172.21.15.115:6789/0 2024-03-25T18:29:19.315 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:29:19.313+0000 7fe8b7063640 1 -- --> v1:172.21.15.1:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe8b00a96f0 con 0x7fe8b00b44b0 2024-03-25T18:29:19.316 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:29:19.313+0000 7fe8b7063640 1 -- --> v1:172.21.15.115:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe8b00ab0e0 con 0x7fe8b00b1b00 2024-03-25T18:29:19.316 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:29:19.313+0000 7fe8b7063640 1 -- --> v1:172.21.15.1:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe8b0145f70 con 0x7fe8b00b4e70 2024-03-25T18:29:19.316 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:29:19.313+0000 7fe8b7063640 10 monclient(hunting): _renew_subs 2024-03-25T18:29:19.316 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:29:19.313+0000 7fe8b7063640 10 monclient(hunting): authenticate will time out at 7639.784286s 2024-03-25T18:29:19.326 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:29:19.324+0000 7fe8b4dff640 1 --1- >> v1:172.21.15.1:6790/0 conn(0x7fe8b00b4e70 0x7fe8b013f940 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.1:6790/0 says I am v1:172.21.15.1:36874/0 (socket says 172.21.15.1:36874) 2024-03-25T18:29:19.326 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:29:19.325+0000 7fe8b4dff640 1 -- 172.21.15.1:0/1276820226 learned_addr learned my addr 172.21.15.1:0/1276820226 (peer_addr_for_me v1:172.21.15.1:0/0) 2024-03-25T18:29:19.334 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:29:19.332+0000 7fe8af7fe640 1 -- 172.21.15.1:0/1276820226 <== mon.0 v1:172.21.15.1:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 545208919 0 0) 0x7fe8b00a96f0 con 0x7fe8b00b44b0 2024-03-25T18:29:19.334 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:29:19.332+0000 7fe8af7fe640 10 monclient(hunting): _negotiate 2024-03-25T18:29:19.334 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:29:19.332+0000 7fe8af7fe640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:29:19.334 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:29:19.332+0000 7fe8af7fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:29:19.334 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:29:19.332+0000 7fe8af7fe640 10 monclient(hunting): my global_id is 4335 2024-03-25T18:29:19.334 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:29:19.332+0000 7fe8af7fe640 1 -- 172.21.15.1:0/1276820226 --> v1:172.21.15.1:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fe8a0003520 con 0x7fe8b00b44b0 2024-03-25T18:29:19.336 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:29:19.334+0000 7fe8af7fe640 1 -- 172.21.15.1:0/1276820226 <== mon.1 v1:172.21.15.115:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4264556555 0 0) 0x7fe8b00ab0e0 con 0x7fe8b00b1b00 2024-03-25T18:29:19.336 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:29:19.334+0000 7fe8af7fe640 10 monclient(hunting): _negotiate 2024-03-25T18:29:19.336 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:29:19.334+0000 7fe8af7fe640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:29:19.336 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:29:19.334+0000 7fe8af7fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:29:19.336 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:29:19.334+0000 7fe8af7fe640 10 monclient(hunting): my global_id is 4330 2024-03-25T18:29:19.336 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:29:19.334+0000 7fe8af7fe640 1 -- 172.21.15.1:0/1276820226 --> v1:172.21.15.115:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fe8b00a96f0 con 0x7fe8b00b1b00 2024-03-25T18:29:19.342 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:29:19.341+0000 7fe8af7fe640 1 -- 172.21.15.1:0/1276820226 <== mon.1 v1:172.21.15.115:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3457763534 0 0) 0x7fe8b00a96f0 con 0x7fe8b00b1b00 2024-03-25T18:29:19.342 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:29:19.341+0000 7fe8af7fe640 1 -- 172.21.15.1:0/1276820226 >> v1:172.21.15.1:6790/0 conn(0x7fe8b00b4e70 legacy=0x7fe8b013f940 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:29:19.343 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:29:19.341+0000 7fe8af7fe640 1 -- 172.21.15.1:0/1276820226 >> v1:172.21.15.1:6789/0 conn(0x7fe8b00b44b0 legacy=0x7fe8b0145860 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:29:19.343 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:29:19.341+0000 7fe8af7fe640 10 monclient: _finish_hunting 0 2024-03-25T18:29:19.343 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:29:19.341+0000 7fe8af7fe640 1 monclient: found mon.noname-c 2024-03-25T18:29:19.343 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:29:19.341+0000 7fe8af7fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:29:19.343 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:29:19.341+0000 7fe8af7fe640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.115:6789/0 2024-03-25T18:29:19.343 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:29:19.341+0000 7fe8af7fe640 1 -- 172.21.15.1:0/1276820226 --> v1:172.21.15.115:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe8b00a9b80 con 0x7fe8b00b1b00 2024-03-25T18:29:19.343 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:29:19.341+0000 7fe8af7fe640 10 monclient: _finish_auth 0 2024-03-25T18:29:19.343 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:29:19.341+0000 7fe8af7fe640 10 monclient: _check_auth_tickets 2024-03-25T18:29:19.343 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:29:19.341+0000 7fe8af7fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:29:19.343 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:29:19.341+0000 7fe8af7fe640 1 -- 172.21.15.1:0/1276820226 <== mon.1 v1:172.21.15.115:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1586965025 0 0) 0x7fe8a80019f0 con 0x7fe8b00b1b00 2024-03-25T18:29:19.343 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:29:19.341+0000 7fe8b7063640 5 monclient: authenticate success, global_id 4330 2024-03-25T18:29:19.343 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:29:19.341+0000 7fe8b7063640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T18:29:19.343 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:29:19.341+0000 7fe8af7fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:29:19.344 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:29:19.341+0000 7fe8af7fe640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-03-25T18:29:19.344 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:29:19.341+0000 7fe8af7fe640 10 monclient: dump: 2024-03-25T18:29:19.344 INFO:teuthology.misc.health.smithi001.stderr:epoch 1 2024-03-25T18:29:19.344 INFO:teuthology.misc.health.smithi001.stderr:fsid c8ea72cf-9e99-4383-9b52-6c23d365fdfc 2024-03-25T18:29:19.344 INFO:teuthology.misc.health.smithi001.stderr:last_changed 2024-03-25T18:25:50.510519+0000 2024-03-25T18:29:19.344 INFO:teuthology.misc.health.smithi001.stderr:created 2024-03-25T18:25:50.510519+0000 2024-03-25T18:29:19.344 INFO:teuthology.misc.health.smithi001.stderr:min_mon_release 19 (squid) 2024-03-25T18:29:19.344 INFO:teuthology.misc.health.smithi001.stderr:election_strategy: 3 2024-03-25T18:29:19.344 INFO:teuthology.misc.health.smithi001.stderr:0: v1:172.21.15.1:6789/0 mon.a 2024-03-25T18:29:19.344 INFO:teuthology.misc.health.smithi001.stderr:1: v1:172.21.15.115:6789/0 mon.b 2024-03-25T18:29:19.344 INFO:teuthology.misc.health.smithi001.stderr:2: v1:172.21.15.1:6790/0 mon.c 2024-03-25T18:29:19.344 INFO:teuthology.misc.health.smithi001.stderr: 2024-03-25T18:29:19.344 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:29:19.341+0000 7fe8b7063640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T18:29:19.356 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:29:19.354+0000 7fe8af7fe640 1 -- 172.21.15.1:0/1276820226 <== mon.1 v1:172.21.15.115:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fe8a8001b50 con 0x7fe8b00b1b00 2024-03-25T18:29:19.356 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:29:19.354+0000 7fe8af7fe640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:29:19.356 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:29:19.354+0000 7fe8af7fe640 1 -- 172.21.15.1:0/1276820226 <== mon.1 v1:172.21.15.115:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1586965025 0 0) 0x7fe8a8001f20 con 0x7fe8b00b1b00 2024-03-25T18:29:19.356 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:29:19.354+0000 7fe8af7fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:29:19.356 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:29:19.354+0000 7fe8af7fe640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-25T18:29:19.356 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:29:19.354+0000 7fe8af7fe640 10 monclient: dump: 2024-03-25T18:29:19.356 INFO:teuthology.misc.health.smithi001.stderr:epoch 1 2024-03-25T18:29:19.357 INFO:teuthology.misc.health.smithi001.stderr:fsid c8ea72cf-9e99-4383-9b52-6c23d365fdfc 2024-03-25T18:29:19.357 INFO:teuthology.misc.health.smithi001.stderr:last_changed 2024-03-25T18:25:50.510519+0000 2024-03-25T18:29:19.357 INFO:teuthology.misc.health.smithi001.stderr:created 2024-03-25T18:25:50.510519+0000 2024-03-25T18:29:19.357 INFO:teuthology.misc.health.smithi001.stderr:min_mon_release 19 (squid) 2024-03-25T18:29:19.357 INFO:teuthology.misc.health.smithi001.stderr:election_strategy: 3 2024-03-25T18:29:19.357 INFO:teuthology.misc.health.smithi001.stderr:0: v1:172.21.15.1:6789/0 mon.a 2024-03-25T18:29:19.357 INFO:teuthology.misc.health.smithi001.stderr:1: v1:172.21.15.115:6789/0 mon.b 2024-03-25T18:29:19.357 INFO:teuthology.misc.health.smithi001.stderr:2: v1:172.21.15.1:6790/0 mon.c 2024-03-25T18:29:19.357 INFO:teuthology.misc.health.smithi001.stderr: 2024-03-25T18:29:19.357 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:29:19.354+0000 7fe8b7063640 10 monclient: get_monmap_and_config success 2024-03-25T18:29:19.357 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:29:19.354+0000 7fe8b7063640 4 set_mon_vals no callback set 2024-03-25T18:29:19.360 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:29:19.355+0000 7fe8b7063640 10 monclient: shutdown 2024-03-25T18:29:19.360 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:29:19.355+0000 7fe8b7063640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:29:19.360 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:29:19.355+0000 7fe8b7063640 1 -- 172.21.15.1:0/1276820226 >> v1:172.21.15.115:6789/0 conn(0x7fe8b00b1b00 legacy=0x7fe8b014dc00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:29:19.360 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:29:19.355+0000 7fe8b7063640 1 -- 172.21.15.1:0/1276820226 shutdown_connections 2024-03-25T18:29:19.360 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:29:19.355+0000 7fe8b7063640 1 -- 172.21.15.1:0/1276820226 >> 172.21.15.1:0/1276820226 conn(0x7fe8b00617a0 msgr2=0x7fe8b0061bb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:29:19.360 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:29:19.355+0000 7fe8b7063640 1 -- 172.21.15.1:0/1276820226 shutdown_connections 2024-03-25T18:29:19.360 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:29:19.355+0000 7fe8b7063640 1 -- 172.21.15.1:0/1276820226 wait complete. 2024-03-25T18:29:19.360 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:29:19.356+0000 7fe8b7063640 10 monclient: build_initial_monmap 2024-03-25T18:29:19.360 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:29:19.356+0000 7fe8b7063640 10 monclient: monmap: 2024-03-25T18:29:19.361 INFO:teuthology.misc.health.smithi001.stderr:epoch 0 2024-03-25T18:29:19.361 INFO:teuthology.misc.health.smithi001.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T18:29:19.361 INFO:teuthology.misc.health.smithi001.stderr:last_changed 0.000000 2024-03-25T18:29:19.361 INFO:teuthology.misc.health.smithi001.stderr:created 0.000000 2024-03-25T18:29:19.361 INFO:teuthology.misc.health.smithi001.stderr:min_mon_release 0 (unknown) 2024-03-25T18:29:19.361 INFO:teuthology.misc.health.smithi001.stderr:election_strategy: 1 2024-03-25T18:29:19.361 INFO:teuthology.misc.health.smithi001.stderr:0: v1:172.21.15.1:6789/0 mon.noname-a 2024-03-25T18:29:19.361 INFO:teuthology.misc.health.smithi001.stderr:1: v1:172.21.15.115:6789/0 mon.noname-b 2024-03-25T18:29:19.361 INFO:teuthology.misc.health.smithi001.stderr:2: v1:172.21.15.1:6790/0 mon.noname-c 2024-03-25T18:29:19.361 INFO:teuthology.misc.health.smithi001.stderr: 2024-03-25T18:29:19.361 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:29:19.356+0000 7fe8b7063640 1 librados: starting msgr at 2024-03-25T18:29:19.361 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:29:19.356+0000 7fe8b7063640 1 librados: starting objecter 2024-03-25T18:29:19.361 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:29:19.356+0000 7fe8b7063640 1 Processor -- start 2024-03-25T18:29:19.361 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:29:19.357+0000 7fe8b7063640 1 -- start start 2024-03-25T18:29:19.362 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:29:19.357+0000 7fe8b7063640 1 librados: setting wanted keys 2024-03-25T18:29:19.362 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:29:19.357+0000 7fe8b7063640 1 librados: calling monclient init 2024-03-25T18:29:19.362 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:29:19.357+0000 7fe8b7063640 10 monclient: init 2024-03-25T18:29:19.362 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:29:19.357+0000 7fe8b7063640 10 monclient: _reopen_session rank -1 2024-03-25T18:29:19.362 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:29:19.357+0000 7fe8b7063640 10 monclient: _add_conns ranks=[2,1,0] 2024-03-25T18:29:19.362 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:29:19.358+0000 7fe8b7063640 10 monclient(hunting): picked mon.noname-c con 0x7fe8b00b1b00 addr v1:172.21.15.1:6790/0 2024-03-25T18:29:19.362 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:29:19.358+0000 7fe8b7063640 10 monclient(hunting): picked mon.noname-b con 0x7fe8b00b44b0 addr v1:172.21.15.115:6789/0 2024-03-25T18:29:19.362 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:29:19.358+0000 7fe8b7063640 10 monclient(hunting): picked mon.noname-a con 0x7fe8b00b4e70 addr v1:172.21.15.1:6789/0 2024-03-25T18:29:19.362 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:29:19.358+0000 7fe8b7063640 1 -- --> v1:172.21.15.1:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe8b01369e0 con 0x7fe8b00b4e70 2024-03-25T18:29:19.362 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:29:19.358+0000 7fe8b7063640 1 -- --> v1:172.21.15.115:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe8b0137bd0 con 0x7fe8b00b44b0 2024-03-25T18:29:19.362 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:29:19.358+0000 7fe8b7063640 1 -- --> v1:172.21.15.1:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe8b0134950 con 0x7fe8b00b1b00 2024-03-25T18:29:19.362 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:29:19.358+0000 7fe8b7063640 10 monclient(hunting): _renew_subs 2024-03-25T18:29:19.362 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:29:19.358+0000 7fe8b7063640 10 monclient(hunting): authenticate will time out at 7639.828724s 2024-03-25T18:29:19.363 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:29:19.361+0000 7fe8affff640 1 --1- >> v1:172.21.15.115:6789/0 conn(0x7fe8b00b44b0 0x7fe8b0133b30 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.115:6789/0 says I am v1:172.21.15.1:35124/0 (socket says 172.21.15.1:35124) 2024-03-25T18:29:19.363 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:29:19.361+0000 7fe8affff640 1 -- 172.21.15.1:0/3689877728 learned_addr learned my addr 172.21.15.1:0/3689877728 (peer_addr_for_me v1:172.21.15.1:0/0) 2024-03-25T18:29:19.363 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:29:19.362+0000 7fe8ad7fa640 10 client.?.objecter ms_handle_connect 0x7fe8b00b44b0 2024-03-25T18:29:19.363 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:29:19.362+0000 7fe8ad7fa640 10 client.?.objecter resend_mon_ops 2024-03-25T18:29:19.375 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:29:19.373+0000 7fe8ad7fa640 10 client.?.objecter ms_handle_connect 0x7fe8b00b4e70 2024-03-25T18:29:19.375 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:29:19.373+0000 7fe8ad7fa640 10 client.?.objecter resend_mon_ops 2024-03-25T18:29:19.376 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:29:19.374+0000 7fe8ad7fa640 1 -- 172.21.15.1:0/3689877728 <== mon.1 v1:172.21.15.115:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1865082657 0 0) 0x7fe8b0137bd0 con 0x7fe8b00b44b0 2024-03-25T18:29:19.376 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:29:19.374+0000 7fe8ad7fa640 10 monclient(hunting): _negotiate 2024-03-25T18:29:19.376 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:29:19.374+0000 7fe8ad7fa640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:29:19.376 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:29:19.374+0000 7fe8ad7fa640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:29:19.376 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:29:19.374+0000 7fe8ad7fa640 10 monclient(hunting): my global_id is 4333 2024-03-25T18:29:19.376 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:29:19.374+0000 7fe8ad7fa640 1 -- 172.21.15.1:0/3689877728 --> v1:172.21.15.115:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fe894004eb0 con 0x7fe8b00b44b0 2024-03-25T18:29:19.377 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:29:19.375+0000 7fe8ad7fa640 10 client.?.objecter ms_handle_connect 0x7fe8b00b1b00 2024-03-25T18:29:19.377 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:29:19.375+0000 7fe8ad7fa640 10 client.?.objecter resend_mon_ops 2024-03-25T18:29:19.386 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:29:19.384+0000 7fe8ad7fa640 1 -- 172.21.15.1:0/3689877728 <== mon.1 v1:172.21.15.115:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 885963349 0 0) 0x7fe894004eb0 con 0x7fe8b00b44b0 2024-03-25T18:29:19.386 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:29:19.384+0000 7fe8ad7fa640 1 -- 172.21.15.1:0/3689877728 --> v1:172.21.15.115:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fe8b0137bd0 con 0x7fe8b00b44b0 2024-03-25T18:29:19.386 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:29:19.384+0000 7fe8ad7fa640 1 -- 172.21.15.1:0/3689877728 <== mon.1 v1:172.21.15.115:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1586965025 0 0) 0x7fe89c001a10 con 0x7fe8b00b44b0 2024-03-25T18:29:19.386 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:29:19.384+0000 7fe8ad7fa640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:29:19.386 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:29:19.384+0000 7fe8ad7fa640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-03-25T18:29:19.386 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:29:19.384+0000 7fe8ad7fa640 10 monclient(hunting): dump: 2024-03-25T18:29:19.386 INFO:teuthology.misc.health.smithi001.stderr:epoch 1 2024-03-25T18:29:19.387 INFO:teuthology.misc.health.smithi001.stderr:fsid c8ea72cf-9e99-4383-9b52-6c23d365fdfc 2024-03-25T18:29:19.387 INFO:teuthology.misc.health.smithi001.stderr:last_changed 2024-03-25T18:25:50.510519+0000 2024-03-25T18:29:19.387 INFO:teuthology.misc.health.smithi001.stderr:created 2024-03-25T18:25:50.510519+0000 2024-03-25T18:29:19.387 INFO:teuthology.misc.health.smithi001.stderr:min_mon_release 19 (squid) 2024-03-25T18:29:19.387 INFO:teuthology.misc.health.smithi001.stderr:election_strategy: 3 2024-03-25T18:29:19.387 INFO:teuthology.misc.health.smithi001.stderr:0: v1:172.21.15.1:6789/0 mon.a 2024-03-25T18:29:19.387 INFO:teuthology.misc.health.smithi001.stderr:1: v1:172.21.15.115:6789/0 mon.b 2024-03-25T18:29:19.387 INFO:teuthology.misc.health.smithi001.stderr:2: v1:172.21.15.1:6790/0 mon.c 2024-03-25T18:29:19.387 INFO:teuthology.misc.health.smithi001.stderr: 2024-03-25T18:29:19.387 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:29:19.384+0000 7fe8ad7fa640 10 monclient(hunting): _finish_auth 0 2024-03-25T18:29:19.396 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:29:19.394+0000 7fe8ad7fa640 1 -- 172.21.15.1:0/3689877728 <== mon.0 v1:172.21.15.1:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3748028208 0 0) 0x7fe8b01369e0 con 0x7fe8b00b4e70 2024-03-25T18:29:19.396 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:29:19.394+0000 7fe8ad7fa640 10 monclient(hunting): _negotiate 2024-03-25T18:29:19.396 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:29:19.394+0000 7fe8ad7fa640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:29:19.396 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:29:19.394+0000 7fe8ad7fa640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:29:19.396 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:29:19.394+0000 7fe8ad7fa640 10 monclient(hunting): my global_id is 4338 2024-03-25T18:29:19.396 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:29:19.394+0000 7fe8ad7fa640 1 -- 172.21.15.1:0/3689877728 --> v1:172.21.15.1:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fe894004eb0 con 0x7fe8b00b4e70 2024-03-25T18:29:19.397 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:29:19.396+0000 7fe8ad7fa640 1 -- 172.21.15.1:0/3689877728 <== mon.2 v1:172.21.15.1:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 887366484 0 0) 0x7fe8b0134950 con 0x7fe8b00b1b00 2024-03-25T18:29:19.397 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:29:19.396+0000 7fe8ad7fa640 10 monclient(hunting): _negotiate 2024-03-25T18:29:19.397 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:29:19.396+0000 7fe8ad7fa640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:29:19.398 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:29:19.396+0000 7fe8ad7fa640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:29:19.398 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:29:19.396+0000 7fe8ad7fa640 10 monclient(hunting): my global_id is 4340 2024-03-25T18:29:19.398 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:29:19.396+0000 7fe8ad7fa640 1 -- 172.21.15.1:0/3689877728 --> v1:172.21.15.1:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fe8b01369e0 con 0x7fe8b00b1b00 2024-03-25T18:29:19.402 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:29:19.399+0000 7fe8ad7fa640 1 -- 172.21.15.1:0/3689877728 <== mon.1 v1:172.21.15.115:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 3385545874 0 0) 0x7fe8b0137bd0 con 0x7fe8b00b44b0 2024-03-25T18:29:19.402 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:29:19.399+0000 7fe8ad7fa640 1 -- 172.21.15.1:0/3689877728 >> v1:172.21.15.1:6790/0 conn(0x7fe8b00b1b00 legacy=0x7fe8b01d76e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:29:19.402 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:29:19.400+0000 7fe8ad7fa640 1 -- 172.21.15.1:0/3689877728 >> v1:172.21.15.1:6789/0 conn(0x7fe8b00b4e70 legacy=0x7fe8b0134240 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:29:19.402 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:29:19.400+0000 7fe8ad7fa640 10 monclient: _finish_hunting 0 2024-03-25T18:29:19.402 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:29:19.400+0000 7fe8ad7fa640 1 monclient: found mon.b 2024-03-25T18:29:19.402 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:29:19.400+0000 7fe8ad7fa640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:29:19.402 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:29:19.400+0000 7fe8ad7fa640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.115:6789/0 2024-03-25T18:29:19.402 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:29:19.400+0000 7fe8ad7fa640 1 -- 172.21.15.1:0/3689877728 --> v1:172.21.15.115:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe8b0137e40 con 0x7fe8b00b44b0 2024-03-25T18:29:19.402 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:29:19.400+0000 7fe8ad7fa640 10 monclient: _finish_auth 0 2024-03-25T18:29:19.403 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:29:19.400+0000 7fe8ad7fa640 10 monclient: _check_auth_tickets 2024-03-25T18:29:19.403 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:29:19.400+0000 7fe8ad7fa640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:29:19.403 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:29:19.400+0000 7fe8b7063640 5 monclient: authenticate success, global_id 4333 2024-03-25T18:29:19.403 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:29:19.400+0000 7fe8b7063640 10 monclient: _renew_subs 2024-03-25T18:29:19.403 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:29:19.400+0000 7fe8b7063640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.115:6789/0 2024-03-25T18:29:19.403 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:29:19.400+0000 7fe8b7063640 1 -- 172.21.15.1:0/3689877728 --> v1:172.21.15.115:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe8b0139b40 con 0x7fe8b00b44b0 2024-03-25T18:29:19.403 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:29:19.400+0000 7fe8b7063640 10 client.4333.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T18:29:19.403 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:29:19.400+0000 7fe8b7063640 10 monclient: _renew_subs 2024-03-25T18:29:19.403 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:29:19.400+0000 7fe8b7063640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.115:6789/0 2024-03-25T18:29:19.403 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:29:19.400+0000 7fe8b7063640 1 -- 172.21.15.1:0/3689877728 --> v1:172.21.15.115:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7fe8b01397e0 con 0x7fe8b00b44b0 2024-03-25T18:29:19.403 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:29:19.400+0000 7fe8b7063640 1 librados: init done 2024-03-25T18:29:19.404 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:29:19.401+0000 7fe8b7063640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T18:29:19.404 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:29:19.401+0000 7fe8b7063640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:29:19.404 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:29:19.401+0000 7fe8b7063640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.115:6789/0 2024-03-25T18:29:19.404 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:29:19.401+0000 7fe8b7063640 1 -- 172.21.15.1:0/3689877728 --> v1:172.21.15.115:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe8b004f730 con 0x7fe8b00b44b0 2024-03-25T18:29:19.674 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:29:19.672+0000 7fe8ad7fa640 1 -- 172.21.15.1:0/3689877728 <== mon.1 v1:172.21.15.115:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fe89c001c90 con 0x7fe8b00b44b0 2024-03-25T18:29:19.674 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:29:19.672+0000 7fe8ad7fa640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:29:19.674 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:29:19.672+0000 7fe8ad7fa640 1 -- 172.21.15.1:0/3689877728 <== mon.1 v1:172.21.15.115:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1586965025 0 0) 0x7fe89c003a60 con 0x7fe8b00b44b0 2024-03-25T18:29:19.674 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:29:19.672+0000 7fe8ad7fa640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:29:19.674 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:29:19.672+0000 7fe8ad7fa640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-25T18:29:19.674 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:29:19.672+0000 7fe8ad7fa640 10 monclient: dump: 2024-03-25T18:29:19.674 INFO:teuthology.misc.health.smithi001.stderr:epoch 1 2024-03-25T18:29:19.674 INFO:teuthology.misc.health.smithi001.stderr:fsid c8ea72cf-9e99-4383-9b52-6c23d365fdfc 2024-03-25T18:29:19.675 INFO:teuthology.misc.health.smithi001.stderr:last_changed 2024-03-25T18:25:50.510519+0000 2024-03-25T18:29:19.675 INFO:teuthology.misc.health.smithi001.stderr:created 2024-03-25T18:25:50.510519+0000 2024-03-25T18:29:19.675 INFO:teuthology.misc.health.smithi001.stderr:min_mon_release 19 (squid) 2024-03-25T18:29:19.675 INFO:teuthology.misc.health.smithi001.stderr:election_strategy: 3 2024-03-25T18:29:19.675 INFO:teuthology.misc.health.smithi001.stderr:0: v1:172.21.15.1:6789/0 mon.a 2024-03-25T18:29:19.675 INFO:teuthology.misc.health.smithi001.stderr:1: v1:172.21.15.115:6789/0 mon.b 2024-03-25T18:29:19.675 INFO:teuthology.misc.health.smithi001.stderr:2: v1:172.21.15.1:6790/0 mon.c 2024-03-25T18:29:19.675 INFO:teuthology.misc.health.smithi001.stderr: 2024-03-25T18:29:19.675 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:29:19.672+0000 7fe8aeffd640 4 set_mon_vals no callback set 2024-03-25T18:29:19.676 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:29:19.673+0000 7fe8ad7fa640 1 -- 172.21.15.1:0/3689877728 <== mon.1 v1:172.21.15.115:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 21787975 0 0) 0x7fe89c003c40 con 0x7fe8b00b44b0 2024-03-25T18:29:19.676 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:29:19.673+0000 7fe8b4dff640 10 monclient: get_auth_request con 0x7fe89406de80 auth_method 0 2024-03-25T18:29:19.676 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:29:19.673+0000 7fe8ad7fa640 10 client.4333.objecter ms_handle_connect 0x7fe89406de80 2024-03-25T18:29:19.677 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:29:19.675+0000 7fe8ad7fa640 1 -- 172.21.15.1:0/3689877728 <== mon.1 v1:172.21.15.115:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 996254230 0 0) 0x7fe89c054870 con 0x7fe8b00b44b0 2024-03-25T18:29:19.677 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:29:19.675+0000 7fe8ad7fa640 10 client.4333.objecter ms_dispatch 0x7fe8b0000b90 osd_map(10..10 src has 1..10) v4 2024-03-25T18:29:19.677 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:29:19.675+0000 7fe8ad7fa640 3 client.4333.objecter handle_osd_map got epochs [10,10] > 0 2024-03-25T18:29:19.677 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:29:19.675+0000 7fe8ad7fa640 3 client.4333.objecter handle_osd_map decoding full epoch 10 2024-03-25T18:29:19.677 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:29:19.675+0000 7fe8ad7fa640 20 client.4333.objecter dump_active .. 0 homeless 2024-03-25T18:29:19.679 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:29:19.677+0000 7fe8ad7fa640 1 -- 172.21.15.1:0/3689877728 <== mon.1 v1:172.21.15.115:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7fe89c0501b0 con 0x7fe8b00b44b0 2024-03-25T18:29:19.679 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:29:19.678+0000 7fe8ad7fa640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:29:19.679 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:29:19.678+0000 7fe8ad7fa640 10 monclient: _finish_command 1 = system:0 2024-03-25T18:29:19.894 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:29:19.892+0000 7fe8b7063640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:29:19.894 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:29:19.892+0000 7fe8b7063640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:29:19.894 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:29:19.892+0000 7fe8b7063640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.115:6789/0 2024-03-25T18:29:19.894 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:29:19.892+0000 7fe8b7063640 1 -- 172.21.15.1:0/3689877728 --> v1:172.21.15.115:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fe8b0136b90 con 0x7fe8b00b44b0 2024-03-25T18:29:19.920 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:29:19.918+0000 7fe8ad7fa640 1 -- 172.21.15.1:0/3689877728 <== mon.1 v1:172.21.15.115:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 4092213348) 0x7fe8b0136b90 con 0x7fe8b00b44b0 2024-03-25T18:29:19.920 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:29:19.918+0000 7fe8ad7fa640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:29:19.920 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:29:19.918+0000 7fe8ad7fa640 10 monclient: _finish_command 2 = system:0 2024-03-25T18:29:19.920 INFO:teuthology.misc.health.smithi001.stdout: 2024-03-25T18:29:19.920 INFO:teuthology.misc.health.smithi001.stdout:{"epoch":10,"fsid":"c8ea72cf-9e99-4383-9b52-6c23d365fdfc","created":"2024-03-25T18:26:38.856638+0000","modified":"2024-03-25T18:26:44.224208+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T18:26:44.224208+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"b9f34922-3a84-49be-8426-807c1eceaefc","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"42bf306f-55e3-4634-9874-8b4b73ac6644","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"292cb23a-18a9-4207-a7cc-4d5727feaec0","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"45e0e9ab-898d-4673-9590-b4117bc5e84a","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"831a5a09-73ab-4a61-9872-088e76bdbc66","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"88cd903d-3b55-4840-9396-6e3cfe005282","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"89db7355-bc9d-46f4-8ab0-e8d9e744e026","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"cac47bee-0914-4f08-896f-348838882866","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T18:29:19.922 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:29:19.920+0000 7fe8b7063640 10 librados: watch_flush enter 2024-03-25T18:29:19.922 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:29:19.920+0000 7fe8b7063640 10 librados: watch_flush exit 2024-03-25T18:29:19.923 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:29:19.920+0000 7fe8b7063640 20 client.4333.objecter shutdown clearing up homeless session... 2024-03-25T18:29:19.923 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:29:19.920+0000 7fe8b7063640 10 client.4333.objecter successfully canceled tick 2024-03-25T18:29:19.923 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:29:19.920+0000 7fe8b7063640 1 -- 172.21.15.1:0/3689877728 >> v1:172.21.15.115:6800/4237379176 conn(0x7fe89406de80 legacy=0x7fe894070340 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:29:19.923 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:29:19.920+0000 7fe8b7063640 10 monclient: shutdown 2024-03-25T18:29:19.923 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:29:19.920+0000 7fe8b7063640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:29:19.923 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:29:19.920+0000 7fe8b7063640 1 -- 172.21.15.1:0/3689877728 >> v1:172.21.15.115:6789/0 conn(0x7fe8b00b44b0 legacy=0x7fe8b0133b30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:29:19.923 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:29:19.920+0000 7fe8b7063640 1 -- 172.21.15.1:0/3689877728 shutdown_connections 2024-03-25T18:29:19.923 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:29:19.920+0000 7fe8b7063640 1 -- 172.21.15.1:0/3689877728 >> 172.21.15.1:0/3689877728 conn(0x7fe8b00617a0 msgr2=0x7fe8b00b6400 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:29:19.923 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:29:19.920+0000 7fe8b7063640 1 -- 172.21.15.1:0/3689877728 shutdown_connections 2024-03-25T18:29:19.923 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:29:19.920+0000 7fe8b7063640 1 -- 172.21.15.1:0/3689877728 wait complete. 2024-03-25T18:29:19.923 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:29:19.921+0000 7fe8b7063640 1 librados: shutdown 2024-03-25T18:29:19.934 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T18:29:26.742 DEBUG:teuthology.orchestra.run.smithi001:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T18:29:26.841 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:29:26.841+0000 7fccc8f4b640 10 monclient: get_monmap_and_config 2024-03-25T18:29:26.842 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:29:26.841+0000 7fccc8f4b640 10 monclient: build_initial_monmap 2024-03-25T18:29:26.842 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:29:26.841+0000 7fccc8f4b640 10 monclient: monmap: 2024-03-25T18:29:26.842 INFO:teuthology.misc.health.smithi001.stderr:epoch 0 2024-03-25T18:29:26.842 INFO:teuthology.misc.health.smithi001.stderr:fsid c8ea72cf-9e99-4383-9b52-6c23d365fdfc 2024-03-25T18:29:26.842 INFO:teuthology.misc.health.smithi001.stderr:last_changed 2024-03-25T18:29:26.841733+0000 2024-03-25T18:29:26.842 INFO:teuthology.misc.health.smithi001.stderr:created 2024-03-25T18:29:26.841733+0000 2024-03-25T18:29:26.842 INFO:teuthology.misc.health.smithi001.stderr:min_mon_release 0 (unknown) 2024-03-25T18:29:26.842 INFO:teuthology.misc.health.smithi001.stderr:election_strategy: 3 2024-03-25T18:29:26.842 INFO:teuthology.misc.health.smithi001.stderr:0: v1:172.21.15.1:6789/0 mon.noname-a 2024-03-25T18:29:26.842 INFO:teuthology.misc.health.smithi001.stderr:1: v1:172.21.15.115:6789/0 mon.noname-c 2024-03-25T18:29:26.842 INFO:teuthology.misc.health.smithi001.stderr:2: v1:172.21.15.1:6790/0 mon.noname-b 2024-03-25T18:29:26.842 INFO:teuthology.misc.health.smithi001.stderr: 2024-03-25T18:29:26.843 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:29:26.842+0000 7fccc8f4b640 1 Processor -- start 2024-03-25T18:29:26.844 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:29:26.842+0000 7fccc8f4b640 1 -- start start 2024-03-25T18:29:26.844 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:29:26.842+0000 7fccc8f4b640 10 monclient: init 2024-03-25T18:29:26.844 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:29:26.842+0000 7fccc8f4b640 10 monclient: _reopen_session rank -1 2024-03-25T18:29:26.844 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:29:26.842+0000 7fccc8f4b640 10 monclient: _add_conns ranks=[0,1,2] 2024-03-25T18:29:26.844 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:29:26.842+0000 7fccc8f4b640 10 monclient(hunting): picked mon.noname-a con 0x7fccc40b4ff0 addr v1:172.21.15.1:6789/0 2024-03-25T18:29:26.844 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:29:26.842+0000 7fccc8f4b640 10 monclient(hunting): picked mon.noname-c con 0x7fccc40b4670 addr v1:172.21.15.115:6789/0 2024-03-25T18:29:26.844 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:29:26.842+0000 7fccc8f4b640 10 monclient(hunting): picked mon.noname-b con 0x7fccc413c480 addr v1:172.21.15.1:6790/0 2024-03-25T18:29:26.844 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:29:26.842+0000 7fccc8f4b640 1 -- --> v1:172.21.15.1:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fccc40ab0e0 con 0x7fccc40b4ff0 2024-03-25T18:29:26.844 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:29:26.842+0000 7fccc8f4b640 1 -- --> v1:172.21.15.115:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fccc40ab3c0 con 0x7fccc40b4670 2024-03-25T18:29:26.844 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:29:26.842+0000 7fccc8f4b640 1 -- --> v1:172.21.15.1:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fccc40a9f30 con 0x7fccc413c480 2024-03-25T18:29:26.844 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:29:26.842+0000 7fccc8f4b640 10 monclient(hunting): _renew_subs 2024-03-25T18:29:26.844 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:29:26.842+0000 7fccc8f4b640 10 monclient(hunting): authenticate will time out at 7647.312639s 2024-03-25T18:29:26.854 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:29:26.853+0000 7fccc1d9b640 1 --1- >> v1:172.21.15.115:6789/0 conn(0x7fccc40b4670 0x7fccc41425c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.115:6789/0 says I am v1:172.21.15.1:53316/0 (socket says 172.21.15.1:53316) 2024-03-25T18:29:26.854 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:29:26.853+0000 7fccc1d9b640 1 -- 172.21.15.1:0/4225733410 learned_addr learned my addr 172.21.15.1:0/4225733410 (peer_addr_for_me v1:172.21.15.1:0/0) 2024-03-25T18:29:26.863 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:29:26.863+0000 7fccc159a640 1 -- 172.21.15.1:0/4225733410 <== mon.0 v1:172.21.15.1:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 867983835 0 0) 0x7fccc40ab0e0 con 0x7fccc40b4ff0 2024-03-25T18:29:26.863 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:29:26.863+0000 7fccc159a640 10 monclient(hunting): _negotiate 2024-03-25T18:29:26.864 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:29:26.863+0000 7fccc159a640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:29:26.864 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:29:26.863+0000 7fccc159a640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:29:26.864 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:29:26.863+0000 7fccc159a640 10 monclient(hunting): my global_id is 4341 2024-03-25T18:29:26.864 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:29:26.863+0000 7fccc159a640 1 -- 172.21.15.1:0/4225733410 --> v1:172.21.15.1:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fccac003520 con 0x7fccc40b4ff0 2024-03-25T18:29:26.874 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:29:26.873+0000 7fccc159a640 1 -- 172.21.15.1:0/4225733410 <== mon.2 v1:172.21.15.1:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3203073605 0 0) 0x7fccc40a9f30 con 0x7fccc413c480 2024-03-25T18:29:26.874 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:29:26.873+0000 7fccc159a640 10 monclient(hunting): _negotiate 2024-03-25T18:29:26.874 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:29:26.873+0000 7fccc159a640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:29:26.874 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:29:26.873+0000 7fccc159a640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:29:26.874 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:29:26.873+0000 7fccc159a640 10 monclient(hunting): my global_id is 4343 2024-03-25T18:29:26.874 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:29:26.873+0000 7fccc159a640 1 -- 172.21.15.1:0/4225733410 --> v1:172.21.15.1:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fccc40ab0e0 con 0x7fccc413c480 2024-03-25T18:29:26.877 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:29:26.876+0000 7fccc159a640 1 -- 172.21.15.1:0/4225733410 <== mon.1 v1:172.21.15.115:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3916303006 0 0) 0x7fccc40ab3c0 con 0x7fccc40b4670 2024-03-25T18:29:26.877 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:29:26.876+0000 7fccc159a640 10 monclient(hunting): _negotiate 2024-03-25T18:29:26.877 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:29:26.876+0000 7fccc159a640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:29:26.877 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:29:26.876+0000 7fccc159a640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:29:26.877 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:29:26.876+0000 7fccc159a640 10 monclient(hunting): my global_id is 4336 2024-03-25T18:29:26.877 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:29:26.876+0000 7fccc159a640 1 -- 172.21.15.1:0/4225733410 --> v1:172.21.15.115:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fccc40a9f30 con 0x7fccc40b4670 2024-03-25T18:29:26.879 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:29:26.878+0000 7fccc159a640 1 -- 172.21.15.1:0/4225733410 <== mon.0 v1:172.21.15.1:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 899667443 0 0) 0x7fccac003520 con 0x7fccc40b4ff0 2024-03-25T18:29:26.879 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:29:26.878+0000 7fccc159a640 1 -- 172.21.15.1:0/4225733410 >> v1:172.21.15.1:6790/0 conn(0x7fccc413c480 legacy=0x7fccc414dc00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:29:26.879 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:29:26.878+0000 7fccc159a640 1 -- 172.21.15.1:0/4225733410 >> v1:172.21.15.115:6789/0 conn(0x7fccc40b4670 legacy=0x7fccc41425c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:29:26.879 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:29:26.878+0000 7fccc159a640 10 monclient: _finish_hunting 0 2024-03-25T18:29:26.879 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:29:26.878+0000 7fccc159a640 1 monclient: found mon.noname-a 2024-03-25T18:29:26.879 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:29:26.878+0000 7fccc159a640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:29:26.880 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:29:26.878+0000 7fccc159a640 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.1:6789/0 2024-03-25T18:29:26.880 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:29:26.878+0000 7fccc159a640 1 -- 172.21.15.1:0/4225733410 --> v1:172.21.15.1:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fccc40a9b70 con 0x7fccc40b4ff0 2024-03-25T18:29:26.880 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:29:26.878+0000 7fccc159a640 10 monclient: _finish_auth 0 2024-03-25T18:29:26.880 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:29:26.878+0000 7fccc159a640 10 monclient: _check_auth_tickets 2024-03-25T18:29:26.880 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:29:26.878+0000 7fccc159a640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:29:26.880 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:29:26.878+0000 7fccc159a640 1 -- 172.21.15.1:0/4225733410 <== mon.0 v1:172.21.15.1:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1586965025 0 0) 0x7fccb4001690 con 0x7fccc40b4ff0 2024-03-25T18:29:26.880 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:29:26.878+0000 7fccc159a640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:29:26.880 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:29:26.878+0000 7fccc159a640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-03-25T18:29:26.880 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:29:26.878+0000 7fccc159a640 10 monclient: dump: 2024-03-25T18:29:26.880 INFO:teuthology.misc.health.smithi001.stderr:epoch 1 2024-03-25T18:29:26.880 INFO:teuthology.misc.health.smithi001.stderr:fsid c8ea72cf-9e99-4383-9b52-6c23d365fdfc 2024-03-25T18:29:26.880 INFO:teuthology.misc.health.smithi001.stderr:last_changed 2024-03-25T18:25:50.510519+0000 2024-03-25T18:29:26.880 INFO:teuthology.misc.health.smithi001.stderr:created 2024-03-25T18:25:50.510519+0000 2024-03-25T18:29:26.880 INFO:teuthology.misc.health.smithi001.stderr:min_mon_release 19 (squid) 2024-03-25T18:29:26.881 INFO:teuthology.misc.health.smithi001.stderr:election_strategy: 3 2024-03-25T18:29:26.881 INFO:teuthology.misc.health.smithi001.stderr:0: v1:172.21.15.1:6789/0 mon.a 2024-03-25T18:29:26.881 INFO:teuthology.misc.health.smithi001.stderr:1: v1:172.21.15.115:6789/0 mon.b 2024-03-25T18:29:26.881 INFO:teuthology.misc.health.smithi001.stderr:2: v1:172.21.15.1:6790/0 mon.c 2024-03-25T18:29:26.881 INFO:teuthology.misc.health.smithi001.stderr: 2024-03-25T18:29:26.881 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:29:26.878+0000 7fccc8f4b640 5 monclient: authenticate success, global_id 4341 2024-03-25T18:29:26.881 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:29:26.878+0000 7fccc8f4b640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T18:29:26.884 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:29:26.883+0000 7fccc159a640 1 -- 172.21.15.1:0/4225733410 <== mon.0 v1:172.21.15.1:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fccb40017f0 con 0x7fccc40b4ff0 2024-03-25T18:29:26.884 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:29:26.883+0000 7fccc159a640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:29:26.884 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:29:26.883+0000 7fccc159a640 1 -- 172.21.15.1:0/4225733410 <== mon.0 v1:172.21.15.1:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1586965025 0 0) 0x7fccb4001c00 con 0x7fccc40b4ff0 2024-03-25T18:29:26.884 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:29:26.883+0000 7fccc8f4b640 10 monclient: get_monmap_and_config success 2024-03-25T18:29:26.884 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:29:26.883+0000 7fccc8f4b640 4 set_mon_vals no callback set 2024-03-25T18:29:26.884 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:29:26.883+0000 7fccc159a640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:29:26.884 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:29:26.883+0000 7fccc159a640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-25T18:29:26.884 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:29:26.883+0000 7fccc159a640 10 monclient: dump: 2024-03-25T18:29:26.884 INFO:teuthology.misc.health.smithi001.stderr:epoch 1 2024-03-25T18:29:26.884 INFO:teuthology.misc.health.smithi001.stderr:fsid c8ea72cf-9e99-4383-9b52-6c23d365fdfc 2024-03-25T18:29:26.885 INFO:teuthology.misc.health.smithi001.stderr:last_changed 2024-03-25T18:25:50.510519+0000 2024-03-25T18:29:26.885 INFO:teuthology.misc.health.smithi001.stderr:created 2024-03-25T18:25:50.510519+0000 2024-03-25T18:29:26.885 INFO:teuthology.misc.health.smithi001.stderr:min_mon_release 19 (squid) 2024-03-25T18:29:26.885 INFO:teuthology.misc.health.smithi001.stderr:election_strategy: 3 2024-03-25T18:29:26.885 INFO:teuthology.misc.health.smithi001.stderr:0: v1:172.21.15.1:6789/0 mon.a 2024-03-25T18:29:26.885 INFO:teuthology.misc.health.smithi001.stderr:1: v1:172.21.15.115:6789/0 mon.b 2024-03-25T18:29:26.885 INFO:teuthology.misc.health.smithi001.stderr:2: v1:172.21.15.1:6790/0 mon.c 2024-03-25T18:29:26.885 INFO:teuthology.misc.health.smithi001.stderr: 2024-03-25T18:29:26.886 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:29:26.884+0000 7fccc8f4b640 10 monclient: shutdown 2024-03-25T18:29:26.886 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:29:26.884+0000 7fccc8f4b640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:29:26.886 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:29:26.884+0000 7fccc8f4b640 1 -- 172.21.15.1:0/4225733410 >> v1:172.21.15.1:6789/0 conn(0x7fccc40b4ff0 legacy=0x7fccc413bd70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:29:26.886 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:29:26.884+0000 7fccc8f4b640 1 -- 172.21.15.1:0/4225733410 shutdown_connections 2024-03-25T18:29:26.886 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:29:26.884+0000 7fccc8f4b640 1 -- 172.21.15.1:0/4225733410 >> 172.21.15.1:0/4225733410 conn(0x7fccc40617a0 msgr2=0x7fccc4061bb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:29:26.886 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:29:26.884+0000 7fccc8f4b640 1 -- 172.21.15.1:0/4225733410 shutdown_connections 2024-03-25T18:29:26.886 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:29:26.884+0000 7fccc8f4b640 1 -- 172.21.15.1:0/4225733410 wait complete. 2024-03-25T18:29:26.886 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:29:26.885+0000 7fccc8f4b640 10 monclient: build_initial_monmap 2024-03-25T18:29:26.886 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:29:26.885+0000 7fccc8f4b640 10 monclient: monmap: 2024-03-25T18:29:26.886 INFO:teuthology.misc.health.smithi001.stderr:epoch 0 2024-03-25T18:29:26.886 INFO:teuthology.misc.health.smithi001.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T18:29:26.887 INFO:teuthology.misc.health.smithi001.stderr:last_changed 0.000000 2024-03-25T18:29:26.887 INFO:teuthology.misc.health.smithi001.stderr:created 0.000000 2024-03-25T18:29:26.887 INFO:teuthology.misc.health.smithi001.stderr:min_mon_release 0 (unknown) 2024-03-25T18:29:26.887 INFO:teuthology.misc.health.smithi001.stderr:election_strategy: 1 2024-03-25T18:29:26.887 INFO:teuthology.misc.health.smithi001.stderr:0: v1:172.21.15.1:6789/0 mon.noname-a 2024-03-25T18:29:26.887 INFO:teuthology.misc.health.smithi001.stderr:1: v1:172.21.15.115:6789/0 mon.noname-b 2024-03-25T18:29:26.887 INFO:teuthology.misc.health.smithi001.stderr:2: v1:172.21.15.1:6790/0 mon.noname-c 2024-03-25T18:29:26.887 INFO:teuthology.misc.health.smithi001.stderr: 2024-03-25T18:29:26.888 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:29:26.886+0000 7fccc8f4b640 1 librados: starting msgr at 2024-03-25T18:29:26.888 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:29:26.886+0000 7fccc8f4b640 1 librados: starting objecter 2024-03-25T18:29:26.889 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:29:26.886+0000 7fccc8f4b640 1 Processor -- start 2024-03-25T18:29:26.889 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:29:26.886+0000 7fccc8f4b640 1 -- start start 2024-03-25T18:29:26.889 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:29:26.886+0000 7fccc8f4b640 1 librados: setting wanted keys 2024-03-25T18:29:26.889 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:29:26.886+0000 7fccc8f4b640 1 librados: calling monclient init 2024-03-25T18:29:26.889 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:29:26.886+0000 7fccc8f4b640 10 monclient: init 2024-03-25T18:29:26.889 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:29:26.887+0000 7fccc8f4b640 10 monclient: _reopen_session rank -1 2024-03-25T18:29:26.889 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:29:26.887+0000 7fccc8f4b640 10 monclient: _add_conns ranks=[0,2,1] 2024-03-25T18:29:26.889 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:29:26.887+0000 7fccc8f4b640 10 monclient(hunting): picked mon.noname-a con 0x7fccc40b4670 addr v1:172.21.15.1:6789/0 2024-03-25T18:29:26.889 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:29:26.887+0000 7fccc8f4b640 10 monclient(hunting): picked mon.noname-c con 0x7fccc40b4ff0 addr v1:172.21.15.1:6790/0 2024-03-25T18:29:26.889 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:29:26.887+0000 7fccc8f4b640 10 monclient(hunting): picked mon.noname-b con 0x7fccc413c480 addr v1:172.21.15.115:6789/0 2024-03-25T18:29:26.889 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:29:26.887+0000 7fccc8f4b640 1 -- --> v1:172.21.15.1:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fccc413d0c0 con 0x7fccc40b4670 2024-03-25T18:29:26.889 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:29:26.887+0000 7fccc8f4b640 1 -- --> v1:172.21.15.115:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fccc4130cc0 con 0x7fccc413c480 2024-03-25T18:29:26.889 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:29:26.887+0000 7fccc8f4b640 1 -- --> v1:172.21.15.1:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fccc4146010 con 0x7fccc40b4ff0 2024-03-25T18:29:26.889 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:29:26.887+0000 7fccc8f4b640 10 monclient(hunting): _renew_subs 2024-03-25T18:29:26.890 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:29:26.887+0000 7fccc8f4b640 10 monclient(hunting): authenticate will time out at 7647.357115s 2024-03-25T18:29:26.895 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:29:26.894+0000 7fccc259c640 1 --1- >> v1:172.21.15.1:6789/0 conn(0x7fccc40b4670 0x7fccc41db480 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.1:6789/0 says I am v1:172.21.15.1:53594/0 (socket says 172.21.15.1:53594) 2024-03-25T18:29:26.895 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:29:26.895+0000 7fccc259c640 1 -- 172.21.15.1:0/943360828 learned_addr learned my addr 172.21.15.1:0/943360828 (peer_addr_for_me v1:172.21.15.1:0/0) 2024-03-25T18:29:26.896 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:29:26.895+0000 7fccb2ffd640 10 client.?.objecter ms_handle_connect 0x7fccc40b4670 2024-03-25T18:29:26.896 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:29:26.895+0000 7fccb2ffd640 10 client.?.objecter resend_mon_ops 2024-03-25T18:29:26.900 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:29:26.899+0000 7fccb2ffd640 10 client.?.objecter ms_handle_connect 0x7fccc413c480 2024-03-25T18:29:26.900 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:29:26.899+0000 7fccb2ffd640 10 client.?.objecter resend_mon_ops 2024-03-25T18:29:26.902 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:29:26.901+0000 7fccb2ffd640 1 -- 172.21.15.1:0/943360828 <== mon.0 v1:172.21.15.1:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 60974155 0 0) 0x7fccc413d0c0 con 0x7fccc40b4670 2024-03-25T18:29:26.902 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:29:26.901+0000 7fccb2ffd640 10 monclient(hunting): _negotiate 2024-03-25T18:29:26.902 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:29:26.901+0000 7fccb2ffd640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:29:26.902 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:29:26.901+0000 7fccb2ffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:29:26.902 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:29:26.901+0000 7fccb2ffd640 10 monclient(hunting): my global_id is 4344 2024-03-25T18:29:26.902 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:29:26.901+0000 7fccb2ffd640 1 -- 172.21.15.1:0/943360828 --> v1:172.21.15.1:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fcca4004eb0 con 0x7fccc40b4670 2024-03-25T18:29:26.911 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:29:26.911+0000 7fccb2ffd640 10 client.?.objecter ms_handle_connect 0x7fccc40b4ff0 2024-03-25T18:29:26.911 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:29:26.911+0000 7fccb2ffd640 10 client.?.objecter resend_mon_ops 2024-03-25T18:29:26.914 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:29:26.913+0000 7fccb2ffd640 1 -- 172.21.15.1:0/943360828 <== mon.1 v1:172.21.15.115:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3200138484 0 0) 0x7fccc4130cc0 con 0x7fccc413c480 2024-03-25T18:29:26.914 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:29:26.913+0000 7fccb2ffd640 10 monclient(hunting): _negotiate 2024-03-25T18:29:26.914 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:29:26.913+0000 7fccb2ffd640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:29:26.914 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:29:26.913+0000 7fccb2ffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:29:26.914 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:29:26.913+0000 7fccb2ffd640 10 monclient(hunting): my global_id is 4339 2024-03-25T18:29:26.914 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:29:26.913+0000 7fccb2ffd640 1 -- 172.21.15.1:0/943360828 --> v1:172.21.15.115:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fccc413d0c0 con 0x7fccc413c480 2024-03-25T18:29:26.914 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:29:26.913+0000 7fccb2ffd640 1 -- 172.21.15.1:0/943360828 <== mon.0 v1:172.21.15.1:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 4291551955 0 0) 0x7fcca4004eb0 con 0x7fccc40b4670 2024-03-25T18:29:26.915 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:29:26.913+0000 7fccb2ffd640 1 -- 172.21.15.1:0/943360828 --> v1:172.21.15.1:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fccc4130cc0 con 0x7fccc40b4670 2024-03-25T18:29:26.915 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:29:26.913+0000 7fccb2ffd640 1 -- 172.21.15.1:0/943360828 <== mon.0 v1:172.21.15.1:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1586965025 0 0) 0x7fccb4001bc0 con 0x7fccc40b4670 2024-03-25T18:29:26.915 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:29:26.913+0000 7fccb2ffd640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:29:26.915 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:29:26.913+0000 7fccb2ffd640 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2024-03-25T18:29:26.915 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:29:26.913+0000 7fccb2ffd640 10 monclient(hunting): dump: 2024-03-25T18:29:26.915 INFO:teuthology.misc.health.smithi001.stderr:epoch 1 2024-03-25T18:29:26.915 INFO:teuthology.misc.health.smithi001.stderr:fsid c8ea72cf-9e99-4383-9b52-6c23d365fdfc 2024-03-25T18:29:26.915 INFO:teuthology.misc.health.smithi001.stderr:last_changed 2024-03-25T18:25:50.510519+0000 2024-03-25T18:29:26.915 INFO:teuthology.misc.health.smithi001.stderr:created 2024-03-25T18:25:50.510519+0000 2024-03-25T18:29:26.915 INFO:teuthology.misc.health.smithi001.stderr:min_mon_release 19 (squid) 2024-03-25T18:29:26.915 INFO:teuthology.misc.health.smithi001.stderr:election_strategy: 3 2024-03-25T18:29:26.915 INFO:teuthology.misc.health.smithi001.stderr:0: v1:172.21.15.1:6789/0 mon.a 2024-03-25T18:29:26.916 INFO:teuthology.misc.health.smithi001.stderr:1: v1:172.21.15.115:6789/0 mon.b 2024-03-25T18:29:26.916 INFO:teuthology.misc.health.smithi001.stderr:2: v1:172.21.15.1:6790/0 mon.c 2024-03-25T18:29:26.916 INFO:teuthology.misc.health.smithi001.stderr: 2024-03-25T18:29:26.916 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:29:26.913+0000 7fccb2ffd640 10 monclient(hunting): _finish_auth 0 2024-03-25T18:29:26.925 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:29:26.924+0000 7fccb2ffd640 1 -- 172.21.15.1:0/943360828 <== mon.0 v1:172.21.15.1:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 3686727788 0 0) 0x7fccc4130cc0 con 0x7fccc40b4670 2024-03-25T18:29:26.925 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:29:26.924+0000 7fccb2ffd640 1 -- 172.21.15.1:0/943360828 >> v1:172.21.15.1:6790/0 conn(0x7fccc40b4ff0 legacy=0x7fccc41dbb90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:29:26.925 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:29:26.924+0000 7fccb2ffd640 1 -- 172.21.15.1:0/943360828 >> v1:172.21.15.115:6789/0 conn(0x7fccc413c480 legacy=0x7fccc41e2750 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:29:26.925 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:29:26.924+0000 7fccb2ffd640 10 monclient: _finish_hunting 0 2024-03-25T18:29:26.925 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:29:26.924+0000 7fccb2ffd640 1 monclient: found mon.a 2024-03-25T18:29:26.925 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:29:26.924+0000 7fccb2ffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:29:26.925 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:29:26.924+0000 7fccb2ffd640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.1:6789/0 2024-03-25T18:29:26.925 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:29:26.924+0000 7fccb2ffd640 1 -- 172.21.15.1:0/943360828 --> v1:172.21.15.1:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fccc413dbd0 con 0x7fccc40b4670 2024-03-25T18:29:26.925 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:29:26.924+0000 7fccb2ffd640 10 monclient: _finish_auth 0 2024-03-25T18:29:26.925 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:29:26.924+0000 7fccb2ffd640 10 monclient: _check_auth_tickets 2024-03-25T18:29:26.925 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:29:26.924+0000 7fccb2ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:29:26.925 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:29:26.924+0000 7fccc8f4b640 5 monclient: authenticate success, global_id 4344 2024-03-25T18:29:26.926 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:29:26.924+0000 7fccc8f4b640 10 monclient: _renew_subs 2024-03-25T18:29:26.926 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:29:26.924+0000 7fccc8f4b640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.1:6789/0 2024-03-25T18:29:26.926 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:29:26.924+0000 7fccc8f4b640 1 -- 172.21.15.1:0/943360828 --> v1:172.21.15.1:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fccc413deb0 con 0x7fccc40b4670 2024-03-25T18:29:26.926 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:29:26.924+0000 7fccc8f4b640 10 client.4344.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T18:29:26.926 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:29:26.924+0000 7fccc8f4b640 10 monclient: _renew_subs 2024-03-25T18:29:26.926 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:29:26.924+0000 7fccc8f4b640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.1:6789/0 2024-03-25T18:29:26.926 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:29:26.924+0000 7fccc8f4b640 1 -- 172.21.15.1:0/943360828 --> v1:172.21.15.1:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7fccc413e470 con 0x7fccc40b4670 2024-03-25T18:29:26.926 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:29:26.924+0000 7fccc8f4b640 1 librados: init done 2024-03-25T18:29:26.927 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:29:26.925+0000 7fccc8f4b640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T18:29:26.927 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:29:26.925+0000 7fccc8f4b640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:29:26.927 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:29:26.925+0000 7fccc8f4b640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.1:6789/0 2024-03-25T18:29:26.927 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:29:26.925+0000 7fccc8f4b640 1 -- 172.21.15.1:0/943360828 --> v1:172.21.15.1:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fccc406dbd0 con 0x7fccc40b4670 2024-03-25T18:29:27.261 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:29:27.260+0000 7fccb2ffd640 1 -- 172.21.15.1:0/943360828 <== mon.0 v1:172.21.15.1:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fccb40011f0 con 0x7fccc40b4670 2024-03-25T18:29:27.261 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:29:27.260+0000 7fccb2ffd640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:29:27.261 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:29:27.260+0000 7fccb2ffd640 1 -- 172.21.15.1:0/943360828 <== mon.0 v1:172.21.15.1:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1586965025 0 0) 0x7fccb4003b00 con 0x7fccc40b4670 2024-03-25T18:29:27.261 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:29:27.260+0000 7fccb2ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:29:27.261 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:29:27.260+0000 7fccb2ffd640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-25T18:29:27.261 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:29:27.260+0000 7fccb2ffd640 10 monclient: dump: 2024-03-25T18:29:27.261 INFO:teuthology.misc.health.smithi001.stderr:epoch 1 2024-03-25T18:29:27.261 INFO:teuthology.misc.health.smithi001.stderr:fsid c8ea72cf-9e99-4383-9b52-6c23d365fdfc 2024-03-25T18:29:27.262 INFO:teuthology.misc.health.smithi001.stderr:last_changed 2024-03-25T18:25:50.510519+0000 2024-03-25T18:29:27.262 INFO:teuthology.misc.health.smithi001.stderr:created 2024-03-25T18:25:50.510519+0000 2024-03-25T18:29:27.262 INFO:teuthology.misc.health.smithi001.stderr:min_mon_release 19 (squid) 2024-03-25T18:29:27.262 INFO:teuthology.misc.health.smithi001.stderr:election_strategy: 3 2024-03-25T18:29:27.262 INFO:teuthology.misc.health.smithi001.stderr:0: v1:172.21.15.1:6789/0 mon.a 2024-03-25T18:29:27.262 INFO:teuthology.misc.health.smithi001.stderr:1: v1:172.21.15.115:6789/0 mon.b 2024-03-25T18:29:27.262 INFO:teuthology.misc.health.smithi001.stderr:2: v1:172.21.15.1:6790/0 mon.c 2024-03-25T18:29:27.262 INFO:teuthology.misc.health.smithi001.stderr: 2024-03-25T18:29:27.262 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:29:27.260+0000 7fccc0d99640 4 set_mon_vals no callback set 2024-03-25T18:29:27.263 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:29:27.261+0000 7fccb2ffd640 1 -- 172.21.15.1:0/943360828 <== mon.0 v1:172.21.15.1:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 21787975 0 0) 0x7fccb4003d20 con 0x7fccc40b4670 2024-03-25T18:29:27.263 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:29:27.261+0000 7fccc1d9b640 10 monclient: get_auth_request con 0x7fcca406dd10 auth_method 0 2024-03-25T18:29:27.263 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:29:27.262+0000 7fccb2ffd640 10 client.4344.objecter ms_handle_connect 0x7fcca406dd10 2024-03-25T18:29:27.269 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:29:27.268+0000 7fccb2ffd640 1 -- 172.21.15.1:0/943360828 <== mon.0 v1:172.21.15.1:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 996254230 0 0) 0x7fccb40548d0 con 0x7fccc40b4670 2024-03-25T18:29:27.269 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:29:27.268+0000 7fccb2ffd640 10 client.4344.objecter ms_dispatch 0x7fccc4000b90 osd_map(10..10 src has 1..10) v4 2024-03-25T18:29:27.269 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:29:27.268+0000 7fccb2ffd640 3 client.4344.objecter handle_osd_map got epochs [10,10] > 0 2024-03-25T18:29:27.269 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:29:27.268+0000 7fccb2ffd640 3 client.4344.objecter handle_osd_map decoding full epoch 10 2024-03-25T18:29:27.269 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:29:27.268+0000 7fccb2ffd640 20 client.4344.objecter dump_active .. 0 homeless 2024-03-25T18:29:27.269 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:29:27.268+0000 7fccb2ffd640 1 -- 172.21.15.1:0/943360828 <== mon.0 v1:172.21.15.1:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7fccb40503e0 con 0x7fccc40b4670 2024-03-25T18:29:27.270 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:29:27.268+0000 7fccb2ffd640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:29:27.270 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:29:27.268+0000 7fccb2ffd640 10 monclient: _finish_command 1 = system:0 2024-03-25T18:29:27.533 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:29:27.532+0000 7fccc8f4b640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:29:27.533 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:29:27.532+0000 7fccc8f4b640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:29:27.533 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:29:27.532+0000 7fccc8f4b640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.1:6789/0 2024-03-25T18:29:27.533 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:29:27.532+0000 7fccc8f4b640 1 -- 172.21.15.1:0/943360828 --> v1:172.21.15.1:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fccc40b19b0 con 0x7fccc40b4670 2024-03-25T18:29:27.570 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:29:27.569+0000 7fccb2ffd640 1 -- 172.21.15.1:0/943360828 <== mon.0 v1:172.21.15.1:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 4092213348) 0x7fccb4052310 con 0x7fccc40b4670 2024-03-25T18:29:27.570 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:29:27.569+0000 7fccb2ffd640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:29:27.570 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:29:27.569+0000 7fccb2ffd640 10 monclient: _finish_command 2 = system:0 2024-03-25T18:29:27.571 INFO:teuthology.misc.health.smithi001.stdout: 2024-03-25T18:29:27.571 INFO:teuthology.misc.health.smithi001.stdout:{"epoch":10,"fsid":"c8ea72cf-9e99-4383-9b52-6c23d365fdfc","created":"2024-03-25T18:26:38.856638+0000","modified":"2024-03-25T18:26:44.224208+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T18:26:44.224208+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"b9f34922-3a84-49be-8426-807c1eceaefc","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"42bf306f-55e3-4634-9874-8b4b73ac6644","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"292cb23a-18a9-4207-a7cc-4d5727feaec0","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"45e0e9ab-898d-4673-9590-b4117bc5e84a","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"831a5a09-73ab-4a61-9872-088e76bdbc66","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"88cd903d-3b55-4840-9396-6e3cfe005282","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"89db7355-bc9d-46f4-8ab0-e8d9e744e026","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"cac47bee-0914-4f08-896f-348838882866","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T18:29:27.573 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:29:27.572+0000 7fccc8f4b640 10 librados: watch_flush enter 2024-03-25T18:29:27.573 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:29:27.572+0000 7fccc8f4b640 10 librados: watch_flush exit 2024-03-25T18:29:27.573 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:29:27.572+0000 7fccc8f4b640 20 client.4344.objecter shutdown clearing up homeless session... 2024-03-25T18:29:27.573 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:29:27.572+0000 7fccc8f4b640 10 client.4344.objecter successfully canceled tick 2024-03-25T18:29:27.573 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:29:27.572+0000 7fccc8f4b640 1 -- 172.21.15.1:0/943360828 >> v1:172.21.15.115:6800/4237379176 conn(0x7fcca406dd10 legacy=0x7fcca40701d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:29:27.573 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:29:27.572+0000 7fccc8f4b640 10 monclient: shutdown 2024-03-25T18:29:27.573 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:29:27.572+0000 7fccc8f4b640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:29:27.573 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:29:27.572+0000 7fccc8f4b640 1 -- 172.21.15.1:0/943360828 >> v1:172.21.15.1:6789/0 conn(0x7fccc40b4670 legacy=0x7fccc41db480 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:29:27.573 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:29:27.572+0000 7fccc8f4b640 1 -- 172.21.15.1:0/943360828 shutdown_connections 2024-03-25T18:29:27.574 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:29:27.572+0000 7fccc8f4b640 1 -- 172.21.15.1:0/943360828 >> 172.21.15.1:0/943360828 conn(0x7fccc40617a0 msgr2=0x7fccc4140760 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:29:27.574 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:29:27.572+0000 7fccc8f4b640 1 -- 172.21.15.1:0/943360828 shutdown_connections 2024-03-25T18:29:27.574 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:29:27.572+0000 7fccc8f4b640 1 -- 172.21.15.1:0/943360828 wait complete. 2024-03-25T18:29:27.574 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:29:27.572+0000 7fccc8f4b640 1 librados: shutdown 2024-03-25T18:29:27.585 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T18:29:34.393 DEBUG:teuthology.orchestra.run.smithi001:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T18:29:34.517 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:29:34.516+0000 7f7fd9a3e640 10 monclient: get_monmap_and_config 2024-03-25T18:29:34.517 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:29:34.516+0000 7f7fd9a3e640 10 monclient: build_initial_monmap 2024-03-25T18:29:34.517 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:29:34.516+0000 7f7fd9a3e640 10 monclient: monmap: 2024-03-25T18:29:34.517 INFO:teuthology.misc.health.smithi001.stderr:epoch 0 2024-03-25T18:29:34.517 INFO:teuthology.misc.health.smithi001.stderr:fsid c8ea72cf-9e99-4383-9b52-6c23d365fdfc 2024-03-25T18:29:34.517 INFO:teuthology.misc.health.smithi001.stderr:last_changed 2024-03-25T18:29:34.517014+0000 2024-03-25T18:29:34.517 INFO:teuthology.misc.health.smithi001.stderr:created 2024-03-25T18:29:34.517014+0000 2024-03-25T18:29:34.518 INFO:teuthology.misc.health.smithi001.stderr:min_mon_release 0 (unknown) 2024-03-25T18:29:34.518 INFO:teuthology.misc.health.smithi001.stderr:election_strategy: 3 2024-03-25T18:29:34.518 INFO:teuthology.misc.health.smithi001.stderr:0: v1:172.21.15.1:6789/0 mon.noname-a 2024-03-25T18:29:34.518 INFO:teuthology.misc.health.smithi001.stderr:1: v1:172.21.15.115:6789/0 mon.noname-c 2024-03-25T18:29:34.518 INFO:teuthology.misc.health.smithi001.stderr:2: v1:172.21.15.1:6790/0 mon.noname-b 2024-03-25T18:29:34.518 INFO:teuthology.misc.health.smithi001.stderr: 2024-03-25T18:29:34.519 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:29:34.517+0000 7f7fd9a3e640 1 Processor -- start 2024-03-25T18:29:34.519 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:29:34.517+0000 7f7fd9a3e640 1 -- start start 2024-03-25T18:29:34.519 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:29:34.517+0000 7f7fd9a3e640 10 monclient: init 2024-03-25T18:29:34.519 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:29:34.517+0000 7f7fd9a3e640 10 monclient: _reopen_session rank -1 2024-03-25T18:29:34.519 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:29:34.517+0000 7f7fd9a3e640 10 monclient: _add_conns ranks=[0,2,1] 2024-03-25T18:29:34.520 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:29:34.517+0000 7f7fd9a3e640 10 monclient(hunting): picked mon.noname-a con 0x7f7fd40b4ff0 addr v1:172.21.15.1:6789/0 2024-03-25T18:29:34.520 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:29:34.517+0000 7f7fd9a3e640 10 monclient(hunting): picked mon.noname-b con 0x7f7fd40b4670 addr v1:172.21.15.1:6790/0 2024-03-25T18:29:34.520 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:29:34.517+0000 7f7fd9a3e640 10 monclient(hunting): picked mon.noname-c con 0x7f7fd413c480 addr v1:172.21.15.115:6789/0 2024-03-25T18:29:34.520 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:29:34.517+0000 7f7fd9a3e640 1 -- --> v1:172.21.15.1:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7fd40ab0e0 con 0x7f7fd40b4ff0 2024-03-25T18:29:34.520 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:29:34.517+0000 7f7fd9a3e640 1 -- --> v1:172.21.15.115:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7fd40ab3c0 con 0x7f7fd413c480 2024-03-25T18:29:34.520 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:29:34.518+0000 7f7fd9a3e640 1 -- --> v1:172.21.15.1:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7fd40a9f30 con 0x7f7fd40b4670 2024-03-25T18:29:34.520 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:29:34.518+0000 7f7fd9a3e640 10 monclient(hunting): _renew_subs 2024-03-25T18:29:34.520 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:29:34.518+0000 7f7fd9a3e640 10 monclient(hunting): authenticate will time out at 7654.987698s 2024-03-25T18:29:34.530 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:29:34.529+0000 7f7fd37fe640 1 --1- >> v1:172.21.15.115:6789/0 conn(0x7f7fd413c480 0x7f7fd414dc00 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.115:6789/0 says I am v1:172.21.15.1:45874/0 (socket says 172.21.15.1:45874) 2024-03-25T18:29:34.531 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:29:34.529+0000 7f7fd37fe640 1 -- 172.21.15.1:0/1712128422 learned_addr learned my addr 172.21.15.1:0/1712128422 (peer_addr_for_me v1:172.21.15.1:0/0) 2024-03-25T18:29:34.538 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:29:34.537+0000 7f7fd27fc640 1 -- 172.21.15.1:0/1712128422 <== mon.1 v1:172.21.15.115:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1776350555 0 0) 0x7f7fd40ab3c0 con 0x7f7fd413c480 2024-03-25T18:29:34.538 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:29:34.537+0000 7f7fd27fc640 10 monclient(hunting): _negotiate 2024-03-25T18:29:34.539 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:29:34.537+0000 7f7fd27fc640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:29:34.539 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:29:34.537+0000 7f7fd27fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:29:34.539 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:29:34.537+0000 7f7fd27fc640 10 monclient(hunting): my global_id is 4342 2024-03-25T18:29:34.539 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:29:34.537+0000 7f7fd27fc640 1 -- 172.21.15.1:0/1712128422 --> v1:172.21.15.115:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f7fc0003520 con 0x7f7fd413c480 2024-03-25T18:29:34.541 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:29:34.540+0000 7f7fd27fc640 1 -- 172.21.15.1:0/1712128422 <== mon.2 v1:172.21.15.1:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1308051375 0 0) 0x7f7fd40a9f30 con 0x7f7fd40b4670 2024-03-25T18:29:34.541 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:29:34.540+0000 7f7fd27fc640 10 monclient(hunting): _negotiate 2024-03-25T18:29:34.541 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:29:34.540+0000 7f7fd27fc640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:29:34.541 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:29:34.540+0000 7f7fd27fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:29:34.541 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:29:34.540+0000 7f7fd27fc640 10 monclient(hunting): my global_id is 4349 2024-03-25T18:29:34.541 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:29:34.540+0000 7f7fd27fc640 1 -- 172.21.15.1:0/1712128422 --> v1:172.21.15.1:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f7fd40ab3c0 con 0x7f7fd40b4670 2024-03-25T18:29:34.546 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:29:34.545+0000 7f7fd27fc640 1 -- 172.21.15.1:0/1712128422 <== mon.1 v1:172.21.15.115:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3124168673 0 0) 0x7f7fc0003520 con 0x7f7fd413c480 2024-03-25T18:29:34.546 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:29:34.545+0000 7f7fd27fc640 1 -- 172.21.15.1:0/1712128422 >> v1:172.21.15.1:6790/0 conn(0x7f7fd40b4670 legacy=0x7f7fd41425c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:29:34.546 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:29:34.545+0000 7f7fd27fc640 1 -- 172.21.15.1:0/1712128422 >> v1:172.21.15.1:6789/0 conn(0x7f7fd40b4ff0 legacy=0x7f7fd413bd70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:29:34.546 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:29:34.545+0000 7f7fd27fc640 10 monclient: _finish_hunting 0 2024-03-25T18:29:34.546 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:29:34.545+0000 7f7fd27fc640 1 monclient: found mon.noname-c 2024-03-25T18:29:34.546 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:29:34.545+0000 7f7fd27fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:29:34.546 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:29:34.545+0000 7f7fd27fc640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.115:6789/0 2024-03-25T18:29:34.546 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:29:34.545+0000 7f7fd27fc640 1 -- 172.21.15.1:0/1712128422 --> v1:172.21.15.115:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7fd40a9b70 con 0x7f7fd413c480 2024-03-25T18:29:34.546 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:29:34.545+0000 7f7fd27fc640 10 monclient: _finish_auth 0 2024-03-25T18:29:34.546 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:29:34.545+0000 7f7fd27fc640 10 monclient: _check_auth_tickets 2024-03-25T18:29:34.546 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:29:34.545+0000 7f7fd27fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:29:34.547 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:29:34.545+0000 7f7fd27fc640 1 -- 172.21.15.1:0/1712128422 <== mon.1 v1:172.21.15.115:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1586965025 0 0) 0x7f7fcc001a10 con 0x7f7fd413c480 2024-03-25T18:29:34.547 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:29:34.545+0000 7f7fd9a3e640 5 monclient: authenticate success, global_id 4342 2024-03-25T18:29:34.547 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:29:34.545+0000 7f7fd9a3e640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T18:29:34.547 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:29:34.545+0000 7f7fd27fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:29:34.547 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:29:34.545+0000 7f7fd27fc640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-03-25T18:29:34.547 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:29:34.545+0000 7f7fd27fc640 10 monclient: dump: 2024-03-25T18:29:34.547 INFO:teuthology.misc.health.smithi001.stderr:epoch 1 2024-03-25T18:29:34.547 INFO:teuthology.misc.health.smithi001.stderr:fsid c8ea72cf-9e99-4383-9b52-6c23d365fdfc 2024-03-25T18:29:34.547 INFO:teuthology.misc.health.smithi001.stderr:last_changed 2024-03-25T18:25:50.510519+0000 2024-03-25T18:29:34.547 INFO:teuthology.misc.health.smithi001.stderr:created 2024-03-25T18:25:50.510519+0000 2024-03-25T18:29:34.547 INFO:teuthology.misc.health.smithi001.stderr:min_mon_release 19 (squid) 2024-03-25T18:29:34.547 INFO:teuthology.misc.health.smithi001.stderr:election_strategy: 3 2024-03-25T18:29:34.547 INFO:teuthology.misc.health.smithi001.stderr:0: v1:172.21.15.1:6789/0 mon.a 2024-03-25T18:29:34.547 INFO:teuthology.misc.health.smithi001.stderr:1: v1:172.21.15.115:6789/0 mon.b 2024-03-25T18:29:34.547 INFO:teuthology.misc.health.smithi001.stderr:2: v1:172.21.15.1:6790/0 mon.c 2024-03-25T18:29:34.548 INFO:teuthology.misc.health.smithi001.stderr: 2024-03-25T18:29:34.548 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:29:34.545+0000 7f7fd9a3e640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T18:29:34.550 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:29:34.549+0000 7f7fd27fc640 1 -- 172.21.15.1:0/1712128422 <== mon.1 v1:172.21.15.115:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f7fcc001b70 con 0x7f7fd413c480 2024-03-25T18:29:34.550 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:29:34.549+0000 7f7fd27fc640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:29:34.550 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:29:34.549+0000 7f7fd27fc640 1 -- 172.21.15.1:0/1712128422 <== mon.1 v1:172.21.15.115:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1586965025 0 0) 0x7f7fcc001f80 con 0x7f7fd413c480 2024-03-25T18:29:34.550 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:29:34.549+0000 7f7fd9a3e640 10 monclient: get_monmap_and_config success 2024-03-25T18:29:34.550 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:29:34.549+0000 7f7fd9a3e640 4 set_mon_vals no callback set 2024-03-25T18:29:34.550 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:29:34.549+0000 7f7fd27fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:29:34.551 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:29:34.549+0000 7f7fd27fc640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-25T18:29:34.551 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:29:34.549+0000 7f7fd27fc640 10 monclient: dump: 2024-03-25T18:29:34.551 INFO:teuthology.misc.health.smithi001.stderr:epoch 1 2024-03-25T18:29:34.551 INFO:teuthology.misc.health.smithi001.stderr:fsid c8ea72cf-9e99-4383-9b52-6c23d365fdfc 2024-03-25T18:29:34.551 INFO:teuthology.misc.health.smithi001.stderr:last_changed 2024-03-25T18:25:50.510519+0000 2024-03-25T18:29:34.551 INFO:teuthology.misc.health.smithi001.stderr:created 2024-03-25T18:25:50.510519+0000 2024-03-25T18:29:34.551 INFO:teuthology.misc.health.smithi001.stderr:min_mon_release 19 (squid) 2024-03-25T18:29:34.551 INFO:teuthology.misc.health.smithi001.stderr:election_strategy: 3 2024-03-25T18:29:34.551 INFO:teuthology.misc.health.smithi001.stderr:0: v1:172.21.15.1:6789/0 mon.a 2024-03-25T18:29:34.551 INFO:teuthology.misc.health.smithi001.stderr:1: v1:172.21.15.115:6789/0 mon.b 2024-03-25T18:29:34.551 INFO:teuthology.misc.health.smithi001.stderr:2: v1:172.21.15.1:6790/0 mon.c 2024-03-25T18:29:34.551 INFO:teuthology.misc.health.smithi001.stderr: 2024-03-25T18:29:34.553 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:29:34.550+0000 7f7fd9a3e640 10 monclient: shutdown 2024-03-25T18:29:34.553 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:29:34.550+0000 7f7fd9a3e640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:29:34.553 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:29:34.550+0000 7f7fd9a3e640 1 -- 172.21.15.1:0/1712128422 >> v1:172.21.15.115:6789/0 conn(0x7f7fd413c480 legacy=0x7f7fd414dc00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:29:34.553 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:29:34.550+0000 7f7fd9a3e640 1 -- 172.21.15.1:0/1712128422 shutdown_connections 2024-03-25T18:29:34.553 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:29:34.550+0000 7f7fd9a3e640 1 -- 172.21.15.1:0/1712128422 >> 172.21.15.1:0/1712128422 conn(0x7f7fd40617a0 msgr2=0x7f7fd4061bb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:29:34.553 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:29:34.550+0000 7f7fd9a3e640 1 -- 172.21.15.1:0/1712128422 shutdown_connections 2024-03-25T18:29:34.553 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:29:34.550+0000 7f7fd9a3e640 1 -- 172.21.15.1:0/1712128422 wait complete. 2024-03-25T18:29:34.553 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:29:34.551+0000 7f7fd9a3e640 10 monclient: build_initial_monmap 2024-03-25T18:29:34.554 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:29:34.551+0000 7f7fd9a3e640 10 monclient: monmap: 2024-03-25T18:29:34.554 INFO:teuthology.misc.health.smithi001.stderr:epoch 0 2024-03-25T18:29:34.554 INFO:teuthology.misc.health.smithi001.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T18:29:34.554 INFO:teuthology.misc.health.smithi001.stderr:last_changed 0.000000 2024-03-25T18:29:34.554 INFO:teuthology.misc.health.smithi001.stderr:created 0.000000 2024-03-25T18:29:34.554 INFO:teuthology.misc.health.smithi001.stderr:min_mon_release 0 (unknown) 2024-03-25T18:29:34.554 INFO:teuthology.misc.health.smithi001.stderr:election_strategy: 1 2024-03-25T18:29:34.554 INFO:teuthology.misc.health.smithi001.stderr:0: v1:172.21.15.1:6789/0 mon.noname-a 2024-03-25T18:29:34.554 INFO:teuthology.misc.health.smithi001.stderr:1: v1:172.21.15.115:6789/0 mon.noname-b 2024-03-25T18:29:34.554 INFO:teuthology.misc.health.smithi001.stderr:2: v1:172.21.15.1:6790/0 mon.noname-c 2024-03-25T18:29:34.554 INFO:teuthology.misc.health.smithi001.stderr: 2024-03-25T18:29:34.554 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:29:34.552+0000 7f7fd9a3e640 1 librados: starting msgr at 2024-03-25T18:29:34.554 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:29:34.552+0000 7f7fd9a3e640 1 librados: starting objecter 2024-03-25T18:29:34.554 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:29:34.552+0000 7f7fd9a3e640 1 Processor -- start 2024-03-25T18:29:34.555 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:29:34.552+0000 7f7fd9a3e640 1 -- start start 2024-03-25T18:29:34.555 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:29:34.552+0000 7f7fd9a3e640 1 librados: setting wanted keys 2024-03-25T18:29:34.555 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:29:34.552+0000 7f7fd9a3e640 1 librados: calling monclient init 2024-03-25T18:29:34.555 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:29:34.552+0000 7f7fd9a3e640 10 monclient: init 2024-03-25T18:29:34.555 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:29:34.552+0000 7f7fd9a3e640 10 monclient: _reopen_session rank -1 2024-03-25T18:29:34.555 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:29:34.552+0000 7f7fd9a3e640 10 monclient: _add_conns ranks=[0,1,2] 2024-03-25T18:29:34.555 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:29:34.552+0000 7f7fd9a3e640 10 monclient(hunting): picked mon.noname-a con 0x7f7fd40b4670 addr v1:172.21.15.1:6789/0 2024-03-25T18:29:34.555 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:29:34.552+0000 7f7fd9a3e640 10 monclient(hunting): picked mon.noname-b con 0x7f7fd40b4ff0 addr v1:172.21.15.115:6789/0 2024-03-25T18:29:34.555 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:29:34.552+0000 7f7fd9a3e640 10 monclient(hunting): picked mon.noname-c con 0x7f7fd413c480 addr v1:172.21.15.1:6790/0 2024-03-25T18:29:34.555 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:29:34.552+0000 7f7fd9a3e640 1 -- --> v1:172.21.15.1:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7fd40ae560 con 0x7f7fd40b4670 2024-03-25T18:29:34.555 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:29:34.552+0000 7f7fd9a3e640 1 -- --> v1:172.21.15.115:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7fd40ad250 con 0x7f7fd40b4ff0 2024-03-25T18:29:34.555 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:29:34.552+0000 7f7fd9a3e640 1 -- --> v1:172.21.15.1:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7fd40b04b0 con 0x7f7fd413c480 2024-03-25T18:29:34.555 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:29:34.552+0000 7f7fd9a3e640 10 monclient(hunting): _renew_subs 2024-03-25T18:29:34.555 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:29:34.552+0000 7f7fd9a3e640 10 monclient(hunting): authenticate will time out at 7655.022622s 2024-03-25T18:29:34.563 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:29:34.562+0000 7f7fd37fe640 1 --1- >> v1:172.21.15.1:6790/0 conn(0x7f7fd413c480 0x7f7fd40af100 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.1:6790/0 says I am v1:172.21.15.1:48736/0 (socket says 172.21.15.1:48736) 2024-03-25T18:29:34.563 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:29:34.562+0000 7f7fd37fe640 1 -- 172.21.15.1:0/11624269 learned_addr learned my addr 172.21.15.1:0/11624269 (peer_addr_for_me v1:172.21.15.1:0/0) 2024-03-25T18:29:34.563 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:29:34.562+0000 7f7fd0ff9640 10 client.?.objecter ms_handle_connect 0x7f7fd413c480 2024-03-25T18:29:34.563 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:29:34.562+0000 7f7fd0ff9640 10 client.?.objecter resend_mon_ops 2024-03-25T18:29:34.573 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:29:34.573+0000 7f7fd0ff9640 10 client.?.objecter ms_handle_connect 0x7f7fd40b4670 2024-03-25T18:29:34.574 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:29:34.573+0000 7f7fd0ff9640 10 client.?.objecter resend_mon_ops 2024-03-25T18:29:34.575 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:29:34.575+0000 7f7fd0ff9640 10 client.?.objecter ms_handle_connect 0x7f7fd40b4ff0 2024-03-25T18:29:34.576 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:29:34.575+0000 7f7fd0ff9640 10 client.?.objecter resend_mon_ops 2024-03-25T18:29:34.576 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:29:34.575+0000 7f7fd0ff9640 1 -- 172.21.15.1:0/11624269 <== mon.2 v1:172.21.15.1:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3309975152 0 0) 0x7f7fd40b04b0 con 0x7f7fd413c480 2024-03-25T18:29:34.576 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:29:34.575+0000 7f7fd0ff9640 10 monclient(hunting): _negotiate 2024-03-25T18:29:34.576 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:29:34.575+0000 7f7fd0ff9640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:29:34.576 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:29:34.575+0000 7f7fd0ff9640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:29:34.577 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:29:34.575+0000 7f7fd0ff9640 10 monclient(hunting): my global_id is 4352 2024-03-25T18:29:34.577 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:29:34.576+0000 7f7fd0ff9640 1 -- 172.21.15.1:0/11624269 --> v1:172.21.15.1:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f7fb4004eb0 con 0x7f7fd413c480 2024-03-25T18:29:34.582 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:29:34.582+0000 7f7fd0ff9640 1 -- 172.21.15.1:0/11624269 <== mon.0 v1:172.21.15.1:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2273678910 0 0) 0x7f7fd40ae560 con 0x7f7fd40b4670 2024-03-25T18:29:34.583 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:29:34.582+0000 7f7fd0ff9640 10 monclient(hunting): _negotiate 2024-03-25T18:29:34.583 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:29:34.582+0000 7f7fd0ff9640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:29:34.583 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:29:34.582+0000 7f7fd0ff9640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:29:34.583 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:29:34.582+0000 7f7fd0ff9640 10 monclient(hunting): my global_id is 4350 2024-03-25T18:29:34.583 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:29:34.582+0000 7f7fd0ff9640 1 -- 172.21.15.1:0/11624269 --> v1:172.21.15.1:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f7fd40b04b0 con 0x7f7fd40b4670 2024-03-25T18:29:34.584 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:29:34.584+0000 7f7fd0ff9640 1 -- 172.21.15.1:0/11624269 <== mon.1 v1:172.21.15.115:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1660684091 0 0) 0x7f7fd40ad250 con 0x7f7fd40b4ff0 2024-03-25T18:29:34.584 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:29:34.584+0000 7f7fd0ff9640 10 monclient(hunting): _negotiate 2024-03-25T18:29:34.585 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:29:34.584+0000 7f7fd0ff9640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:29:34.585 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:29:34.584+0000 7f7fd0ff9640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:29:34.585 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:29:34.584+0000 7f7fd0ff9640 10 monclient(hunting): my global_id is 4345 2024-03-25T18:29:34.585 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:29:34.584+0000 7f7fd0ff9640 1 -- 172.21.15.1:0/11624269 --> v1:172.21.15.115:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f7fd40ae560 con 0x7f7fd40b4ff0 2024-03-25T18:29:34.589 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:29:34.588+0000 7f7fd0ff9640 1 -- 172.21.15.1:0/11624269 <== mon.2 v1:172.21.15.1:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3278075606 0 0) 0x7f7fb4004eb0 con 0x7f7fd413c480 2024-03-25T18:29:34.589 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:29:34.589+0000 7f7fd0ff9640 1 -- 172.21.15.1:0/11624269 --> v1:172.21.15.1:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f7fd40ad250 con 0x7f7fd413c480 2024-03-25T18:29:34.589 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:29:34.589+0000 7f7fd0ff9640 1 -- 172.21.15.1:0/11624269 <== mon.2 v1:172.21.15.1:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1586965025 0 0) 0x7f7fcc001ec0 con 0x7f7fd413c480 2024-03-25T18:29:34.589 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:29:34.589+0000 7f7fd0ff9640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:29:34.590 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:29:34.589+0000 7f7fd0ff9640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-03-25T18:29:34.590 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:29:34.589+0000 7f7fd0ff9640 10 monclient(hunting): dump: 2024-03-25T18:29:34.590 INFO:teuthology.misc.health.smithi001.stderr:epoch 1 2024-03-25T18:29:34.590 INFO:teuthology.misc.health.smithi001.stderr:fsid c8ea72cf-9e99-4383-9b52-6c23d365fdfc 2024-03-25T18:29:34.590 INFO:teuthology.misc.health.smithi001.stderr:last_changed 2024-03-25T18:25:50.510519+0000 2024-03-25T18:29:34.590 INFO:teuthology.misc.health.smithi001.stderr:created 2024-03-25T18:25:50.510519+0000 2024-03-25T18:29:34.590 INFO:teuthology.misc.health.smithi001.stderr:min_mon_release 19 (squid) 2024-03-25T18:29:34.590 INFO:teuthology.misc.health.smithi001.stderr:election_strategy: 3 2024-03-25T18:29:34.590 INFO:teuthology.misc.health.smithi001.stderr:0: v1:172.21.15.1:6789/0 mon.a 2024-03-25T18:29:34.590 INFO:teuthology.misc.health.smithi001.stderr:1: v1:172.21.15.115:6789/0 mon.b 2024-03-25T18:29:34.590 INFO:teuthology.misc.health.smithi001.stderr:2: v1:172.21.15.1:6790/0 mon.c 2024-03-25T18:29:34.590 INFO:teuthology.misc.health.smithi001.stderr: 2024-03-25T18:29:34.590 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:29:34.589+0000 7f7fd0ff9640 10 monclient(hunting): _finish_auth 0 2024-03-25T18:29:34.591 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:29:34.590+0000 7f7fd0ff9640 1 -- 172.21.15.1:0/11624269 <== mon.0 v1:172.21.15.1:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 935491607 0 0) 0x7f7fd40b04b0 con 0x7f7fd40b4670 2024-03-25T18:29:34.591 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:29:34.590+0000 7f7fd0ff9640 1 -- 172.21.15.1:0/11624269 --> v1:172.21.15.1:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f7fb4004eb0 con 0x7f7fd40b4670 2024-03-25T18:29:34.591 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:29:34.590+0000 7f7fd0ff9640 1 -- 172.21.15.1:0/11624269 <== mon.0 v1:172.21.15.1:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1586965025 0 0) 0x7f7fc40018b0 con 0x7f7fd40b4670 2024-03-25T18:29:34.591 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:29:34.590+0000 7f7fd0ff9640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:29:34.591 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:29:34.590+0000 7f7fd0ff9640 10 monclient(hunting): got monmap 1 from mon.a (according to old e1) 2024-03-25T18:29:34.591 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:29:34.590+0000 7f7fd0ff9640 10 monclient(hunting): dump: 2024-03-25T18:29:34.591 INFO:teuthology.misc.health.smithi001.stderr:epoch 1 2024-03-25T18:29:34.592 INFO:teuthology.misc.health.smithi001.stderr:fsid c8ea72cf-9e99-4383-9b52-6c23d365fdfc 2024-03-25T18:29:34.592 INFO:teuthology.misc.health.smithi001.stderr:last_changed 2024-03-25T18:25:50.510519+0000 2024-03-25T18:29:34.592 INFO:teuthology.misc.health.smithi001.stderr:created 2024-03-25T18:25:50.510519+0000 2024-03-25T18:29:34.592 INFO:teuthology.misc.health.smithi001.stderr:min_mon_release 19 (squid) 2024-03-25T18:29:34.592 INFO:teuthology.misc.health.smithi001.stderr:election_strategy: 3 2024-03-25T18:29:34.592 INFO:teuthology.misc.health.smithi001.stderr:0: v1:172.21.15.1:6789/0 mon.a 2024-03-25T18:29:34.592 INFO:teuthology.misc.health.smithi001.stderr:1: v1:172.21.15.115:6789/0 mon.b 2024-03-25T18:29:34.592 INFO:teuthology.misc.health.smithi001.stderr:2: v1:172.21.15.1:6790/0 mon.c 2024-03-25T18:29:34.592 INFO:teuthology.misc.health.smithi001.stderr: 2024-03-25T18:29:34.595 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:29:34.594+0000 7f7fd0ff9640 1 -- 172.21.15.1:0/11624269 <== mon.1 v1:172.21.15.115:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 900696975 0 0) 0x7f7fd40ae560 con 0x7f7fd40b4ff0 2024-03-25T18:29:34.595 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:29:34.594+0000 7f7fd0ff9640 1 -- 172.21.15.1:0/11624269 --> v1:172.21.15.115:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f7fd40b04b0 con 0x7f7fd40b4ff0 2024-03-25T18:29:34.595 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:29:34.595+0000 7f7fd0ff9640 1 -- 172.21.15.1:0/11624269 <== mon.1 v1:172.21.15.115:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1586965025 0 0) 0x7f7fbc001930 con 0x7f7fd40b4ff0 2024-03-25T18:29:34.595 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:29:34.595+0000 7f7fd0ff9640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:29:34.596 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:29:34.595+0000 7f7fd0ff9640 10 monclient(hunting): got monmap 1 from mon.b (according to old e1) 2024-03-25T18:29:34.596 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:29:34.595+0000 7f7fd0ff9640 10 monclient(hunting): dump: 2024-03-25T18:29:34.596 INFO:teuthology.misc.health.smithi001.stderr:epoch 1 2024-03-25T18:29:34.596 INFO:teuthology.misc.health.smithi001.stderr:fsid c8ea72cf-9e99-4383-9b52-6c23d365fdfc 2024-03-25T18:29:34.596 INFO:teuthology.misc.health.smithi001.stderr:last_changed 2024-03-25T18:25:50.510519+0000 2024-03-25T18:29:34.596 INFO:teuthology.misc.health.smithi001.stderr:created 2024-03-25T18:25:50.510519+0000 2024-03-25T18:29:34.596 INFO:teuthology.misc.health.smithi001.stderr:min_mon_release 19 (squid) 2024-03-25T18:29:34.596 INFO:teuthology.misc.health.smithi001.stderr:election_strategy: 3 2024-03-25T18:29:34.596 INFO:teuthology.misc.health.smithi001.stderr:0: v1:172.21.15.1:6789/0 mon.a 2024-03-25T18:29:34.596 INFO:teuthology.misc.health.smithi001.stderr:1: v1:172.21.15.115:6789/0 mon.b 2024-03-25T18:29:34.596 INFO:teuthology.misc.health.smithi001.stderr:2: v1:172.21.15.1:6790/0 mon.c 2024-03-25T18:29:34.596 INFO:teuthology.misc.health.smithi001.stderr: 2024-03-25T18:29:34.598 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:29:34.596+0000 7f7fd0ff9640 1 -- 172.21.15.1:0/11624269 <== mon.0 v1:172.21.15.1:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 1762070387 0 0) 0x7f7fb4004eb0 con 0x7f7fd40b4670 2024-03-25T18:29:34.598 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:29:34.596+0000 7f7fd0ff9640 1 -- 172.21.15.1:0/11624269 >> v1:172.21.15.1:6790/0 conn(0x7f7fd413c480 legacy=0x7f7fd40af100 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:29:34.598 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:29:34.596+0000 7f7fd0ff9640 1 -- 172.21.15.1:0/11624269 >> v1:172.21.15.115:6789/0 conn(0x7f7fd40b4ff0 legacy=0x7f7fd40ae9f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:29:34.598 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:29:34.596+0000 7f7fd0ff9640 10 monclient: _finish_hunting 0 2024-03-25T18:29:34.598 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:29:34.596+0000 7f7fd0ff9640 1 monclient: found mon.a 2024-03-25T18:29:34.598 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:29:34.596+0000 7f7fd0ff9640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:29:34.598 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:29:34.596+0000 7f7fd0ff9640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.1:6789/0 2024-03-25T18:29:34.598 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:29:34.596+0000 7f7fd0ff9640 1 -- 172.21.15.1:0/11624269 --> v1:172.21.15.1:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7fd40b0790 con 0x7f7fd40b4670 2024-03-25T18:29:34.598 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:29:34.596+0000 7f7fd0ff9640 10 monclient: _finish_auth 0 2024-03-25T18:29:34.598 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:29:34.596+0000 7f7fd0ff9640 10 monclient: _check_auth_tickets 2024-03-25T18:29:34.598 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:29:34.596+0000 7f7fd0ff9640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:29:34.599 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:29:34.596+0000 7f7fd9a3e640 5 monclient: authenticate success, global_id 4350 2024-03-25T18:29:34.599 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:29:34.596+0000 7f7fd9a3e640 10 monclient: _renew_subs 2024-03-25T18:29:34.599 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:29:34.596+0000 7f7fd9a3e640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.1:6789/0 2024-03-25T18:29:34.599 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:29:34.596+0000 7f7fd9a3e640 1 -- 172.21.15.1:0/11624269 --> v1:172.21.15.1:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7fd40b0a70 con 0x7f7fd40b4670 2024-03-25T18:29:34.599 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:29:34.596+0000 7f7fd9a3e640 10 client.4350.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T18:29:34.599 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:29:34.596+0000 7f7fd9a3e640 10 monclient: _renew_subs 2024-03-25T18:29:34.599 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:29:34.596+0000 7f7fd9a3e640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.1:6789/0 2024-03-25T18:29:34.599 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:29:34.596+0000 7f7fd9a3e640 1 -- 172.21.15.1:0/11624269 --> v1:172.21.15.1:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f7fd41e35a0 con 0x7f7fd40b4670 2024-03-25T18:29:34.599 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:29:34.596+0000 7f7fd9a3e640 1 librados: init done 2024-03-25T18:29:34.599 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:29:34.597+0000 7f7fd9a3e640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T18:29:34.599 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:29:34.597+0000 7f7fd9a3e640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:29:34.599 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:29:34.597+0000 7f7fd9a3e640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.1:6789/0 2024-03-25T18:29:34.599 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:29:34.597+0000 7f7fd9a3e640 1 -- 172.21.15.1:0/11624269 --> v1:172.21.15.1:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7fd40a9880 con 0x7f7fd40b4670 2024-03-25T18:29:34.817 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:29:34.817+0000 7f7fd0ff9640 1 -- 172.21.15.1:0/11624269 <== mon.0 v1:172.21.15.1:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f7fc40035f0 con 0x7f7fd40b4670 2024-03-25T18:29:34.818 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:29:34.817+0000 7f7fd0ff9640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:29:34.818 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:29:34.817+0000 7f7fd0ff9640 1 -- 172.21.15.1:0/11624269 <== mon.0 v1:172.21.15.1:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1586965025 0 0) 0x7f7fc4003900 con 0x7f7fd40b4670 2024-03-25T18:29:34.818 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:29:34.817+0000 7f7fd0ff9640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:29:34.818 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:29:34.817+0000 7f7fd0ff9640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-25T18:29:34.818 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:29:34.817+0000 7f7fd0ff9640 10 monclient: dump: 2024-03-25T18:29:34.818 INFO:teuthology.misc.health.smithi001.stderr:epoch 1 2024-03-25T18:29:34.818 INFO:teuthology.misc.health.smithi001.stderr:fsid c8ea72cf-9e99-4383-9b52-6c23d365fdfc 2024-03-25T18:29:34.818 INFO:teuthology.misc.health.smithi001.stderr:last_changed 2024-03-25T18:25:50.510519+0000 2024-03-25T18:29:34.818 INFO:teuthology.misc.health.smithi001.stderr:created 2024-03-25T18:25:50.510519+0000 2024-03-25T18:29:34.818 INFO:teuthology.misc.health.smithi001.stderr:min_mon_release 19 (squid) 2024-03-25T18:29:34.818 INFO:teuthology.misc.health.smithi001.stderr:election_strategy: 3 2024-03-25T18:29:34.818 INFO:teuthology.misc.health.smithi001.stderr:0: v1:172.21.15.1:6789/0 mon.a 2024-03-25T18:29:34.818 INFO:teuthology.misc.health.smithi001.stderr:1: v1:172.21.15.115:6789/0 mon.b 2024-03-25T18:29:34.819 INFO:teuthology.misc.health.smithi001.stderr:2: v1:172.21.15.1:6790/0 mon.c 2024-03-25T18:29:34.819 INFO:teuthology.misc.health.smithi001.stderr: 2024-03-25T18:29:34.819 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:29:34.817+0000 7f7fd1ffb640 4 set_mon_vals no callback set 2024-03-25T18:29:34.917 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:29:34.916+0000 7f7fd0ff9640 1 -- 172.21.15.1:0/11624269 <== mon.0 v1:172.21.15.1:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 21787975 0 0) 0x7f7fc40198d0 con 0x7f7fd40b4670 2024-03-25T18:29:34.917 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:29:34.916+0000 7f7fcbfff640 10 monclient: get_auth_request con 0x7f7fb406e160 auth_method 0 2024-03-25T18:29:34.917 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:29:34.917+0000 7f7fd0ff9640 10 client.4350.objecter ms_handle_connect 0x7f7fb406e160 2024-03-25T18:29:34.924 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:29:34.923+0000 7f7fd0ff9640 1 -- 172.21.15.1:0/11624269 <== mon.0 v1:172.21.15.1:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 996254230 0 0) 0x7f7fc4053550 con 0x7f7fd40b4670 2024-03-25T18:29:34.924 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:29:34.923+0000 7f7fd0ff9640 10 client.4350.objecter ms_dispatch 0x7f7fd4000b90 osd_map(10..10 src has 1..10) v4 2024-03-25T18:29:34.924 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:29:34.923+0000 7f7fd0ff9640 3 client.4350.objecter handle_osd_map got epochs [10,10] > 0 2024-03-25T18:29:34.924 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:29:34.923+0000 7f7fd0ff9640 3 client.4350.objecter handle_osd_map decoding full epoch 10 2024-03-25T18:29:34.924 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:29:34.923+0000 7f7fd0ff9640 20 client.4350.objecter dump_active .. 0 homeless 2024-03-25T18:29:34.924 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:29:34.923+0000 7f7fd0ff9640 1 -- 172.21.15.1:0/11624269 <== mon.0 v1:172.21.15.1:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f7fc404f060 con 0x7f7fd40b4670 2024-03-25T18:29:34.924 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:29:34.923+0000 7f7fd0ff9640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:29:34.924 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:29:34.923+0000 7f7fd0ff9640 10 monclient: _finish_command 1 = system:0 2024-03-25T18:29:35.140 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:29:35.140+0000 7f7fd9a3e640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:29:35.141 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:29:35.140+0000 7f7fd9a3e640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:29:35.141 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:29:35.140+0000 7f7fd9a3e640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.1:6789/0 2024-03-25T18:29:35.141 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:29:35.140+0000 7f7fd9a3e640 1 -- 172.21.15.1:0/11624269 --> v1:172.21.15.1:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f7fd40688c0 con 0x7f7fd40b4670 2024-03-25T18:29:35.215 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:29:35.214+0000 7f7fd0ff9640 1 -- 172.21.15.1:0/11624269 <== mon.0 v1:172.21.15.1:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 4092213348) 0x7f7fc4050f90 con 0x7f7fd40b4670 2024-03-25T18:29:35.216 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:29:35.214+0000 7f7fd0ff9640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:29:35.216 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:29:35.214+0000 7f7fd0ff9640 10 monclient: _finish_command 2 = system:0 2024-03-25T18:29:35.216 INFO:teuthology.misc.health.smithi001.stdout: 2024-03-25T18:29:35.216 INFO:teuthology.misc.health.smithi001.stdout:{"epoch":10,"fsid":"c8ea72cf-9e99-4383-9b52-6c23d365fdfc","created":"2024-03-25T18:26:38.856638+0000","modified":"2024-03-25T18:26:44.224208+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T18:26:44.224208+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"b9f34922-3a84-49be-8426-807c1eceaefc","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"42bf306f-55e3-4634-9874-8b4b73ac6644","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"292cb23a-18a9-4207-a7cc-4d5727feaec0","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"45e0e9ab-898d-4673-9590-b4117bc5e84a","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"831a5a09-73ab-4a61-9872-088e76bdbc66","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"88cd903d-3b55-4840-9396-6e3cfe005282","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"89db7355-bc9d-46f4-8ab0-e8d9e744e026","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"cac47bee-0914-4f08-896f-348838882866","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T18:29:35.218 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:29:35.217+0000 7f7fd9a3e640 10 librados: watch_flush enter 2024-03-25T18:29:35.219 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:29:35.217+0000 7f7fd9a3e640 10 librados: watch_flush exit 2024-03-25T18:29:35.219 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:29:35.217+0000 7f7fd9a3e640 20 client.4350.objecter shutdown clearing up homeless session... 2024-03-25T18:29:35.219 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:29:35.217+0000 7f7fd9a3e640 10 client.4350.objecter successfully canceled tick 2024-03-25T18:29:35.219 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:29:35.217+0000 7f7fd9a3e640 1 -- 172.21.15.1:0/11624269 >> v1:172.21.15.115:6800/4237379176 conn(0x7f7fb406e160 legacy=0x7f7fb4070620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:29:35.219 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:29:35.217+0000 7f7fd9a3e640 10 monclient: shutdown 2024-03-25T18:29:35.219 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:29:35.217+0000 7f7fd9a3e640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:29:35.219 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:29:35.217+0000 7f7fd9a3e640 1 -- 172.21.15.1:0/11624269 >> v1:172.21.15.1:6789/0 conn(0x7f7fd40b4670 legacy=0x7f7fd40b1200 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:29:35.219 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:29:35.217+0000 7f7fd9a3e640 1 -- 172.21.15.1:0/11624269 shutdown_connections 2024-03-25T18:29:35.219 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:29:35.217+0000 7f7fd9a3e640 1 -- 172.21.15.1:0/11624269 >> 172.21.15.1:0/11624269 conn(0x7f7fd40617a0 msgr2=0x7f7fd40b70e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:29:35.219 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:29:35.218+0000 7f7fd9a3e640 1 -- 172.21.15.1:0/11624269 shutdown_connections 2024-03-25T18:29:35.219 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:29:35.218+0000 7f7fd9a3e640 1 -- 172.21.15.1:0/11624269 wait complete. 2024-03-25T18:29:35.219 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:29:35.218+0000 7f7fd9a3e640 1 librados: shutdown 2024-03-25T18:29:35.229 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T18:29:36.362 INFO:tasks.ceph.mon.a.smithi001.stderr:==00:00:03:14.921 34876== Warning: unimplemented fcntl command: 1036 2024-03-25T18:29:36.373 INFO:tasks.ceph.mon.a.smithi001.stderr:==00:00:03:14.931 34876== Warning: unimplemented fcntl command: 1036 2024-03-25T18:29:36.397 INFO:tasks.ceph.mon.a.smithi001.stderr:==00:00:03:14.956 34876== Warning: unimplemented fcntl command: 1036 2024-03-25T18:29:36.417 INFO:tasks.ceph.mon.b.smithi115.stderr:==00:00:03:14.901 34836== Warning: unimplemented fcntl command: 1036 2024-03-25T18:29:36.421 INFO:tasks.ceph.mon.c.smithi001.stderr:==00:00:03:14.979 34877== Warning: unimplemented fcntl command: 1036 2024-03-25T18:29:36.430 INFO:tasks.ceph.mon.b.smithi115.stderr:==00:00:03:14.913 34836== Warning: unimplemented fcntl command: 1036 2024-03-25T18:29:36.441 INFO:tasks.ceph.mon.c.smithi001.stderr:==00:00:03:15.000 34877== Warning: unimplemented fcntl command: 1036 2024-03-25T18:29:36.463 INFO:tasks.ceph.mon.b.smithi115.stderr:==00:00:03:14.947 34836== Warning: unimplemented fcntl command: 1036 2024-03-25T18:29:36.470 INFO:tasks.ceph.mon.c.smithi001.stderr:==00:00:03:15.029 34877== Warning: unimplemented fcntl command: 1036 2024-03-25T18:29:42.037 DEBUG:teuthology.orchestra.run.smithi001:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T18:29:42.152 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:29:42.151+0000 7f656f016640 10 monclient: get_monmap_and_config 2024-03-25T18:29:42.152 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:29:42.151+0000 7f656f016640 10 monclient: build_initial_monmap 2024-03-25T18:29:42.152 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:29:42.151+0000 7f656f016640 10 monclient: monmap: 2024-03-25T18:29:42.153 INFO:teuthology.misc.health.smithi001.stderr:epoch 0 2024-03-25T18:29:42.153 INFO:teuthology.misc.health.smithi001.stderr:fsid c8ea72cf-9e99-4383-9b52-6c23d365fdfc 2024-03-25T18:29:42.153 INFO:teuthology.misc.health.smithi001.stderr:last_changed 2024-03-25T18:29:42.152315+0000 2024-03-25T18:29:42.153 INFO:teuthology.misc.health.smithi001.stderr:created 2024-03-25T18:29:42.152315+0000 2024-03-25T18:29:42.153 INFO:teuthology.misc.health.smithi001.stderr:min_mon_release 0 (unknown) 2024-03-25T18:29:42.153 INFO:teuthology.misc.health.smithi001.stderr:election_strategy: 3 2024-03-25T18:29:42.153 INFO:teuthology.misc.health.smithi001.stderr:0: v1:172.21.15.1:6789/0 mon.noname-a 2024-03-25T18:29:42.153 INFO:teuthology.misc.health.smithi001.stderr:1: v1:172.21.15.115:6789/0 mon.noname-c 2024-03-25T18:29:42.153 INFO:teuthology.misc.health.smithi001.stderr:2: v1:172.21.15.1:6790/0 mon.noname-b 2024-03-25T18:29:42.153 INFO:teuthology.misc.health.smithi001.stderr: 2024-03-25T18:29:42.155 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:29:42.152+0000 7f656f016640 1 Processor -- start 2024-03-25T18:29:42.155 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:29:42.152+0000 7f656f016640 1 -- start start 2024-03-25T18:29:42.155 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:29:42.152+0000 7f656f016640 10 monclient: init 2024-03-25T18:29:42.155 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:29:42.153+0000 7f656f016640 10 monclient: _reopen_session rank -1 2024-03-25T18:29:42.155 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:29:42.153+0000 7f656f016640 10 monclient: _add_conns ranks=[1,0,2] 2024-03-25T18:29:42.155 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:29:42.153+0000 7f656f016640 10 monclient(hunting): picked mon.noname-c con 0x7f65680b4ff0 addr v1:172.21.15.115:6789/0 2024-03-25T18:29:42.156 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:29:42.153+0000 7f656f016640 10 monclient(hunting): picked mon.noname-a con 0x7f65680b4670 addr v1:172.21.15.1:6789/0 2024-03-25T18:29:42.156 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:29:42.153+0000 7f656f016640 10 monclient(hunting): picked mon.noname-b con 0x7f656813c480 addr v1:172.21.15.1:6790/0 2024-03-25T18:29:42.156 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:29:42.153+0000 7f656f016640 1 -- --> v1:172.21.15.1:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f65680ab0e0 con 0x7f65680b4670 2024-03-25T18:29:42.156 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:29:42.153+0000 7f656f016640 1 -- --> v1:172.21.15.115:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f65680ab3c0 con 0x7f65680b4ff0 2024-03-25T18:29:42.156 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:29:42.153+0000 7f656f016640 1 -- --> v1:172.21.15.1:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f65680a9f30 con 0x7f656813c480 2024-03-25T18:29:42.156 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:29:42.153+0000 7f656f016640 10 monclient(hunting): _renew_subs 2024-03-25T18:29:42.156 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:29:42.153+0000 7f656f016640 10 monclient(hunting): authenticate will time out at 7662.623528s 2024-03-25T18:29:42.157 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:29:42.155+0000 7f656cdb2640 1 --1- >> v1:172.21.15.115:6789/0 conn(0x7f65680b4ff0 0x7f656813bd70 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.115:6789/0 says I am v1:172.21.15.1:50554/0 (socket says 172.21.15.1:50554) 2024-03-25T18:29:42.157 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:29:42.155+0000 7f656cdb2640 1 -- 172.21.15.1:0/2578867163 learned_addr learned my addr 172.21.15.1:0/2578867163 (peer_addr_for_me v1:172.21.15.1:0/0) 2024-03-25T18:29:42.160 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:29:42.159+0000 7f65677fe640 1 -- 172.21.15.1:0/2578867163 <== mon.1 v1:172.21.15.115:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3994599959 0 0) 0x7f65680ab3c0 con 0x7f65680b4ff0 2024-03-25T18:29:42.160 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:29:42.159+0000 7f65677fe640 10 monclient(hunting): _negotiate 2024-03-25T18:29:42.160 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:29:42.159+0000 7f65677fe640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:29:42.160 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:29:42.159+0000 7f65677fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:29:42.160 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:29:42.159+0000 7f65677fe640 10 monclient(hunting): my global_id is 4348 2024-03-25T18:29:42.160 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:29:42.159+0000 7f65677fe640 1 -- 172.21.15.1:0/2578867163 --> v1:172.21.15.115:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f6558003520 con 0x7f65680b4ff0 2024-03-25T18:29:42.167 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:29:42.165+0000 7f65677fe640 1 -- 172.21.15.1:0/2578867163 <== mon.2 v1:172.21.15.1:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 732868675 0 0) 0x7f65680a9f30 con 0x7f656813c480 2024-03-25T18:29:42.167 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:29:42.165+0000 7f65677fe640 10 monclient(hunting): _negotiate 2024-03-25T18:29:42.167 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:29:42.165+0000 7f65677fe640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:29:42.167 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:29:42.165+0000 7f65677fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:29:42.167 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:29:42.165+0000 7f65677fe640 10 monclient(hunting): my global_id is 4355 2024-03-25T18:29:42.167 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:29:42.165+0000 7f65677fe640 1 -- 172.21.15.1:0/2578867163 --> v1:172.21.15.1:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f65680ab3c0 con 0x7f656813c480 2024-03-25T18:29:42.168 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:29:42.166+0000 7f65677fe640 1 -- 172.21.15.1:0/2578867163 <== mon.1 v1:172.21.15.115:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2063789998 0 0) 0x7f6558003520 con 0x7f65680b4ff0 2024-03-25T18:29:42.168 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:29:42.166+0000 7f65677fe640 1 -- 172.21.15.1:0/2578867163 >> v1:172.21.15.1:6790/0 conn(0x7f656813c480 legacy=0x7f656814dc00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:29:42.168 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:29:42.166+0000 7f65677fe640 1 -- 172.21.15.1:0/2578867163 >> v1:172.21.15.1:6789/0 conn(0x7f65680b4670 legacy=0x7f65681425c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-25T18:29:42.168 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:29:42.166+0000 7f65677fe640 10 monclient: _finish_hunting 0 2024-03-25T18:29:42.168 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:29:42.166+0000 7f65677fe640 1 monclient: found mon.noname-c 2024-03-25T18:29:42.168 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:29:42.166+0000 7f65677fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:29:42.168 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:29:42.166+0000 7f65677fe640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.115:6789/0 2024-03-25T18:29:42.168 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:29:42.166+0000 7f65677fe640 1 -- 172.21.15.1:0/2578867163 --> v1:172.21.15.115:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f65680a9b70 con 0x7f65680b4ff0 2024-03-25T18:29:42.168 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:29:42.166+0000 7f65677fe640 10 monclient: _finish_auth 0 2024-03-25T18:29:42.168 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:29:42.166+0000 7f65677fe640 10 monclient: _check_auth_tickets 2024-03-25T18:29:42.168 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:29:42.166+0000 7f65677fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:29:42.168 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:29:42.166+0000 7f65677fe640 1 -- 172.21.15.1:0/2578867163 <== mon.1 v1:172.21.15.115:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1586965025 0 0) 0x7f655c001690 con 0x7f65680b4ff0 2024-03-25T18:29:42.168 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:29:42.166+0000 7f656f016640 5 monclient: authenticate success, global_id 4348 2024-03-25T18:29:42.168 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:29:42.166+0000 7f656f016640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T18:29:42.169 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:29:42.166+0000 7f65677fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:29:42.169 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:29:42.166+0000 7f65677fe640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-03-25T18:29:42.169 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:29:42.166+0000 7f65677fe640 10 monclient: dump: 2024-03-25T18:29:42.169 INFO:teuthology.misc.health.smithi001.stderr:epoch 1 2024-03-25T18:29:42.169 INFO:teuthology.misc.health.smithi001.stderr:fsid c8ea72cf-9e99-4383-9b52-6c23d365fdfc 2024-03-25T18:29:42.169 INFO:teuthology.misc.health.smithi001.stderr:last_changed 2024-03-25T18:25:50.510519+0000 2024-03-25T18:29:42.169 INFO:teuthology.misc.health.smithi001.stderr:created 2024-03-25T18:25:50.510519+0000 2024-03-25T18:29:42.169 INFO:teuthology.misc.health.smithi001.stderr:min_mon_release 19 (squid) 2024-03-25T18:29:42.169 INFO:teuthology.misc.health.smithi001.stderr:election_strategy: 3 2024-03-25T18:29:42.169 INFO:teuthology.misc.health.smithi001.stderr:0: v1:172.21.15.1:6789/0 mon.a 2024-03-25T18:29:42.169 INFO:teuthology.misc.health.smithi001.stderr:1: v1:172.21.15.115:6789/0 mon.b 2024-03-25T18:29:42.169 INFO:teuthology.misc.health.smithi001.stderr:2: v1:172.21.15.1:6790/0 mon.c 2024-03-25T18:29:42.169 INFO:teuthology.misc.health.smithi001.stderr: 2024-03-25T18:29:42.169 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:29:42.166+0000 7f656f016640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T18:29:42.175 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:29:42.174+0000 7f65677fe640 1 -- 172.21.15.1:0/2578867163 <== mon.1 v1:172.21.15.115:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f655c0017f0 con 0x7f65680b4ff0 2024-03-25T18:29:42.175 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:29:42.174+0000 7f65677fe640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:29:42.175 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:29:42.174+0000 7f65677fe640 1 -- 172.21.15.1:0/2578867163 <== mon.1 v1:172.21.15.115:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1586965025 0 0) 0x7f655c001c00 con 0x7f65680b4ff0 2024-03-25T18:29:42.175 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:29:42.174+0000 7f656f016640 10 monclient: get_monmap_and_config success 2024-03-25T18:29:42.175 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:29:42.174+0000 7f656f016640 4 set_mon_vals no callback set 2024-03-25T18:29:42.175 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:29:42.174+0000 7f65677fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:29:42.175 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:29:42.174+0000 7f65677fe640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-25T18:29:42.175 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:29:42.174+0000 7f65677fe640 10 monclient: dump: 2024-03-25T18:29:42.176 INFO:teuthology.misc.health.smithi001.stderr:epoch 1 2024-03-25T18:29:42.176 INFO:teuthology.misc.health.smithi001.stderr:fsid c8ea72cf-9e99-4383-9b52-6c23d365fdfc 2024-03-25T18:29:42.176 INFO:teuthology.misc.health.smithi001.stderr:last_changed 2024-03-25T18:25:50.510519+0000 2024-03-25T18:29:42.176 INFO:teuthology.misc.health.smithi001.stderr:created 2024-03-25T18:25:50.510519+0000 2024-03-25T18:29:42.176 INFO:teuthology.misc.health.smithi001.stderr:min_mon_release 19 (squid) 2024-03-25T18:29:42.176 INFO:teuthology.misc.health.smithi001.stderr:election_strategy: 3 2024-03-25T18:29:42.176 INFO:teuthology.misc.health.smithi001.stderr:0: v1:172.21.15.1:6789/0 mon.a 2024-03-25T18:29:42.176 INFO:teuthology.misc.health.smithi001.stderr:1: v1:172.21.15.115:6789/0 mon.b 2024-03-25T18:29:42.176 INFO:teuthology.misc.health.smithi001.stderr:2: v1:172.21.15.1:6790/0 mon.c 2024-03-25T18:29:42.176 INFO:teuthology.misc.health.smithi001.stderr: 2024-03-25T18:29:42.179 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:29:42.175+0000 7f656f016640 10 monclient: shutdown 2024-03-25T18:29:42.179 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:29:42.175+0000 7f656f016640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:29:42.179 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:29:42.175+0000 7f656f016640 1 -- 172.21.15.1:0/2578867163 >> v1:172.21.15.115:6789/0 conn(0x7f65680b4ff0 legacy=0x7f656813bd70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:29:42.179 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:29:42.175+0000 7f656f016640 1 -- 172.21.15.1:0/2578867163 shutdown_connections 2024-03-25T18:29:42.179 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:29:42.175+0000 7f656f016640 1 -- 172.21.15.1:0/2578867163 >> 172.21.15.1:0/2578867163 conn(0x7f65680617a0 msgr2=0x7f6568061bb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:29:42.179 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:29:42.175+0000 7f656f016640 1 -- 172.21.15.1:0/2578867163 shutdown_connections 2024-03-25T18:29:42.179 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:29:42.175+0000 7f656f016640 1 -- 172.21.15.1:0/2578867163 wait complete. 2024-03-25T18:29:42.179 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:29:42.176+0000 7f656f016640 10 monclient: build_initial_monmap 2024-03-25T18:29:42.179 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:29:42.176+0000 7f656f016640 10 monclient: monmap: 2024-03-25T18:29:42.179 INFO:teuthology.misc.health.smithi001.stderr:epoch 0 2024-03-25T18:29:42.179 INFO:teuthology.misc.health.smithi001.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T18:29:42.180 INFO:teuthology.misc.health.smithi001.stderr:last_changed 0.000000 2024-03-25T18:29:42.180 INFO:teuthology.misc.health.smithi001.stderr:created 0.000000 2024-03-25T18:29:42.180 INFO:teuthology.misc.health.smithi001.stderr:min_mon_release 0 (unknown) 2024-03-25T18:29:42.180 INFO:teuthology.misc.health.smithi001.stderr:election_strategy: 1 2024-03-25T18:29:42.180 INFO:teuthology.misc.health.smithi001.stderr:0: v1:172.21.15.1:6789/0 mon.noname-a 2024-03-25T18:29:42.180 INFO:teuthology.misc.health.smithi001.stderr:1: v1:172.21.15.115:6789/0 mon.noname-b 2024-03-25T18:29:42.180 INFO:teuthology.misc.health.smithi001.stderr:2: v1:172.21.15.1:6790/0 mon.noname-c 2024-03-25T18:29:42.180 INFO:teuthology.misc.health.smithi001.stderr: 2024-03-25T18:29:42.180 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:29:42.176+0000 7f656f016640 1 librados: starting msgr at 2024-03-25T18:29:42.180 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:29:42.176+0000 7f656f016640 1 librados: starting objecter 2024-03-25T18:29:42.180 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:29:42.176+0000 7f656f016640 1 Processor -- start 2024-03-25T18:29:42.180 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:29:42.176+0000 7f656f016640 1 -- start start 2024-03-25T18:29:42.180 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:29:42.176+0000 7f656f016640 1 librados: setting wanted keys 2024-03-25T18:29:42.180 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:29:42.176+0000 7f656f016640 1 librados: calling monclient init 2024-03-25T18:29:42.181 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:29:42.176+0000 7f656f016640 10 monclient: init 2024-03-25T18:29:42.181 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:29:42.177+0000 7f656f016640 10 monclient: _reopen_session rank -1 2024-03-25T18:29:42.181 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:29:42.177+0000 7f656f016640 10 monclient: _add_conns ranks=[1,0,2] 2024-03-25T18:29:42.181 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:29:42.177+0000 7f656f016640 10 monclient(hunting): picked mon.noname-b con 0x7f65680b4670 addr v1:172.21.15.115:6789/0 2024-03-25T18:29:42.181 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:29:42.177+0000 7f656f016640 10 monclient(hunting): picked mon.noname-a con 0x7f65680b4ff0 addr v1:172.21.15.1:6789/0 2024-03-25T18:29:42.181 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:29:42.177+0000 7f656f016640 10 monclient(hunting): picked mon.noname-c con 0x7f656813c480 addr v1:172.21.15.1:6790/0 2024-03-25T18:29:42.181 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:29:42.177+0000 7f656f016640 1 -- --> v1:172.21.15.1:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f65681461e0 con 0x7f65680b4ff0 2024-03-25T18:29:42.181 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:29:42.177+0000 7f656f016640 1 -- --> v1:172.21.15.115:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6568144e70 con 0x7f65680b4670 2024-03-25T18:29:42.181 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:29:42.177+0000 7f656f016640 1 -- --> v1:172.21.15.1:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f656813da70 con 0x7f656813c480 2024-03-25T18:29:42.181 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:29:42.177+0000 7f656f016640 10 monclient(hunting): _renew_subs 2024-03-25T18:29:42.181 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:29:42.177+0000 7f656f016640 10 monclient(hunting): authenticate will time out at 7662.647422s 2024-03-25T18:29:42.188 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:29:42.187+0000 7f656d5b3640 1 --1- >> v1:172.21.15.1:6790/0 conn(0x7f656813c480 0x7f656813eb60 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.1:6790/0 says I am v1:172.21.15.1:53378/0 (socket says 172.21.15.1:53378) 2024-03-25T18:29:42.188 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:29:42.187+0000 7f656d5b3640 1 -- 172.21.15.1:0/39367169 learned_addr learned my addr 172.21.15.1:0/39367169 (peer_addr_for_me v1:172.21.15.1:0/0) 2024-03-25T18:29:42.189 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:29:42.188+0000 7f65657fa640 10 client.?.objecter ms_handle_connect 0x7f656813c480 2024-03-25T18:29:42.189 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:29:42.188+0000 7f65657fa640 10 client.?.objecter resend_mon_ops 2024-03-25T18:29:42.191 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:29:42.191+0000 7f65657fa640 10 client.?.objecter ms_handle_connect 0x7f65680b4670 2024-03-25T18:29:42.191 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:29:42.191+0000 7f65657fa640 10 client.?.objecter resend_mon_ops 2024-03-25T18:29:42.200 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:29:42.199+0000 7f65657fa640 1 -- 172.21.15.1:0/39367169 <== mon.1 v1:172.21.15.115:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 618862126 0 0) 0x7f6568144e70 con 0x7f65680b4670 2024-03-25T18:29:42.200 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:29:42.199+0000 7f65657fa640 10 monclient(hunting): _negotiate 2024-03-25T18:29:42.200 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:29:42.199+0000 7f65657fa640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:29:42.200 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:29:42.199+0000 7f65657fa640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:29:42.200 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:29:42.199+0000 7f65657fa640 10 monclient(hunting): my global_id is 4351 2024-03-25T18:29:42.200 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:29:42.199+0000 7f65657fa640 1 -- 172.21.15.1:0/39367169 --> v1:172.21.15.115:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f654c004eb0 con 0x7f65680b4670 2024-03-25T18:29:42.206 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:29:42.205+0000 7f65657fa640 10 client.?.objecter ms_handle_connect 0x7f65680b4ff0 2024-03-25T18:29:42.206 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:29:42.205+0000 7f65657fa640 10 client.?.objecter resend_mon_ops 2024-03-25T18:29:42.206 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:29:42.205+0000 7f65657fa640 1 -- 172.21.15.1:0/39367169 <== mon.2 v1:172.21.15.1:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1937122263 0 0) 0x7f656813da70 con 0x7f656813c480 2024-03-25T18:29:42.207 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:29:42.205+0000 7f65657fa640 10 monclient(hunting): _negotiate 2024-03-25T18:29:42.207 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:29:42.205+0000 7f65657fa640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:29:42.207 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:29:42.205+0000 7f65657fa640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:29:42.207 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:29:42.205+0000 7f65657fa640 10 monclient(hunting): my global_id is 4358 2024-03-25T18:29:42.207 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:29:42.206+0000 7f65657fa640 1 -- 172.21.15.1:0/39367169 --> v1:172.21.15.1:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f6568144e70 con 0x7f656813c480 2024-03-25T18:29:42.218 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:29:42.217+0000 7f65657fa640 1 -- 172.21.15.1:0/39367169 <== mon.1 v1:172.21.15.115:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3091037270 0 0) 0x7f654c004eb0 con 0x7f65680b4670 2024-03-25T18:29:42.218 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:29:42.217+0000 7f65657fa640 1 -- 172.21.15.1:0/39367169 --> v1:172.21.15.115:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f656813da70 con 0x7f65680b4670 2024-03-25T18:29:42.218 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:29:42.217+0000 7f65657fa640 1 -- 172.21.15.1:0/39367169 <== mon.1 v1:172.21.15.115:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1586965025 0 0) 0x7f655c001b00 con 0x7f65680b4670 2024-03-25T18:29:42.218 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:29:42.217+0000 7f65657fa640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:29:42.219 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:29:42.217+0000 7f65657fa640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-03-25T18:29:42.219 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:29:42.217+0000 7f65657fa640 10 monclient(hunting): dump: 2024-03-25T18:29:42.219 INFO:teuthology.misc.health.smithi001.stderr:epoch 1 2024-03-25T18:29:42.219 INFO:teuthology.misc.health.smithi001.stderr:fsid c8ea72cf-9e99-4383-9b52-6c23d365fdfc 2024-03-25T18:29:42.219 INFO:teuthology.misc.health.smithi001.stderr:last_changed 2024-03-25T18:25:50.510519+0000 2024-03-25T18:29:42.219 INFO:teuthology.misc.health.smithi001.stderr:created 2024-03-25T18:25:50.510519+0000 2024-03-25T18:29:42.219 INFO:teuthology.misc.health.smithi001.stderr:min_mon_release 19 (squid) 2024-03-25T18:29:42.219 INFO:teuthology.misc.health.smithi001.stderr:election_strategy: 3 2024-03-25T18:29:42.219 INFO:teuthology.misc.health.smithi001.stderr:0: v1:172.21.15.1:6789/0 mon.a 2024-03-25T18:29:42.219 INFO:teuthology.misc.health.smithi001.stderr:1: v1:172.21.15.115:6789/0 mon.b 2024-03-25T18:29:42.219 INFO:teuthology.misc.health.smithi001.stderr:2: v1:172.21.15.1:6790/0 mon.c 2024-03-25T18:29:42.219 INFO:teuthology.misc.health.smithi001.stderr: 2024-03-25T18:29:42.219 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:29:42.217+0000 7f65657fa640 10 monclient(hunting): _finish_auth 0 2024-03-25T18:29:42.222 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:29:42.220+0000 7f65657fa640 1 -- 172.21.15.1:0/39367169 <== mon.2 v1:172.21.15.1:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 4144169424 0 0) 0x7f6568144e70 con 0x7f656813c480 2024-03-25T18:29:42.222 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:29:42.220+0000 7f65657fa640 1 -- 172.21.15.1:0/39367169 --> v1:172.21.15.1:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f654c004eb0 con 0x7f656813c480 2024-03-25T18:29:42.222 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:29:42.220+0000 7f65657fa640 1 -- 172.21.15.1:0/39367169 <== mon.2 v1:172.21.15.1:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1586965025 0 0) 0x7f6560001ca0 con 0x7f656813c480 2024-03-25T18:29:42.222 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:29:42.220+0000 7f65657fa640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:29:42.222 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:29:42.221+0000 7f65657fa640 10 monclient(hunting): got monmap 1 from mon.c (according to old e1) 2024-03-25T18:29:42.222 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:29:42.221+0000 7f65657fa640 10 monclient(hunting): dump: 2024-03-25T18:29:42.223 INFO:teuthology.misc.health.smithi001.stderr:epoch 1 2024-03-25T18:29:42.223 INFO:teuthology.misc.health.smithi001.stderr:fsid c8ea72cf-9e99-4383-9b52-6c23d365fdfc 2024-03-25T18:29:42.223 INFO:teuthology.misc.health.smithi001.stderr:last_changed 2024-03-25T18:25:50.510519+0000 2024-03-25T18:29:42.223 INFO:teuthology.misc.health.smithi001.stderr:created 2024-03-25T18:25:50.510519+0000 2024-03-25T18:29:42.223 INFO:teuthology.misc.health.smithi001.stderr:min_mon_release 19 (squid) 2024-03-25T18:29:42.223 INFO:teuthology.misc.health.smithi001.stderr:election_strategy: 3 2024-03-25T18:29:42.223 INFO:teuthology.misc.health.smithi001.stderr:0: v1:172.21.15.1:6789/0 mon.a 2024-03-25T18:29:42.223 INFO:teuthology.misc.health.smithi001.stderr:1: v1:172.21.15.115:6789/0 mon.b 2024-03-25T18:29:42.223 INFO:teuthology.misc.health.smithi001.stderr:2: v1:172.21.15.1:6790/0 mon.c 2024-03-25T18:29:42.223 INFO:teuthology.misc.health.smithi001.stderr: 2024-03-25T18:29:42.230 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:29:42.230+0000 7f65657fa640 1 -- 172.21.15.1:0/39367169 <== mon.0 v1:172.21.15.1:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2000907097 0 0) 0x7f65681461e0 con 0x7f65680b4ff0 2024-03-25T18:29:42.231 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:29:42.230+0000 7f65657fa640 10 monclient(hunting): _negotiate 2024-03-25T18:29:42.231 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:29:42.230+0000 7f65657fa640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:29:42.231 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:29:42.230+0000 7f65657fa640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:29:42.231 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:29:42.230+0000 7f65657fa640 10 monclient(hunting): my global_id is 4353 2024-03-25T18:29:42.231 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:29:42.230+0000 7f65657fa640 1 -- 172.21.15.1:0/39367169 --> v1:172.21.15.1:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f6568144e70 con 0x7f65680b4ff0 2024-03-25T18:29:42.242 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:29:42.240+0000 7f65657fa640 1 -- 172.21.15.1:0/39367169 <== mon.2 v1:172.21.15.1:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 2745284232 0 0) 0x7f654c004eb0 con 0x7f656813c480 2024-03-25T18:29:42.242 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:29:42.240+0000 7f65657fa640 1 -- 172.21.15.1:0/39367169 >> v1:172.21.15.115:6789/0 conn(0x7f65680b4670 legacy=0x7f6568143660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:29:42.242 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:29:42.240+0000 7f65657fa640 1 -- 172.21.15.1:0/39367169 >> v1:172.21.15.1:6789/0 conn(0x7f65680b4ff0 legacy=0x7f656813e450 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:29:42.242 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:29:42.240+0000 7f65657fa640 10 monclient: _finish_hunting 0 2024-03-25T18:29:42.242 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:29:42.240+0000 7f65657fa640 1 monclient: found mon.c 2024-03-25T18:29:42.243 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:29:42.240+0000 7f65657fa640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:29:42.243 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:29:42.240+0000 7f65657fa640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.1:6790/0 2024-03-25T18:29:42.243 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:29:42.240+0000 7f65657fa640 1 -- 172.21.15.1:0/39367169 --> v1:172.21.15.1:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f656813dd50 con 0x7f656813c480 2024-03-25T18:29:42.243 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:29:42.240+0000 7f65657fa640 10 monclient: _finish_auth 0 2024-03-25T18:29:42.243 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:29:42.240+0000 7f65657fa640 10 monclient: _check_auth_tickets 2024-03-25T18:29:42.243 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:29:42.240+0000 7f65657fa640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:29:42.243 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:29:42.240+0000 7f656f016640 5 monclient: authenticate success, global_id 4358 2024-03-25T18:29:42.243 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:29:42.240+0000 7f656f016640 10 monclient: _renew_subs 2024-03-25T18:29:42.243 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:29:42.240+0000 7f656f016640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.1:6790/0 2024-03-25T18:29:42.243 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:29:42.240+0000 7f656f016640 1 -- 172.21.15.1:0/39367169 --> v1:172.21.15.1:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f656813e030 con 0x7f656813c480 2024-03-25T18:29:42.243 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:29:42.241+0000 7f656f016640 10 client.4358.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T18:29:42.243 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:29:42.241+0000 7f656f016640 10 monclient: _renew_subs 2024-03-25T18:29:42.243 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:29:42.241+0000 7f656f016640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.1:6790/0 2024-03-25T18:29:42.243 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:29:42.241+0000 7f656f016640 1 -- 172.21.15.1:0/39367169 --> v1:172.21.15.1:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f656813f270 con 0x7f656813c480 2024-03-25T18:29:42.243 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:29:42.241+0000 7f656f016640 1 librados: init done 2024-03-25T18:29:42.244 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:29:42.241+0000 7f656f016640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T18:29:42.244 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:29:42.241+0000 7f656f016640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:29:42.244 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:29:42.241+0000 7f656f016640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.1:6790/0 2024-03-25T18:29:42.244 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:29:42.241+0000 7f656f016640 1 -- 172.21.15.1:0/39367169 --> v1:172.21.15.1:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6568146530 con 0x7f656813c480 2024-03-25T18:29:42.565 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:29:42.564+0000 7f65657fa640 1 -- 172.21.15.1:0/39367169 <== mon.2 v1:172.21.15.1:6790/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f6560003990 con 0x7f656813c480 2024-03-25T18:29:42.565 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:29:42.564+0000 7f65657fa640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:29:42.565 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:29:42.564+0000 7f65657fa640 1 -- 172.21.15.1:0/39367169 <== mon.2 v1:172.21.15.1:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1586965025 0 0) 0x7f6560003de0 con 0x7f656813c480 2024-03-25T18:29:42.565 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:29:42.564+0000 7f65657fa640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:29:42.565 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:29:42.564+0000 7f65657fa640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-25T18:29:42.565 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:29:42.564+0000 7f65657fa640 10 monclient: dump: 2024-03-25T18:29:42.565 INFO:teuthology.misc.health.smithi001.stderr:epoch 1 2024-03-25T18:29:42.565 INFO:teuthology.misc.health.smithi001.stderr:fsid c8ea72cf-9e99-4383-9b52-6c23d365fdfc 2024-03-25T18:29:42.565 INFO:teuthology.misc.health.smithi001.stderr:last_changed 2024-03-25T18:25:50.510519+0000 2024-03-25T18:29:42.565 INFO:teuthology.misc.health.smithi001.stderr:created 2024-03-25T18:25:50.510519+0000 2024-03-25T18:29:42.565 INFO:teuthology.misc.health.smithi001.stderr:min_mon_release 19 (squid) 2024-03-25T18:29:42.566 INFO:teuthology.misc.health.smithi001.stderr:election_strategy: 3 2024-03-25T18:29:42.566 INFO:teuthology.misc.health.smithi001.stderr:0: v1:172.21.15.1:6789/0 mon.a 2024-03-25T18:29:42.566 INFO:teuthology.misc.health.smithi001.stderr:1: v1:172.21.15.115:6789/0 mon.b 2024-03-25T18:29:42.566 INFO:teuthology.misc.health.smithi001.stderr:2: v1:172.21.15.1:6790/0 mon.c 2024-03-25T18:29:42.566 INFO:teuthology.misc.health.smithi001.stderr: 2024-03-25T18:29:42.566 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:29:42.564+0000 7f6566ffd640 4 set_mon_vals no callback set 2024-03-25T18:29:42.567 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:29:42.565+0000 7f65657fa640 1 -- 172.21.15.1:0/39367169 <== mon.2 v1:172.21.15.1:6790/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 21787975 0 0) 0x7f6560004000 con 0x7f656813c480 2024-03-25T18:29:42.567 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:29:42.565+0000 7f656cdb2640 10 monclient: get_auth_request con 0x7f654c06dee0 auth_method 0 2024-03-25T18:29:42.567 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:29:42.565+0000 7f65657fa640 10 client.4358.objecter ms_handle_connect 0x7f654c06dee0 2024-03-25T18:29:42.572 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:29:42.571+0000 7f65657fa640 1 -- 172.21.15.1:0/39367169 <== mon.2 v1:172.21.15.1:6790/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 996254230 0 0) 0x7f6560054ad0 con 0x7f656813c480 2024-03-25T18:29:42.572 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:29:42.571+0000 7f65657fa640 10 client.4358.objecter ms_dispatch 0x7f6568000b90 osd_map(10..10 src has 1..10) v4 2024-03-25T18:29:42.572 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:29:42.571+0000 7f65657fa640 3 client.4358.objecter handle_osd_map got epochs [10,10] > 0 2024-03-25T18:29:42.572 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:29:42.571+0000 7f65657fa640 3 client.4358.objecter handle_osd_map decoding full epoch 10 2024-03-25T18:29:42.572 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:29:42.571+0000 7f65657fa640 20 client.4358.objecter dump_active .. 0 homeless 2024-03-25T18:29:42.572 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:29:42.571+0000 7f65657fa640 1 -- 172.21.15.1:0/39367169 <== mon.2 v1:172.21.15.1:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f65600505e0 con 0x7f656813c480 2024-03-25T18:29:42.572 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:29:42.571+0000 7f65657fa640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:29:42.572 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:29:42.571+0000 7f65657fa640 10 monclient: _finish_command 1 = system:0 2024-03-25T18:29:42.842 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:29:42.841+0000 7f656f016640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:29:42.842 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:29:42.841+0000 7f656f016640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:29:42.842 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:29:42.841+0000 7f656f016640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.1:6790/0 2024-03-25T18:29:42.842 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:29:42.841+0000 7f656f016640 1 -- 172.21.15.1:0/39367169 --> v1:172.21.15.1:6790/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f656806dbd0 con 0x7f656813c480 2024-03-25T18:29:42.946 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:29:42.945+0000 7f65657fa640 1 -- 172.21.15.1:0/39367169 <== mon.2 v1:172.21.15.1:6790/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 4092213348) 0x7f6560052510 con 0x7f656813c480 2024-03-25T18:29:42.946 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:29:42.945+0000 7f65657fa640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:29:42.946 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:29:42.945+0000 7f65657fa640 10 monclient: _finish_command 2 = system:0 2024-03-25T18:29:42.947 INFO:teuthology.misc.health.smithi001.stdout: 2024-03-25T18:29:42.947 INFO:teuthology.misc.health.smithi001.stdout:{"epoch":10,"fsid":"c8ea72cf-9e99-4383-9b52-6c23d365fdfc","created":"2024-03-25T18:26:38.856638+0000","modified":"2024-03-25T18:26:44.224208+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T18:26:44.224208+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"b9f34922-3a84-49be-8426-807c1eceaefc","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"42bf306f-55e3-4634-9874-8b4b73ac6644","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"292cb23a-18a9-4207-a7cc-4d5727feaec0","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"45e0e9ab-898d-4673-9590-b4117bc5e84a","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"831a5a09-73ab-4a61-9872-088e76bdbc66","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"88cd903d-3b55-4840-9396-6e3cfe005282","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"89db7355-bc9d-46f4-8ab0-e8d9e744e026","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"cac47bee-0914-4f08-896f-348838882866","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T18:29:42.948 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:29:42.947+0000 7f656f016640 10 librados: watch_flush enter 2024-03-25T18:29:42.949 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:29:42.947+0000 7f656f016640 10 librados: watch_flush exit 2024-03-25T18:29:42.949 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:29:42.947+0000 7f656f016640 20 client.4358.objecter shutdown clearing up homeless session... 2024-03-25T18:29:42.949 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:29:42.947+0000 7f656f016640 10 client.4358.objecter successfully canceled tick 2024-03-25T18:29:42.949 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:29:42.947+0000 7f656f016640 1 -- 172.21.15.1:0/39367169 >> v1:172.21.15.115:6800/4237379176 conn(0x7f654c06dee0 legacy=0x7f654c0703a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:29:42.949 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:29:42.947+0000 7f656f016640 10 monclient: shutdown 2024-03-25T18:29:42.949 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:29:42.947+0000 7f656f016640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:29:42.949 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:29:42.947+0000 7f656f016640 1 -- 172.21.15.1:0/39367169 >> v1:172.21.15.1:6790/0 conn(0x7f656813c480 legacy=0x7f656813eb60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:29:42.949 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:29:42.947+0000 7f656f016640 1 -- 172.21.15.1:0/39367169 shutdown_connections 2024-03-25T18:29:42.949 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:29:42.947+0000 7f656f016640 1 -- 172.21.15.1:0/39367169 >> 172.21.15.1:0/39367169 conn(0x7f65680617a0 msgr2=0x7f6568141bd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:29:42.949 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:29:42.947+0000 7f656f016640 1 -- 172.21.15.1:0/39367169 shutdown_connections 2024-03-25T18:29:42.949 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:29:42.948+0000 7f656f016640 1 -- 172.21.15.1:0/39367169 wait complete. 2024-03-25T18:29:42.949 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:29:42.948+0000 7f656f016640 1 librados: shutdown 2024-03-25T18:29:42.959 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T18:29:49.766 DEBUG:teuthology.orchestra.run.smithi001:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T18:29:49.875 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:29:49.874+0000 7fa1d7a57640 10 monclient: get_monmap_and_config 2024-03-25T18:29:49.875 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:29:49.874+0000 7fa1d7a57640 10 monclient: build_initial_monmap 2024-03-25T18:29:49.875 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:29:49.874+0000 7fa1d7a57640 10 monclient: monmap: 2024-03-25T18:29:49.875 INFO:teuthology.misc.health.smithi001.stderr:epoch 0 2024-03-25T18:29:49.875 INFO:teuthology.misc.health.smithi001.stderr:fsid c8ea72cf-9e99-4383-9b52-6c23d365fdfc 2024-03-25T18:29:49.875 INFO:teuthology.misc.health.smithi001.stderr:last_changed 2024-03-25T18:29:49.874708+0000 2024-03-25T18:29:49.875 INFO:teuthology.misc.health.smithi001.stderr:created 2024-03-25T18:29:49.874708+0000 2024-03-25T18:29:49.875 INFO:teuthology.misc.health.smithi001.stderr:min_mon_release 0 (unknown) 2024-03-25T18:29:49.875 INFO:teuthology.misc.health.smithi001.stderr:election_strategy: 3 2024-03-25T18:29:49.876 INFO:teuthology.misc.health.smithi001.stderr:0: v1:172.21.15.1:6789/0 mon.noname-a 2024-03-25T18:29:49.876 INFO:teuthology.misc.health.smithi001.stderr:1: v1:172.21.15.115:6789/0 mon.noname-c 2024-03-25T18:29:49.876 INFO:teuthology.misc.health.smithi001.stderr:2: v1:172.21.15.1:6790/0 mon.noname-b 2024-03-25T18:29:49.876 INFO:teuthology.misc.health.smithi001.stderr: 2024-03-25T18:29:49.876 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:29:49.874+0000 7fa1d7a57640 1 Processor -- start 2024-03-25T18:29:49.877 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:29:49.875+0000 7fa1d7a57640 1 -- start start 2024-03-25T18:29:49.877 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:29:49.875+0000 7fa1d7a57640 10 monclient: init 2024-03-25T18:29:49.877 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:29:49.875+0000 7fa1d7a57640 10 monclient: _reopen_session rank -1 2024-03-25T18:29:49.877 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:29:49.875+0000 7fa1d7a57640 10 monclient: _add_conns ranks=[1,2,0] 2024-03-25T18:29:49.877 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:29:49.875+0000 7fa1d7a57640 10 monclient(hunting): picked mon.noname-c con 0x7fa1d00b4ff0 addr v1:172.21.15.115:6789/0 2024-03-25T18:29:49.877 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:29:49.875+0000 7fa1d7a57640 10 monclient(hunting): picked mon.noname-b con 0x7fa1d00b4670 addr v1:172.21.15.1:6790/0 2024-03-25T18:29:49.877 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:29:49.875+0000 7fa1d7a57640 10 monclient(hunting): picked mon.noname-a con 0x7fa1d013c480 addr v1:172.21.15.1:6789/0 2024-03-25T18:29:49.877 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:29:49.875+0000 7fa1d7a57640 1 -- --> v1:172.21.15.1:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa1d00ab0e0 con 0x7fa1d013c480 2024-03-25T18:29:49.877 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:29:49.875+0000 7fa1d7a57640 1 -- --> v1:172.21.15.115:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa1d00ab3c0 con 0x7fa1d00b4ff0 2024-03-25T18:29:49.877 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:29:49.875+0000 7fa1d7a57640 1 -- --> v1:172.21.15.1:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa1d00a9f30 con 0x7fa1d00b4670 2024-03-25T18:29:49.878 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:29:49.875+0000 7fa1d7a57640 10 monclient(hunting): _renew_subs 2024-03-25T18:29:49.878 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:29:49.875+0000 7fa1d7a57640 10 monclient(hunting): authenticate will time out at 7670.345487s 2024-03-25T18:29:49.882 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:29:49.882+0000 7fa1d4ff2640 1 --1- >> v1:172.21.15.1:6790/0 conn(0x7fa1d00b4670 0x7fa1d01425c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.1:6790/0 says I am v1:172.21.15.1:44460/0 (socket says 172.21.15.1:44460) 2024-03-25T18:29:49.882 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:29:49.882+0000 7fa1d4ff2640 1 -- 172.21.15.1:0/1011342206 learned_addr learned my addr 172.21.15.1:0/1011342206 (peer_addr_for_me v1:172.21.15.1:0/0) 2024-03-25T18:29:49.891 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:29:49.891+0000 7fa1c7fff640 1 -- 172.21.15.1:0/1011342206 <== mon.2 v1:172.21.15.1:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2449535509 0 0) 0x7fa1d00a9f30 con 0x7fa1d00b4670 2024-03-25T18:29:49.891 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:29:49.891+0000 7fa1c7fff640 10 monclient(hunting): _negotiate 2024-03-25T18:29:49.891 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:29:49.891+0000 7fa1c7fff640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:29:49.891 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:29:49.891+0000 7fa1c7fff640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:29:49.892 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:29:49.891+0000 7fa1c7fff640 10 monclient(hunting): my global_id is 4361 2024-03-25T18:29:49.892 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:29:49.891+0000 7fa1c7fff640 1 -- 172.21.15.1:0/1011342206 --> v1:172.21.15.1:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fa1bc003520 con 0x7fa1d00b4670 2024-03-25T18:29:49.903 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:29:49.902+0000 7fa1c7fff640 1 -- 172.21.15.1:0/1011342206 <== mon.2 v1:172.21.15.1:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3102505754 0 0) 0x7fa1bc003520 con 0x7fa1d00b4670 2024-03-25T18:29:49.903 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:29:49.902+0000 7fa1c7fff640 1 -- 172.21.15.1:0/1011342206 >> v1:172.21.15.115:6789/0 conn(0x7fa1d00b4ff0 legacy=0x7fa1d013bd70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:29:49.903 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:29:49.902+0000 7fa1c7fff640 1 -- 172.21.15.1:0/1011342206 >> v1:172.21.15.1:6789/0 conn(0x7fa1d013c480 legacy=0x7fa1d014dc00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-25T18:29:49.903 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:29:49.902+0000 7fa1c7fff640 10 monclient: _finish_hunting 0 2024-03-25T18:29:49.903 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:29:49.902+0000 7fa1c7fff640 1 monclient: found mon.noname-b 2024-03-25T18:29:49.903 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:29:49.902+0000 7fa1c7fff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:29:49.903 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:29:49.902+0000 7fa1c7fff640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.1:6790/0 2024-03-25T18:29:49.904 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:29:49.902+0000 7fa1c7fff640 1 -- 172.21.15.1:0/1011342206 --> v1:172.21.15.1:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa1d00a9b70 con 0x7fa1d00b4670 2024-03-25T18:29:49.904 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:29:49.902+0000 7fa1c7fff640 10 monclient: _finish_auth 0 2024-03-25T18:29:49.904 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:29:49.902+0000 7fa1c7fff640 10 monclient: _check_auth_tickets 2024-03-25T18:29:49.904 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:29:49.902+0000 7fa1c7fff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:29:49.904 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:29:49.902+0000 7fa1c7fff640 1 -- 172.21.15.1:0/1011342206 <== mon.2 v1:172.21.15.1:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1586965025 0 0) 0x7fa1c0001690 con 0x7fa1d00b4670 2024-03-25T18:29:49.904 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:29:49.902+0000 7fa1c7fff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:29:49.904 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:29:49.902+0000 7fa1c7fff640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-03-25T18:29:49.904 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:29:49.902+0000 7fa1c7fff640 10 monclient: dump: 2024-03-25T18:29:49.904 INFO:teuthology.misc.health.smithi001.stderr:epoch 1 2024-03-25T18:29:49.904 INFO:teuthology.misc.health.smithi001.stderr:fsid c8ea72cf-9e99-4383-9b52-6c23d365fdfc 2024-03-25T18:29:49.904 INFO:teuthology.misc.health.smithi001.stderr:last_changed 2024-03-25T18:25:50.510519+0000 2024-03-25T18:29:49.904 INFO:teuthology.misc.health.smithi001.stderr:created 2024-03-25T18:25:50.510519+0000 2024-03-25T18:29:49.904 INFO:teuthology.misc.health.smithi001.stderr:min_mon_release 19 (squid) 2024-03-25T18:29:49.904 INFO:teuthology.misc.health.smithi001.stderr:election_strategy: 3 2024-03-25T18:29:49.905 INFO:teuthology.misc.health.smithi001.stderr:0: v1:172.21.15.1:6789/0 mon.a 2024-03-25T18:29:49.905 INFO:teuthology.misc.health.smithi001.stderr:1: v1:172.21.15.115:6789/0 mon.b 2024-03-25T18:29:49.905 INFO:teuthology.misc.health.smithi001.stderr:2: v1:172.21.15.1:6790/0 mon.c 2024-03-25T18:29:49.905 INFO:teuthology.misc.health.smithi001.stderr: 2024-03-25T18:29:49.905 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:29:49.902+0000 7fa1d7a57640 5 monclient: authenticate success, global_id 4361 2024-03-25T18:29:49.905 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:29:49.902+0000 7fa1d7a57640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T18:29:49.916 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:29:49.916+0000 7fa1c7fff640 1 -- 172.21.15.1:0/1011342206 <== mon.2 v1:172.21.15.1:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fa1c0001860 con 0x7fa1d00b4670 2024-03-25T18:29:49.917 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:29:49.916+0000 7fa1c7fff640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:29:49.917 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:29:49.916+0000 7fa1c7fff640 1 -- 172.21.15.1:0/1011342206 <== mon.2 v1:172.21.15.1:6790/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1586965025 0 0) 0x7fa1c0001cb0 con 0x7fa1d00b4670 2024-03-25T18:29:49.917 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:29:49.916+0000 7fa1c7fff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:29:49.917 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:29:49.916+0000 7fa1c7fff640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-25T18:29:49.917 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:29:49.916+0000 7fa1c7fff640 10 monclient: dump: 2024-03-25T18:29:49.917 INFO:teuthology.misc.health.smithi001.stderr:epoch 1 2024-03-25T18:29:49.917 INFO:teuthology.misc.health.smithi001.stderr:fsid c8ea72cf-9e99-4383-9b52-6c23d365fdfc 2024-03-25T18:29:49.917 INFO:teuthology.misc.health.smithi001.stderr:last_changed 2024-03-25T18:25:50.510519+0000 2024-03-25T18:29:49.917 INFO:teuthology.misc.health.smithi001.stderr:created 2024-03-25T18:25:50.510519+0000 2024-03-25T18:29:49.917 INFO:teuthology.misc.health.smithi001.stderr:min_mon_release 19 (squid) 2024-03-25T18:29:49.917 INFO:teuthology.misc.health.smithi001.stderr:election_strategy: 3 2024-03-25T18:29:49.917 INFO:teuthology.misc.health.smithi001.stderr:0: v1:172.21.15.1:6789/0 mon.a 2024-03-25T18:29:49.917 INFO:teuthology.misc.health.smithi001.stderr:1: v1:172.21.15.115:6789/0 mon.b 2024-03-25T18:29:49.918 INFO:teuthology.misc.health.smithi001.stderr:2: v1:172.21.15.1:6790/0 mon.c 2024-03-25T18:29:49.918 INFO:teuthology.misc.health.smithi001.stderr: 2024-03-25T18:29:49.918 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:29:49.916+0000 7fa1d7a57640 10 monclient: get_monmap_and_config success 2024-03-25T18:29:49.918 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:29:49.916+0000 7fa1d7a57640 4 set_mon_vals no callback set 2024-03-25T18:29:49.919 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:29:49.917+0000 7fa1d7a57640 10 monclient: shutdown 2024-03-25T18:29:49.919 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:29:49.917+0000 7fa1d7a57640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:29:49.919 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:29:49.917+0000 7fa1d7a57640 1 -- 172.21.15.1:0/1011342206 >> v1:172.21.15.1:6790/0 conn(0x7fa1d00b4670 legacy=0x7fa1d01425c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:29:49.920 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:29:49.917+0000 7fa1d7a57640 1 -- 172.21.15.1:0/1011342206 shutdown_connections 2024-03-25T18:29:49.920 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:29:49.917+0000 7fa1d7a57640 1 -- 172.21.15.1:0/1011342206 >> 172.21.15.1:0/1011342206 conn(0x7fa1d00617a0 msgr2=0x7fa1d0061bb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:29:49.920 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:29:49.917+0000 7fa1d7a57640 1 -- 172.21.15.1:0/1011342206 shutdown_connections 2024-03-25T18:29:49.920 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:29:49.917+0000 7fa1d7a57640 1 -- 172.21.15.1:0/1011342206 wait complete. 2024-03-25T18:29:49.920 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:29:49.918+0000 7fa1d7a57640 10 monclient: build_initial_monmap 2024-03-25T18:29:49.920 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:29:49.918+0000 7fa1d7a57640 10 monclient: monmap: 2024-03-25T18:29:49.920 INFO:teuthology.misc.health.smithi001.stderr:epoch 0 2024-03-25T18:29:49.920 INFO:teuthology.misc.health.smithi001.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T18:29:49.920 INFO:teuthology.misc.health.smithi001.stderr:last_changed 0.000000 2024-03-25T18:29:49.920 INFO:teuthology.misc.health.smithi001.stderr:created 0.000000 2024-03-25T18:29:49.920 INFO:teuthology.misc.health.smithi001.stderr:min_mon_release 0 (unknown) 2024-03-25T18:29:49.920 INFO:teuthology.misc.health.smithi001.stderr:election_strategy: 1 2024-03-25T18:29:49.920 INFO:teuthology.misc.health.smithi001.stderr:0: v1:172.21.15.1:6789/0 mon.noname-a 2024-03-25T18:29:49.920 INFO:teuthology.misc.health.smithi001.stderr:1: v1:172.21.15.115:6789/0 mon.noname-b 2024-03-25T18:29:49.920 INFO:teuthology.misc.health.smithi001.stderr:2: v1:172.21.15.1:6790/0 mon.noname-c 2024-03-25T18:29:49.921 INFO:teuthology.misc.health.smithi001.stderr: 2024-03-25T18:29:49.921 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:29:49.918+0000 7fa1d7a57640 1 librados: starting msgr at 2024-03-25T18:29:49.921 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:29:49.918+0000 7fa1d7a57640 1 librados: starting objecter 2024-03-25T18:29:49.921 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:29:49.918+0000 7fa1d7a57640 1 Processor -- start 2024-03-25T18:29:49.921 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:29:49.918+0000 7fa1d7a57640 1 -- start start 2024-03-25T18:29:49.921 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:29:49.918+0000 7fa1d7a57640 1 librados: setting wanted keys 2024-03-25T18:29:49.921 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:29:49.918+0000 7fa1d7a57640 1 librados: calling monclient init 2024-03-25T18:29:49.921 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:29:49.918+0000 7fa1d7a57640 10 monclient: init 2024-03-25T18:29:49.922 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:29:49.919+0000 7fa1d7a57640 10 monclient: _reopen_session rank -1 2024-03-25T18:29:49.922 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:29:49.919+0000 7fa1d7a57640 10 monclient: _add_conns ranks=[2,1,0] 2024-03-25T18:29:49.922 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:29:49.919+0000 7fa1d7a57640 10 monclient(hunting): picked mon.noname-c con 0x7fa1d00b4670 addr v1:172.21.15.1:6790/0 2024-03-25T18:29:49.923 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:29:49.919+0000 7fa1d7a57640 10 monclient(hunting): picked mon.noname-b con 0x7fa1d00b4ff0 addr v1:172.21.15.115:6789/0 2024-03-25T18:29:49.923 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:29:49.919+0000 7fa1d7a57640 10 monclient(hunting): picked mon.noname-a con 0x7fa1d013c480 addr v1:172.21.15.1:6789/0 2024-03-25T18:29:49.923 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:29:49.919+0000 7fa1d7a57640 1 -- --> v1:172.21.15.1:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa1d01441b0 con 0x7fa1d013c480 2024-03-25T18:29:49.923 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:29:49.919+0000 7fa1d7a57640 1 -- --> v1:172.21.15.115:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa1d0144490 con 0x7fa1d00b4ff0 2024-03-25T18:29:49.923 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:29:49.919+0000 7fa1d7a57640 1 -- --> v1:172.21.15.1:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa1d0144770 con 0x7fa1d00b4670 2024-03-25T18:29:49.923 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:29:49.919+0000 7fa1d7a57640 10 monclient(hunting): _renew_subs 2024-03-25T18:29:49.923 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:29:49.919+0000 7fa1d7a57640 10 monclient(hunting): authenticate will time out at 7670.389607s 2024-03-25T18:29:49.931 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:29:49.931+0000 7fa1d4ff2640 1 --1- >> v1:172.21.15.115:6789/0 conn(0x7fa1d00b4ff0 0x7fa1d012fcb0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.115:6789/0 says I am v1:172.21.15.1:52326/0 (socket says 172.21.15.1:52326) 2024-03-25T18:29:49.931 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:29:49.931+0000 7fa1d4ff2640 1 -- 172.21.15.1:0/1806627694 learned_addr learned my addr 172.21.15.1:0/1806627694 (peer_addr_for_me v1:172.21.15.1:0/0) 2024-03-25T18:29:49.934 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:29:49.934+0000 7fa1c5ffb640 10 client.?.objecter ms_handle_connect 0x7fa1d00b4ff0 2024-03-25T18:29:49.935 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:29:49.934+0000 7fa1c5ffb640 10 client.?.objecter resend_mon_ops 2024-03-25T18:29:49.938 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:29:49.938+0000 7fa1c5ffb640 10 client.?.objecter ms_handle_connect 0x7fa1d013c480 2024-03-25T18:29:49.938 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:29:49.938+0000 7fa1c5ffb640 10 client.?.objecter resend_mon_ops 2024-03-25T18:29:49.943 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:29:49.942+0000 7fa1c5ffb640 10 client.?.objecter ms_handle_connect 0x7fa1d00b4670 2024-03-25T18:29:49.943 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:29:49.942+0000 7fa1c5ffb640 10 client.?.objecter resend_mon_ops 2024-03-25T18:29:49.945 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:29:49.944+0000 7fa1c5ffb640 1 -- 172.21.15.1:0/1806627694 <== mon.1 v1:172.21.15.115:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3939049162 0 0) 0x7fa1d0144490 con 0x7fa1d00b4ff0 2024-03-25T18:29:49.945 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:29:49.944+0000 7fa1c5ffb640 10 monclient(hunting): _negotiate 2024-03-25T18:29:49.945 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:29:49.944+0000 7fa1c5ffb640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:29:49.945 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:29:49.944+0000 7fa1c5ffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:29:49.945 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:29:49.944+0000 7fa1c5ffb640 10 monclient(hunting): my global_id is 4357 2024-03-25T18:29:49.945 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:29:49.944+0000 7fa1c5ffb640 1 -- 172.21.15.1:0/1806627694 --> v1:172.21.15.115:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fa1b4004eb0 con 0x7fa1d00b4ff0 2024-03-25T18:29:49.956 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:29:49.956+0000 7fa1c5ffb640 1 -- 172.21.15.1:0/1806627694 <== mon.0 v1:172.21.15.1:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 187286317 0 0) 0x7fa1d01441b0 con 0x7fa1d013c480 2024-03-25T18:29:49.957 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:29:49.956+0000 7fa1c5ffb640 10 monclient(hunting): _negotiate 2024-03-25T18:29:49.957 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:29:49.956+0000 7fa1c5ffb640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:29:49.957 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:29:49.956+0000 7fa1c5ffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:29:49.957 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:29:49.956+0000 7fa1c5ffb640 10 monclient(hunting): my global_id is 4356 2024-03-25T18:29:49.957 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:29:49.956+0000 7fa1c5ffb640 1 -- 172.21.15.1:0/1806627694 --> v1:172.21.15.1:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fa1d0144490 con 0x7fa1d013c480 2024-03-25T18:29:49.968 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:29:49.967+0000 7fa1c5ffb640 1 -- 172.21.15.1:0/1806627694 <== mon.2 v1:172.21.15.1:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2278336504 0 0) 0x7fa1d0144770 con 0x7fa1d00b4670 2024-03-25T18:29:49.968 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:29:49.967+0000 7fa1c5ffb640 10 monclient(hunting): _negotiate 2024-03-25T18:29:49.968 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:29:49.967+0000 7fa1c5ffb640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:29:49.968 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:29:49.967+0000 7fa1c5ffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:29:49.968 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:29:49.967+0000 7fa1c5ffb640 10 monclient(hunting): my global_id is 4364 2024-03-25T18:29:49.968 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:29:49.967+0000 7fa1c5ffb640 1 -- 172.21.15.1:0/1806627694 --> v1:172.21.15.1:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fa1d01441b0 con 0x7fa1d00b4670 2024-03-25T18:29:49.977 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:29:49.977+0000 7fa1c5ffb640 1 -- 172.21.15.1:0/1806627694 <== mon.1 v1:172.21.15.115:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1486533948 0 0) 0x7fa1b4004eb0 con 0x7fa1d00b4ff0 2024-03-25T18:29:49.978 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:29:49.977+0000 7fa1c5ffb640 1 -- 172.21.15.1:0/1806627694 --> v1:172.21.15.115:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fa1d0144770 con 0x7fa1d00b4ff0 2024-03-25T18:29:49.978 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:29:49.977+0000 7fa1c5ffb640 1 -- 172.21.15.1:0/1806627694 <== mon.1 v1:172.21.15.115:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1586965025 0 0) 0x7fa1c0001b70 con 0x7fa1d00b4ff0 2024-03-25T18:29:49.978 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:29:49.977+0000 7fa1c5ffb640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:29:49.978 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:29:49.977+0000 7fa1c5ffb640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-03-25T18:29:49.978 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:29:49.977+0000 7fa1c5ffb640 10 monclient(hunting): dump: 2024-03-25T18:29:49.978 INFO:teuthology.misc.health.smithi001.stderr:epoch 1 2024-03-25T18:29:49.978 INFO:teuthology.misc.health.smithi001.stderr:fsid c8ea72cf-9e99-4383-9b52-6c23d365fdfc 2024-03-25T18:29:49.978 INFO:teuthology.misc.health.smithi001.stderr:last_changed 2024-03-25T18:25:50.510519+0000 2024-03-25T18:29:49.978 INFO:teuthology.misc.health.smithi001.stderr:created 2024-03-25T18:25:50.510519+0000 2024-03-25T18:29:49.978 INFO:teuthology.misc.health.smithi001.stderr:min_mon_release 19 (squid) 2024-03-25T18:29:49.978 INFO:teuthology.misc.health.smithi001.stderr:election_strategy: 3 2024-03-25T18:29:49.978 INFO:teuthology.misc.health.smithi001.stderr:0: v1:172.21.15.1:6789/0 mon.a 2024-03-25T18:29:49.978 INFO:teuthology.misc.health.smithi001.stderr:1: v1:172.21.15.115:6789/0 mon.b 2024-03-25T18:29:49.979 INFO:teuthology.misc.health.smithi001.stderr:2: v1:172.21.15.1:6790/0 mon.c 2024-03-25T18:29:49.979 INFO:teuthology.misc.health.smithi001.stderr: 2024-03-25T18:29:49.979 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:29:49.977+0000 7fa1c5ffb640 10 monclient(hunting): _finish_auth 0 2024-03-25T18:29:49.989 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:29:49.989+0000 7fa1c5ffb640 1 -- 172.21.15.1:0/1806627694 <== mon.0 v1:172.21.15.1:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1210991804 0 0) 0x7fa1d0144490 con 0x7fa1d013c480 2024-03-25T18:29:49.989 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:29:49.989+0000 7fa1c5ffb640 1 -- 172.21.15.1:0/1806627694 --> v1:172.21.15.1:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fa1b4004eb0 con 0x7fa1d013c480 2024-03-25T18:29:49.989 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:29:49.989+0000 7fa1c5ffb640 1 -- 172.21.15.1:0/1806627694 <== mon.0 v1:172.21.15.1:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1586965025 0 0) 0x7fa1c8001a70 con 0x7fa1d013c480 2024-03-25T18:29:49.989 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:29:49.989+0000 7fa1c5ffb640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:29:49.990 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:29:49.989+0000 7fa1c5ffb640 10 monclient(hunting): got monmap 1 from mon.a (according to old e1) 2024-03-25T18:29:49.990 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:29:49.989+0000 7fa1c5ffb640 10 monclient(hunting): dump: 2024-03-25T18:29:49.990 INFO:teuthology.misc.health.smithi001.stderr:epoch 1 2024-03-25T18:29:49.990 INFO:teuthology.misc.health.smithi001.stderr:fsid c8ea72cf-9e99-4383-9b52-6c23d365fdfc 2024-03-25T18:29:49.990 INFO:teuthology.misc.health.smithi001.stderr:last_changed 2024-03-25T18:25:50.510519+0000 2024-03-25T18:29:49.990 INFO:teuthology.misc.health.smithi001.stderr:created 2024-03-25T18:25:50.510519+0000 2024-03-25T18:29:49.990 INFO:teuthology.misc.health.smithi001.stderr:min_mon_release 19 (squid) 2024-03-25T18:29:49.990 INFO:teuthology.misc.health.smithi001.stderr:election_strategy: 3 2024-03-25T18:29:49.990 INFO:teuthology.misc.health.smithi001.stderr:0: v1:172.21.15.1:6789/0 mon.a 2024-03-25T18:29:49.990 INFO:teuthology.misc.health.smithi001.stderr:1: v1:172.21.15.115:6789/0 mon.b 2024-03-25T18:29:49.990 INFO:teuthology.misc.health.smithi001.stderr:2: v1:172.21.15.1:6790/0 mon.c 2024-03-25T18:29:49.990 INFO:teuthology.misc.health.smithi001.stderr: 2024-03-25T18:29:49.995 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:29:49.994+0000 7fa1c5ffb640 1 -- 172.21.15.1:0/1806627694 <== mon.2 v1:172.21.15.1:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 4273359360 0 0) 0x7fa1d01441b0 con 0x7fa1d00b4670 2024-03-25T18:29:49.995 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:29:49.994+0000 7fa1c5ffb640 1 -- 172.21.15.1:0/1806627694 --> v1:172.21.15.1:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fa1d0144490 con 0x7fa1d00b4670 2024-03-25T18:29:49.995 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:29:49.994+0000 7fa1c5ffb640 1 -- 172.21.15.1:0/1806627694 <== mon.2 v1:172.21.15.1:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1586965025 0 0) 0x7fa1cc001830 con 0x7fa1d00b4670 2024-03-25T18:29:49.995 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:29:49.994+0000 7fa1c5ffb640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:29:49.996 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:29:49.994+0000 7fa1c5ffb640 10 monclient(hunting): got monmap 1 from mon.c (according to old e1) 2024-03-25T18:29:49.996 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:29:49.994+0000 7fa1c5ffb640 10 monclient(hunting): dump: 2024-03-25T18:29:49.996 INFO:teuthology.misc.health.smithi001.stderr:epoch 1 2024-03-25T18:29:49.996 INFO:teuthology.misc.health.smithi001.stderr:fsid c8ea72cf-9e99-4383-9b52-6c23d365fdfc 2024-03-25T18:29:49.996 INFO:teuthology.misc.health.smithi001.stderr:last_changed 2024-03-25T18:25:50.510519+0000 2024-03-25T18:29:49.996 INFO:teuthology.misc.health.smithi001.stderr:created 2024-03-25T18:25:50.510519+0000 2024-03-25T18:29:49.996 INFO:teuthology.misc.health.smithi001.stderr:min_mon_release 19 (squid) 2024-03-25T18:29:49.996 INFO:teuthology.misc.health.smithi001.stderr:election_strategy: 3 2024-03-25T18:29:49.996 INFO:teuthology.misc.health.smithi001.stderr:0: v1:172.21.15.1:6789/0 mon.a 2024-03-25T18:29:49.996 INFO:teuthology.misc.health.smithi001.stderr:1: v1:172.21.15.115:6789/0 mon.b 2024-03-25T18:29:49.996 INFO:teuthology.misc.health.smithi001.stderr:2: v1:172.21.15.1:6790/0 mon.c 2024-03-25T18:29:49.996 INFO:teuthology.misc.health.smithi001.stderr: 2024-03-25T18:29:50.003 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:29:50.001+0000 7fa1c5ffb640 1 -- 172.21.15.1:0/1806627694 <== mon.1 v1:172.21.15.115:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 2182176858 0 0) 0x7fa1d0144770 con 0x7fa1d00b4ff0 2024-03-25T18:29:50.003 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:29:50.002+0000 7fa1c5ffb640 1 -- 172.21.15.1:0/1806627694 >> v1:172.21.15.1:6790/0 conn(0x7fa1d00b4670 legacy=0x7fa1d0144e70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:29:50.003 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:29:50.002+0000 7fa1c5ffb640 1 -- 172.21.15.1:0/1806627694 >> v1:172.21.15.1:6789/0 conn(0x7fa1d013c480 legacy=0x7fa1d01303c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:29:50.003 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:29:50.002+0000 7fa1c5ffb640 10 monclient: _finish_hunting 0 2024-03-25T18:29:50.003 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:29:50.002+0000 7fa1c5ffb640 1 monclient: found mon.b 2024-03-25T18:29:50.003 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:29:50.002+0000 7fa1c5ffb640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:29:50.003 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:29:50.002+0000 7fa1c5ffb640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.115:6789/0 2024-03-25T18:29:50.004 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:29:50.002+0000 7fa1c5ffb640 1 -- 172.21.15.1:0/1806627694 --> v1:172.21.15.115:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa1d0144a50 con 0x7fa1d00b4ff0 2024-03-25T18:29:50.004 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:29:50.002+0000 7fa1c5ffb640 10 monclient: _finish_auth 0 2024-03-25T18:29:50.004 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:29:50.002+0000 7fa1c5ffb640 10 monclient: _check_auth_tickets 2024-03-25T18:29:50.004 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:29:50.002+0000 7fa1c5ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:29:50.004 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:29:50.002+0000 7fa1d7a57640 5 monclient: authenticate success, global_id 4357 2024-03-25T18:29:50.004 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:29:50.002+0000 7fa1d7a57640 10 monclient: _renew_subs 2024-03-25T18:29:50.004 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:29:50.002+0000 7fa1d7a57640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.115:6789/0 2024-03-25T18:29:50.004 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:29:50.002+0000 7fa1d7a57640 1 -- 172.21.15.1:0/1806627694 --> v1:172.21.15.115:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa1d0143660 con 0x7fa1d00b4ff0 2024-03-25T18:29:50.004 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:29:50.002+0000 7fa1d7a57640 10 client.4357.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T18:29:50.004 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:29:50.002+0000 7fa1d7a57640 10 monclient: _renew_subs 2024-03-25T18:29:50.004 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:29:50.002+0000 7fa1d7a57640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.115:6789/0 2024-03-25T18:29:50.004 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:29:50.002+0000 7fa1d7a57640 1 -- 172.21.15.1:0/1806627694 --> v1:172.21.15.115:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7fa1d0130ad0 con 0x7fa1d00b4ff0 2024-03-25T18:29:50.004 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:29:50.002+0000 7fa1d7a57640 1 librados: init done 2024-03-25T18:29:50.005 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:29:50.003+0000 7fa1d7a57640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T18:29:50.005 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:29:50.003+0000 7fa1d7a57640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:29:50.005 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:29:50.003+0000 7fa1d7a57640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.115:6789/0 2024-03-25T18:29:50.005 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:29:50.003+0000 7fa1d7a57640 1 -- 172.21.15.1:0/1806627694 --> v1:172.21.15.115:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa1d006dbd0 con 0x7fa1d00b4ff0 2024-03-25T18:29:50.032 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:29:50.031+0000 7fa1c5ffb640 1 -- 172.21.15.1:0/1806627694 <== mon.1 v1:172.21.15.115:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fa1c0004bf0 con 0x7fa1d00b4ff0 2024-03-25T18:29:50.032 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:29:50.031+0000 7fa1c5ffb640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:29:50.032 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:29:50.031+0000 7fa1c5ffb640 1 -- 172.21.15.1:0/1806627694 <== mon.1 v1:172.21.15.115:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1586965025 0 0) 0x7fa1c0004f00 con 0x7fa1d00b4ff0 2024-03-25T18:29:50.032 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:29:50.031+0000 7fa1c5ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:29:50.032 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:29:50.031+0000 7fa1c5ffb640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-25T18:29:50.032 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:29:50.031+0000 7fa1c5ffb640 10 monclient: dump: 2024-03-25T18:29:50.032 INFO:teuthology.misc.health.smithi001.stderr:epoch 1 2024-03-25T18:29:50.032 INFO:teuthology.misc.health.smithi001.stderr:fsid c8ea72cf-9e99-4383-9b52-6c23d365fdfc 2024-03-25T18:29:50.032 INFO:teuthology.misc.health.smithi001.stderr:last_changed 2024-03-25T18:25:50.510519+0000 2024-03-25T18:29:50.032 INFO:teuthology.misc.health.smithi001.stderr:created 2024-03-25T18:25:50.510519+0000 2024-03-25T18:29:50.033 INFO:teuthology.misc.health.smithi001.stderr:min_mon_release 19 (squid) 2024-03-25T18:29:50.033 INFO:teuthology.misc.health.smithi001.stderr:election_strategy: 3 2024-03-25T18:29:50.033 INFO:teuthology.misc.health.smithi001.stderr:0: v1:172.21.15.1:6789/0 mon.a 2024-03-25T18:29:50.033 INFO:teuthology.misc.health.smithi001.stderr:1: v1:172.21.15.115:6789/0 mon.b 2024-03-25T18:29:50.033 INFO:teuthology.misc.health.smithi001.stderr:2: v1:172.21.15.1:6790/0 mon.c 2024-03-25T18:29:50.033 INFO:teuthology.misc.health.smithi001.stderr: 2024-03-25T18:29:50.033 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:29:50.031+0000 7fa1c77fe640 4 set_mon_vals no callback set 2024-03-25T18:29:50.033 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:29:50.032+0000 7fa1c5ffb640 1 -- 172.21.15.1:0/1806627694 <== mon.1 v1:172.21.15.115:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 21787975 0 0) 0x7fa1c001aed0 con 0x7fa1d00b4ff0 2024-03-25T18:29:50.034 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:29:50.032+0000 7fa1d57f3640 10 monclient: get_auth_request con 0x7fa1b406e0b0 auth_method 0 2024-03-25T18:29:50.034 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:29:50.033+0000 7fa1c5ffb640 10 client.4357.objecter ms_handle_connect 0x7fa1b406e0b0 2024-03-25T18:29:50.035 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:29:50.034+0000 7fa1c5ffb640 1 -- 172.21.15.1:0/1806627694 <== mon.1 v1:172.21.15.115:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 996254230 0 0) 0x7fa1c00548a0 con 0x7fa1d00b4ff0 2024-03-25T18:29:50.035 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:29:50.034+0000 7fa1c5ffb640 10 client.4357.objecter ms_dispatch 0x7fa1d0000b90 osd_map(10..10 src has 1..10) v4 2024-03-25T18:29:50.035 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:29:50.034+0000 7fa1c5ffb640 3 client.4357.objecter handle_osd_map got epochs [10,10] > 0 2024-03-25T18:29:50.035 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:29:50.034+0000 7fa1c5ffb640 3 client.4357.objecter handle_osd_map decoding full epoch 10 2024-03-25T18:29:50.035 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:29:50.034+0000 7fa1c5ffb640 20 client.4357.objecter dump_active .. 0 homeless 2024-03-25T18:29:50.305 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:29:50.304+0000 7fa1c5ffb640 1 -- 172.21.15.1:0/1806627694 <== mon.1 v1:172.21.15.115:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7fa1c00503b0 con 0x7fa1d00b4ff0 2024-03-25T18:29:50.305 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:29:50.305+0000 7fa1c5ffb640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:29:50.306 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:29:50.305+0000 7fa1c5ffb640 10 monclient: _finish_command 1 = system:0 2024-03-25T18:29:50.563 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:29:50.562+0000 7fa1d7a57640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:29:50.564 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:29:50.562+0000 7fa1d7a57640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:29:50.564 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:29:50.562+0000 7fa1d7a57640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.115:6789/0 2024-03-25T18:29:50.564 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:29:50.562+0000 7fa1d7a57640 1 -- 172.21.15.1:0/1806627694 --> v1:172.21.15.115:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fa1d01437f0 con 0x7fa1d00b4ff0 2024-03-25T18:29:50.604 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:29:50.603+0000 7fa1c5ffb640 1 -- 172.21.15.1:0/1806627694 <== mon.1 v1:172.21.15.115:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 4092213348) 0x7fa1c00522e0 con 0x7fa1d00b4ff0 2024-03-25T18:29:50.604 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:29:50.603+0000 7fa1c5ffb640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:29:50.604 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:29:50.603+0000 7fa1c5ffb640 10 monclient: _finish_command 2 = system:0 2024-03-25T18:29:50.605 INFO:teuthology.misc.health.smithi001.stdout: 2024-03-25T18:29:50.605 INFO:teuthology.misc.health.smithi001.stdout:{"epoch":10,"fsid":"c8ea72cf-9e99-4383-9b52-6c23d365fdfc","created":"2024-03-25T18:26:38.856638+0000","modified":"2024-03-25T18:26:44.224208+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T18:26:44.224208+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"b9f34922-3a84-49be-8426-807c1eceaefc","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"42bf306f-55e3-4634-9874-8b4b73ac6644","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"292cb23a-18a9-4207-a7cc-4d5727feaec0","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"45e0e9ab-898d-4673-9590-b4117bc5e84a","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"831a5a09-73ab-4a61-9872-088e76bdbc66","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"88cd903d-3b55-4840-9396-6e3cfe005282","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"89db7355-bc9d-46f4-8ab0-e8d9e744e026","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"cac47bee-0914-4f08-896f-348838882866","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T18:29:50.607 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:29:50.605+0000 7fa1d7a57640 10 librados: watch_flush enter 2024-03-25T18:29:50.607 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:29:50.605+0000 7fa1d7a57640 10 librados: watch_flush exit 2024-03-25T18:29:50.607 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:29:50.605+0000 7fa1d7a57640 20 client.4357.objecter shutdown clearing up homeless session... 2024-03-25T18:29:50.607 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:29:50.605+0000 7fa1d7a57640 10 client.4357.objecter successfully canceled tick 2024-03-25T18:29:50.607 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:29:50.605+0000 7fa1d7a57640 1 -- 172.21.15.1:0/1806627694 >> v1:172.21.15.115:6800/4237379176 conn(0x7fa1b406e0b0 legacy=0x7fa1b4070570 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:29:50.607 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:29:50.605+0000 7fa1d7a57640 10 monclient: shutdown 2024-03-25T18:29:50.607 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:29:50.605+0000 7fa1d7a57640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:29:50.607 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:29:50.605+0000 7fa1d7a57640 1 -- 172.21.15.1:0/1806627694 >> v1:172.21.15.115:6789/0 conn(0x7fa1d00b4ff0 legacy=0x7fa1d012fcb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:29:50.607 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:29:50.606+0000 7fa1d7a57640 1 -- 172.21.15.1:0/1806627694 shutdown_connections 2024-03-25T18:29:50.608 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:29:50.606+0000 7fa1d7a57640 1 -- 172.21.15.1:0/1806627694 >> 172.21.15.1:0/1806627694 conn(0x7fa1d00617a0 msgr2=0x7fa1d0140e50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:29:50.608 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:29:50.606+0000 7fa1d7a57640 1 -- 172.21.15.1:0/1806627694 shutdown_connections 2024-03-25T18:29:50.608 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:29:50.606+0000 7fa1d7a57640 1 -- 172.21.15.1:0/1806627694 wait complete. 2024-03-25T18:29:50.608 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:29:50.606+0000 7fa1d7a57640 1 librados: shutdown 2024-03-25T18:29:50.619 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T18:29:56.425 INFO:tasks.ceph.mon.a.smithi001.stderr:==00:00:03:34.983 34876== Warning: unimplemented fcntl command: 1036 2024-03-25T18:29:56.431 INFO:tasks.ceph.mon.a.smithi001.stderr:==00:00:03:34.990 34876== Warning: unimplemented fcntl command: 1036 2024-03-25T18:29:56.460 INFO:tasks.ceph.mon.a.smithi001.stderr:==00:00:03:35.019 34876== Warning: unimplemented fcntl command: 1036 2024-03-25T18:29:56.485 INFO:tasks.ceph.mon.c.smithi001.stderr:==00:00:03:35.043 34877== Warning: unimplemented fcntl command: 1036 2024-03-25T18:29:56.485 INFO:tasks.ceph.mon.b.smithi115.stderr:==00:00:03:34.969 34836== Warning: unimplemented fcntl command: 1036 2024-03-25T18:29:56.489 INFO:tasks.ceph.mon.b.smithi115.stderr:==00:00:03:34.973 34836== Warning: unimplemented fcntl command: 1036 2024-03-25T18:29:56.495 INFO:tasks.ceph.mon.c.smithi001.stderr:==00:00:03:35.054 34877== Warning: unimplemented fcntl command: 1036 2024-03-25T18:29:56.529 INFO:tasks.ceph.mon.c.smithi001.stderr:==00:00:03:35.088 34877== Warning: unimplemented fcntl command: 1036 2024-03-25T18:29:56.530 INFO:tasks.ceph.mon.b.smithi115.stderr:==00:00:03:35.013 34836== Warning: unimplemented fcntl command: 1036 2024-03-25T18:29:57.425 DEBUG:teuthology.orchestra.run.smithi001:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T18:29:57.523 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:29:57.522+0000 7fd4118fb640 10 monclient: get_monmap_and_config 2024-03-25T18:29:57.523 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:29:57.522+0000 7fd4118fb640 10 monclient: build_initial_monmap 2024-03-25T18:29:57.523 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:29:57.522+0000 7fd4118fb640 10 monclient: monmap: 2024-03-25T18:29:57.523 INFO:teuthology.misc.health.smithi001.stderr:epoch 0 2024-03-25T18:29:57.523 INFO:teuthology.misc.health.smithi001.stderr:fsid c8ea72cf-9e99-4383-9b52-6c23d365fdfc 2024-03-25T18:29:57.523 INFO:teuthology.misc.health.smithi001.stderr:last_changed 2024-03-25T18:29:57.522915+0000 2024-03-25T18:29:57.523 INFO:teuthology.misc.health.smithi001.stderr:created 2024-03-25T18:29:57.522915+0000 2024-03-25T18:29:57.523 INFO:teuthology.misc.health.smithi001.stderr:min_mon_release 0 (unknown) 2024-03-25T18:29:57.523 INFO:teuthology.misc.health.smithi001.stderr:election_strategy: 3 2024-03-25T18:29:57.524 INFO:teuthology.misc.health.smithi001.stderr:0: v1:172.21.15.1:6789/0 mon.noname-a 2024-03-25T18:29:57.524 INFO:teuthology.misc.health.smithi001.stderr:1: v1:172.21.15.115:6789/0 mon.noname-c 2024-03-25T18:29:57.524 INFO:teuthology.misc.health.smithi001.stderr:2: v1:172.21.15.1:6790/0 mon.noname-b 2024-03-25T18:29:57.524 INFO:teuthology.misc.health.smithi001.stderr: 2024-03-25T18:29:57.525 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:29:57.523+0000 7fd4118fb640 1 Processor -- start 2024-03-25T18:29:57.525 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:29:57.523+0000 7fd4118fb640 1 -- start start 2024-03-25T18:29:57.525 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:29:57.523+0000 7fd4118fb640 10 monclient: init 2024-03-25T18:29:57.525 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:29:57.524+0000 7fd4118fb640 10 monclient: _reopen_session rank -1 2024-03-25T18:29:57.526 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:29:57.524+0000 7fd4118fb640 10 monclient: _add_conns ranks=[2,1,0] 2024-03-25T18:29:57.526 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:29:57.524+0000 7fd4118fb640 10 monclient(hunting): picked mon.noname-b con 0x7fd40c0b4ff0 addr v1:172.21.15.1:6790/0 2024-03-25T18:29:57.526 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:29:57.524+0000 7fd4118fb640 10 monclient(hunting): picked mon.noname-c con 0x7fd40c0b4670 addr v1:172.21.15.115:6789/0 2024-03-25T18:29:57.526 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:29:57.524+0000 7fd4118fb640 10 monclient(hunting): picked mon.noname-a con 0x7fd40c13c480 addr v1:172.21.15.1:6789/0 2024-03-25T18:29:57.526 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:29:57.524+0000 7fd4118fb640 1 -- --> v1:172.21.15.1:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd40c0ab0e0 con 0x7fd40c13c480 2024-03-25T18:29:57.526 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:29:57.524+0000 7fd4118fb640 1 -- --> v1:172.21.15.115:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd40c0ab3c0 con 0x7fd40c0b4670 2024-03-25T18:29:57.526 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:29:57.524+0000 7fd4118fb640 1 -- --> v1:172.21.15.1:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd40c0a9f30 con 0x7fd40c0b4ff0 2024-03-25T18:29:57.526 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:29:57.524+0000 7fd4118fb640 10 monclient(hunting): _renew_subs 2024-03-25T18:29:57.526 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:29:57.524+0000 7fd4118fb640 10 monclient(hunting): authenticate will time out at 7677.993827s 2024-03-25T18:29:57.526 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:29:57.525+0000 7fd403fff640 1 --1- >> v1:172.21.15.115:6789/0 conn(0x7fd40c0b4670 0x7fd40c1425c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.115:6789/0 says I am v1:172.21.15.1:56980/0 (socket says 172.21.15.1:56980) 2024-03-25T18:29:57.526 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:29:57.525+0000 7fd403fff640 1 -- 172.21.15.1:0/2733303679 learned_addr learned my addr 172.21.15.1:0/2733303679 (peer_addr_for_me v1:172.21.15.1:0/0) 2024-03-25T18:29:57.613 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:29:57.612+0000 7fd40a7fc640 1 -- 172.21.15.1:0/2733303679 <== mon.0 v1:172.21.15.1:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1689284994 0 0) 0x7fd40c0ab0e0 con 0x7fd40c13c480 2024-03-25T18:29:57.613 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:29:57.612+0000 7fd40a7fc640 10 monclient(hunting): _negotiate 2024-03-25T18:29:57.613 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:29:57.612+0000 7fd40a7fc640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:29:57.613 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:29:57.612+0000 7fd40a7fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:29:57.613 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:29:57.612+0000 7fd40a7fc640 10 monclient(hunting): my global_id is 4359 2024-03-25T18:29:57.613 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:29:57.612+0000 7fd40a7fc640 1 -- 172.21.15.1:0/2733303679 --> v1:172.21.15.1:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fd3f8003520 con 0x7fd40c13c480 2024-03-25T18:29:57.641 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:29:57.640+0000 7fd40a7fc640 1 -- 172.21.15.1:0/2733303679 <== mon.0 v1:172.21.15.1:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1240403148 0 0) 0x7fd3f8003520 con 0x7fd40c13c480 2024-03-25T18:29:57.641 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:29:57.640+0000 7fd40a7fc640 1 -- 172.21.15.1:0/2733303679 >> v1:172.21.15.1:6790/0 conn(0x7fd40c0b4ff0 legacy=0x7fd40c13bd70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:29:57.642 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:29:57.640+0000 7fd40a7fc640 1 -- 172.21.15.1:0/2733303679 >> v1:172.21.15.115:6789/0 conn(0x7fd40c0b4670 legacy=0x7fd40c1425c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:29:57.642 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:29:57.640+0000 7fd40a7fc640 10 monclient: _finish_hunting 0 2024-03-25T18:29:57.642 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:29:57.640+0000 7fd40a7fc640 1 monclient: found mon.noname-a 2024-03-25T18:29:57.642 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:29:57.640+0000 7fd40a7fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:29:57.646 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:29:57.640+0000 7fd40a7fc640 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.1:6789/0 2024-03-25T18:29:57.646 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:29:57.640+0000 7fd40a7fc640 1 -- 172.21.15.1:0/2733303679 --> v1:172.21.15.1:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd40c0a9b70 con 0x7fd40c13c480 2024-03-25T18:29:57.646 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:29:57.640+0000 7fd40a7fc640 10 monclient: _finish_auth 0 2024-03-25T18:29:57.646 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:29:57.640+0000 7fd40a7fc640 10 monclient: _check_auth_tickets 2024-03-25T18:29:57.646 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:29:57.640+0000 7fd40a7fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:29:57.646 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:29:57.640+0000 7fd40a7fc640 1 -- 172.21.15.1:0/2733303679 <== mon.0 v1:172.21.15.1:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1586965025 0 0) 0x7fd404001a10 con 0x7fd40c13c480 2024-03-25T18:29:57.647 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:29:57.640+0000 7fd40a7fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:29:57.647 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:29:57.640+0000 7fd40a7fc640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-03-25T18:29:57.647 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:29:57.640+0000 7fd40a7fc640 10 monclient: dump: 2024-03-25T18:29:57.647 INFO:teuthology.misc.health.smithi001.stderr:epoch 1 2024-03-25T18:29:57.647 INFO:teuthology.misc.health.smithi001.stderr:fsid c8ea72cf-9e99-4383-9b52-6c23d365fdfc 2024-03-25T18:29:57.647 INFO:teuthology.misc.health.smithi001.stderr:last_changed 2024-03-25T18:25:50.510519+0000 2024-03-25T18:29:57.648 INFO:teuthology.misc.health.smithi001.stderr:created 2024-03-25T18:25:50.510519+0000 2024-03-25T18:29:57.648 INFO:teuthology.misc.health.smithi001.stderr:min_mon_release 19 (squid) 2024-03-25T18:29:57.648 INFO:teuthology.misc.health.smithi001.stderr:election_strategy: 3 2024-03-25T18:29:57.648 INFO:teuthology.misc.health.smithi001.stderr:0: v1:172.21.15.1:6789/0 mon.a 2024-03-25T18:29:57.648 INFO:teuthology.misc.health.smithi001.stderr:1: v1:172.21.15.115:6789/0 mon.b 2024-03-25T18:29:57.648 INFO:teuthology.misc.health.smithi001.stderr:2: v1:172.21.15.1:6790/0 mon.c 2024-03-25T18:29:57.649 INFO:teuthology.misc.health.smithi001.stderr: 2024-03-25T18:29:57.649 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:29:57.640+0000 7fd4118fb640 5 monclient: authenticate success, global_id 4359 2024-03-25T18:29:57.649 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:29:57.640+0000 7fd4118fb640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T18:29:57.652 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:29:57.651+0000 7fd40a7fc640 1 -- 172.21.15.1:0/2733303679 <== mon.0 v1:172.21.15.1:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fd404001b70 con 0x7fd40c13c480 2024-03-25T18:29:57.652 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:29:57.651+0000 7fd40a7fc640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:29:57.652 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:29:57.651+0000 7fd40a7fc640 1 -- 172.21.15.1:0/2733303679 <== mon.0 v1:172.21.15.1:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1586965025 0 0) 0x7fd404001fc0 con 0x7fd40c13c480 2024-03-25T18:29:57.653 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:29:57.651+0000 7fd4118fb640 10 monclient: get_monmap_and_config success 2024-03-25T18:29:57.653 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:29:57.651+0000 7fd4118fb640 4 set_mon_vals no callback set 2024-03-25T18:29:57.653 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:29:57.651+0000 7fd40a7fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:29:57.653 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:29:57.651+0000 7fd40a7fc640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-25T18:29:57.653 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:29:57.651+0000 7fd40a7fc640 10 monclient: dump: 2024-03-25T18:29:57.653 INFO:teuthology.misc.health.smithi001.stderr:epoch 1 2024-03-25T18:29:57.653 INFO:teuthology.misc.health.smithi001.stderr:fsid c8ea72cf-9e99-4383-9b52-6c23d365fdfc 2024-03-25T18:29:57.653 INFO:teuthology.misc.health.smithi001.stderr:last_changed 2024-03-25T18:25:50.510519+0000 2024-03-25T18:29:57.654 INFO:teuthology.misc.health.smithi001.stderr:created 2024-03-25T18:25:50.510519+0000 2024-03-25T18:29:57.654 INFO:teuthology.misc.health.smithi001.stderr:min_mon_release 19 (squid) 2024-03-25T18:29:57.654 INFO:teuthology.misc.health.smithi001.stderr:election_strategy: 3 2024-03-25T18:29:57.654 INFO:teuthology.misc.health.smithi001.stderr:0: v1:172.21.15.1:6789/0 mon.a 2024-03-25T18:29:57.654 INFO:teuthology.misc.health.smithi001.stderr:1: v1:172.21.15.115:6789/0 mon.b 2024-03-25T18:29:57.654 INFO:teuthology.misc.health.smithi001.stderr:2: v1:172.21.15.1:6790/0 mon.c 2024-03-25T18:29:57.654 INFO:teuthology.misc.health.smithi001.stderr: 2024-03-25T18:29:57.657 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:29:57.652+0000 7fd4118fb640 10 monclient: shutdown 2024-03-25T18:29:57.657 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:29:57.652+0000 7fd4118fb640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:29:57.657 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:29:57.652+0000 7fd4118fb640 1 -- 172.21.15.1:0/2733303679 >> v1:172.21.15.1:6789/0 conn(0x7fd40c13c480 legacy=0x7fd40c14dc00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:29:57.657 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:29:57.652+0000 7fd4118fb640 1 -- 172.21.15.1:0/2733303679 shutdown_connections 2024-03-25T18:29:57.657 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:29:57.652+0000 7fd4118fb640 1 -- 172.21.15.1:0/2733303679 >> 172.21.15.1:0/2733303679 conn(0x7fd40c0617a0 msgr2=0x7fd40c061bb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:29:57.657 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:29:57.652+0000 7fd4118fb640 1 -- 172.21.15.1:0/2733303679 shutdown_connections 2024-03-25T18:29:57.657 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:29:57.652+0000 7fd4118fb640 1 -- 172.21.15.1:0/2733303679 wait complete. 2024-03-25T18:29:57.657 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:29:57.653+0000 7fd4118fb640 10 monclient: build_initial_monmap 2024-03-25T18:29:57.658 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:29:57.653+0000 7fd4118fb640 10 monclient: monmap: 2024-03-25T18:29:57.658 INFO:teuthology.misc.health.smithi001.stderr:epoch 0 2024-03-25T18:29:57.658 INFO:teuthology.misc.health.smithi001.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T18:29:57.658 INFO:teuthology.misc.health.smithi001.stderr:last_changed 0.000000 2024-03-25T18:29:57.658 INFO:teuthology.misc.health.smithi001.stderr:created 0.000000 2024-03-25T18:29:57.658 INFO:teuthology.misc.health.smithi001.stderr:min_mon_release 0 (unknown) 2024-03-25T18:29:57.658 INFO:teuthology.misc.health.smithi001.stderr:election_strategy: 1 2024-03-25T18:29:57.658 INFO:teuthology.misc.health.smithi001.stderr:0: v1:172.21.15.1:6789/0 mon.noname-a 2024-03-25T18:29:57.658 INFO:teuthology.misc.health.smithi001.stderr:1: v1:172.21.15.115:6789/0 mon.noname-b 2024-03-25T18:29:57.658 INFO:teuthology.misc.health.smithi001.stderr:2: v1:172.21.15.1:6790/0 mon.noname-c 2024-03-25T18:29:57.658 INFO:teuthology.misc.health.smithi001.stderr: 2024-03-25T18:29:57.658 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:29:57.653+0000 7fd4118fb640 1 librados: starting msgr at 2024-03-25T18:29:57.658 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:29:57.653+0000 7fd4118fb640 1 librados: starting objecter 2024-03-25T18:29:57.658 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:29:57.653+0000 7fd4118fb640 1 Processor -- start 2024-03-25T18:29:57.658 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:29:57.653+0000 7fd4118fb640 1 -- start start 2024-03-25T18:29:57.659 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:29:57.653+0000 7fd4118fb640 1 librados: setting wanted keys 2024-03-25T18:29:57.659 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:29:57.653+0000 7fd4118fb640 1 librados: calling monclient init 2024-03-25T18:29:57.659 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:29:57.653+0000 7fd4118fb640 10 monclient: init 2024-03-25T18:29:57.659 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:29:57.654+0000 7fd4118fb640 10 monclient: _reopen_session rank -1 2024-03-25T18:29:57.659 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:29:57.654+0000 7fd4118fb640 10 monclient: _add_conns ranks=[1,0,2] 2024-03-25T18:29:57.659 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:29:57.654+0000 7fd4118fb640 10 monclient(hunting): picked mon.noname-b con 0x7fd40c0b4670 addr v1:172.21.15.115:6789/0 2024-03-25T18:29:57.659 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:29:57.654+0000 7fd4118fb640 10 monclient(hunting): picked mon.noname-a con 0x7fd40c0b4ff0 addr v1:172.21.15.1:6789/0 2024-03-25T18:29:57.659 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:29:57.654+0000 7fd4118fb640 10 monclient(hunting): picked mon.noname-c con 0x7fd40c13c480 addr v1:172.21.15.1:6790/0 2024-03-25T18:29:57.659 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:29:57.654+0000 7fd4118fb640 1 -- --> v1:172.21.15.1:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd40c1dbf50 con 0x7fd40c0b4ff0 2024-03-25T18:29:57.659 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:29:57.654+0000 7fd4118fb640 1 -- --> v1:172.21.15.115:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd40c144730 con 0x7fd40c0b4670 2024-03-25T18:29:57.659 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:29:57.654+0000 7fd4118fb640 1 -- --> v1:172.21.15.1:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd40c144a10 con 0x7fd40c13c480 2024-03-25T18:29:57.659 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:29:57.654+0000 7fd4118fb640 10 monclient(hunting): _renew_subs 2024-03-25T18:29:57.659 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:29:57.654+0000 7fd4118fb640 10 monclient(hunting): authenticate will time out at 7678.124187s 2024-03-25T18:29:57.677 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:29:57.676+0000 7fd40affd640 1 --1- >> v1:172.21.15.115:6789/0 conn(0x7fd40c0b4670 0x7fd40c144e30 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.115:6789/0 says I am v1:172.21.15.1:57044/0 (socket says 172.21.15.1:57044) 2024-03-25T18:29:57.677 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:29:57.676+0000 7fd40affd640 1 -- 172.21.15.1:0/1585760823 learned_addr learned my addr 172.21.15.1:0/1585760823 (peer_addr_for_me v1:172.21.15.1:0/0) 2024-03-25T18:29:57.680 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:29:57.679+0000 7fd408ff9640 10 client.?.objecter ms_handle_connect 0x7fd40c0b4ff0 2024-03-25T18:29:57.680 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:29:57.679+0000 7fd408ff9640 10 client.?.objecter resend_mon_ops 2024-03-25T18:29:57.680 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:29:57.680+0000 7fd408ff9640 10 client.?.objecter ms_handle_connect 0x7fd40c0b4670 2024-03-25T18:29:57.680 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:29:57.680+0000 7fd408ff9640 10 client.?.objecter resend_mon_ops 2024-03-25T18:29:57.691 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:29:57.690+0000 7fd408ff9640 10 client.?.objecter ms_handle_connect 0x7fd40c13c480 2024-03-25T18:29:57.691 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:29:57.690+0000 7fd408ff9640 10 client.?.objecter resend_mon_ops 2024-03-25T18:29:57.692 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:29:57.691+0000 7fd408ff9640 1 -- 172.21.15.1:0/1585760823 <== mon.1 v1:172.21.15.115:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1793338515 0 0) 0x7fd40c144730 con 0x7fd40c0b4670 2024-03-25T18:29:57.692 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:29:57.691+0000 7fd408ff9640 10 monclient(hunting): _negotiate 2024-03-25T18:29:57.692 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:29:57.691+0000 7fd408ff9640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:29:57.692 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:29:57.691+0000 7fd408ff9640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:29:57.692 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:29:57.691+0000 7fd408ff9640 10 monclient(hunting): my global_id is 4363 2024-03-25T18:29:57.692 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:29:57.691+0000 7fd408ff9640 1 -- 172.21.15.1:0/1585760823 --> v1:172.21.15.115:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fd3ec004eb0 con 0x7fd40c0b4670 2024-03-25T18:29:57.701 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:29:57.700+0000 7fd408ff9640 1 -- 172.21.15.1:0/1585760823 <== mon.2 v1:172.21.15.1:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3808665641 0 0) 0x7fd40c144a10 con 0x7fd40c13c480 2024-03-25T18:29:57.701 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:29:57.700+0000 7fd408ff9640 10 monclient(hunting): _negotiate 2024-03-25T18:29:57.701 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:29:57.700+0000 7fd408ff9640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:29:57.701 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:29:57.700+0000 7fd408ff9640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:29:57.701 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:29:57.700+0000 7fd408ff9640 10 monclient(hunting): my global_id is 4370 2024-03-25T18:29:57.701 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:29:57.700+0000 7fd408ff9640 1 -- 172.21.15.1:0/1585760823 --> v1:172.21.15.1:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fd40c144730 con 0x7fd40c13c480 2024-03-25T18:29:57.702 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:29:57.701+0000 7fd408ff9640 1 -- 172.21.15.1:0/1585760823 <== mon.0 v1:172.21.15.1:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 669388553 0 0) 0x7fd40c1dbf50 con 0x7fd40c0b4ff0 2024-03-25T18:29:57.702 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:29:57.701+0000 7fd408ff9640 10 monclient(hunting): _negotiate 2024-03-25T18:29:57.702 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:29:57.701+0000 7fd408ff9640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:29:57.702 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:29:57.701+0000 7fd408ff9640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:29:57.702 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:29:57.701+0000 7fd408ff9640 10 monclient(hunting): my global_id is 4362 2024-03-25T18:29:57.702 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:29:57.701+0000 7fd408ff9640 1 -- 172.21.15.1:0/1585760823 --> v1:172.21.15.1:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fd40c144a10 con 0x7fd40c0b4ff0 2024-03-25T18:29:57.717 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:29:57.716+0000 7fd408ff9640 1 -- 172.21.15.1:0/1585760823 <== mon.1 v1:172.21.15.115:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3308242297 0 0) 0x7fd3ec004eb0 con 0x7fd40c0b4670 2024-03-25T18:29:57.717 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:29:57.716+0000 7fd408ff9640 1 -- 172.21.15.1:0/1585760823 --> v1:172.21.15.115:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fd40c1dbf50 con 0x7fd40c0b4670 2024-03-25T18:29:57.717 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:29:57.716+0000 7fd408ff9640 1 -- 172.21.15.1:0/1585760823 <== mon.1 v1:172.21.15.115:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1586965025 0 0) 0x7fd3fc0019d0 con 0x7fd40c0b4670 2024-03-25T18:29:57.717 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:29:57.716+0000 7fd408ff9640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:29:57.717 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:29:57.716+0000 7fd408ff9640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-03-25T18:29:57.717 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:29:57.716+0000 7fd408ff9640 10 monclient(hunting): dump: 2024-03-25T18:29:57.718 INFO:teuthology.misc.health.smithi001.stderr:epoch 1 2024-03-25T18:29:57.718 INFO:teuthology.misc.health.smithi001.stderr:fsid c8ea72cf-9e99-4383-9b52-6c23d365fdfc 2024-03-25T18:29:57.718 INFO:teuthology.misc.health.smithi001.stderr:last_changed 2024-03-25T18:25:50.510519+0000 2024-03-25T18:29:57.718 INFO:teuthology.misc.health.smithi001.stderr:created 2024-03-25T18:25:50.510519+0000 2024-03-25T18:29:57.718 INFO:teuthology.misc.health.smithi001.stderr:min_mon_release 19 (squid) 2024-03-25T18:29:57.718 INFO:teuthology.misc.health.smithi001.stderr:election_strategy: 3 2024-03-25T18:29:57.718 INFO:teuthology.misc.health.smithi001.stderr:0: v1:172.21.15.1:6789/0 mon.a 2024-03-25T18:29:57.718 INFO:teuthology.misc.health.smithi001.stderr:1: v1:172.21.15.115:6789/0 mon.b 2024-03-25T18:29:57.718 INFO:teuthology.misc.health.smithi001.stderr:2: v1:172.21.15.1:6790/0 mon.c 2024-03-25T18:29:57.718 INFO:teuthology.misc.health.smithi001.stderr: 2024-03-25T18:29:57.718 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:29:57.716+0000 7fd408ff9640 10 monclient(hunting): _finish_auth 0 2024-03-25T18:29:57.729 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:29:57.728+0000 7fd408ff9640 1 -- 172.21.15.1:0/1585760823 <== mon.2 v1:172.21.15.1:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1879555364 0 0) 0x7fd40c144730 con 0x7fd40c13c480 2024-03-25T18:29:57.730 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:29:57.728+0000 7fd408ff9640 1 -- 172.21.15.1:0/1585760823 --> v1:172.21.15.1:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fd3ec004eb0 con 0x7fd40c13c480 2024-03-25T18:29:57.730 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:29:57.728+0000 7fd408ff9640 1 -- 172.21.15.1:0/1585760823 <== mon.2 v1:172.21.15.1:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1586965025 0 0) 0x7fd404001ec0 con 0x7fd40c13c480 2024-03-25T18:29:57.730 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:29:57.728+0000 7fd408ff9640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:29:57.730 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:29:57.728+0000 7fd408ff9640 10 monclient(hunting): got monmap 1 from mon.c (according to old e1) 2024-03-25T18:29:57.730 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:29:57.728+0000 7fd408ff9640 10 monclient(hunting): dump: 2024-03-25T18:29:57.730 INFO:teuthology.misc.health.smithi001.stderr:epoch 1 2024-03-25T18:29:57.731 INFO:teuthology.misc.health.smithi001.stderr:fsid c8ea72cf-9e99-4383-9b52-6c23d365fdfc 2024-03-25T18:29:57.731 INFO:teuthology.misc.health.smithi001.stderr:last_changed 2024-03-25T18:25:50.510519+0000 2024-03-25T18:29:57.731 INFO:teuthology.misc.health.smithi001.stderr:created 2024-03-25T18:25:50.510519+0000 2024-03-25T18:29:57.731 INFO:teuthology.misc.health.smithi001.stderr:min_mon_release 19 (squid) 2024-03-25T18:29:57.731 INFO:teuthology.misc.health.smithi001.stderr:election_strategy: 3 2024-03-25T18:29:57.731 INFO:teuthology.misc.health.smithi001.stderr:0: v1:172.21.15.1:6789/0 mon.a 2024-03-25T18:29:57.731 INFO:teuthology.misc.health.smithi001.stderr:1: v1:172.21.15.115:6789/0 mon.b 2024-03-25T18:29:57.731 INFO:teuthology.misc.health.smithi001.stderr:2: v1:172.21.15.1:6790/0 mon.c 2024-03-25T18:29:57.731 INFO:teuthology.misc.health.smithi001.stderr: 2024-03-25T18:29:57.737 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:29:57.736+0000 7fd408ff9640 1 -- 172.21.15.1:0/1585760823 <== mon.0 v1:172.21.15.1:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1500920034 0 0) 0x7fd40c144a10 con 0x7fd40c0b4ff0 2024-03-25T18:29:57.737 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:29:57.736+0000 7fd408ff9640 1 -- 172.21.15.1:0/1585760823 --> v1:172.21.15.1:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fd40c144730 con 0x7fd40c0b4ff0 2024-03-25T18:29:57.737 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:29:57.736+0000 7fd408ff9640 1 -- 172.21.15.1:0/1585760823 <== mon.0 v1:172.21.15.1:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1586965025 0 0) 0x7fd3f4001860 con 0x7fd40c0b4ff0 2024-03-25T18:29:57.737 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:29:57.736+0000 7fd408ff9640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:29:57.737 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:29:57.736+0000 7fd408ff9640 10 monclient(hunting): got monmap 1 from mon.a (according to old e1) 2024-03-25T18:29:57.737 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:29:57.736+0000 7fd408ff9640 10 monclient(hunting): dump: 2024-03-25T18:29:57.737 INFO:teuthology.misc.health.smithi001.stderr:epoch 1 2024-03-25T18:29:57.737 INFO:teuthology.misc.health.smithi001.stderr:fsid c8ea72cf-9e99-4383-9b52-6c23d365fdfc 2024-03-25T18:29:57.737 INFO:teuthology.misc.health.smithi001.stderr:last_changed 2024-03-25T18:25:50.510519+0000 2024-03-25T18:29:57.737 INFO:teuthology.misc.health.smithi001.stderr:created 2024-03-25T18:25:50.510519+0000 2024-03-25T18:29:57.738 INFO:teuthology.misc.health.smithi001.stderr:min_mon_release 19 (squid) 2024-03-25T18:29:57.738 INFO:teuthology.misc.health.smithi001.stderr:election_strategy: 3 2024-03-25T18:29:57.738 INFO:teuthology.misc.health.smithi001.stderr:0: v1:172.21.15.1:6789/0 mon.a 2024-03-25T18:29:57.738 INFO:teuthology.misc.health.smithi001.stderr:1: v1:172.21.15.115:6789/0 mon.b 2024-03-25T18:29:57.738 INFO:teuthology.misc.health.smithi001.stderr:2: v1:172.21.15.1:6790/0 mon.c 2024-03-25T18:29:57.738 INFO:teuthology.misc.health.smithi001.stderr: 2024-03-25T18:29:57.748 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:29:57.746+0000 7fd408ff9640 1 -- 172.21.15.1:0/1585760823 <== mon.1 v1:172.21.15.115:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 202159388 0 0) 0x7fd40c1dbf50 con 0x7fd40c0b4670 2024-03-25T18:29:57.748 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:29:57.747+0000 7fd408ff9640 1 -- 172.21.15.1:0/1585760823 >> v1:172.21.15.1:6790/0 conn(0x7fd40c13c480 legacy=0x7fd40c1db840 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:29:57.748 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:29:57.747+0000 7fd408ff9640 1 -- 172.21.15.1:0/1585760823 >> v1:172.21.15.1:6789/0 conn(0x7fd40c0b4ff0 legacy=0x7fd40c1db130 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:29:57.748 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:29:57.747+0000 7fd408ff9640 10 monclient: _finish_hunting 0 2024-03-25T18:29:57.748 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:29:57.747+0000 7fd408ff9640 1 monclient: found mon.b 2024-03-25T18:29:57.748 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:29:57.747+0000 7fd408ff9640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:29:57.748 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:29:57.747+0000 7fd408ff9640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.115:6789/0 2024-03-25T18:29:57.748 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:29:57.747+0000 7fd408ff9640 1 -- 172.21.15.1:0/1585760823 --> v1:172.21.15.115:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd40c13eb00 con 0x7fd40c0b4670 2024-03-25T18:29:57.748 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:29:57.747+0000 7fd408ff9640 10 monclient: _finish_auth 0 2024-03-25T18:29:57.749 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:29:57.747+0000 7fd408ff9640 10 monclient: _check_auth_tickets 2024-03-25T18:29:57.749 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:29:57.747+0000 7fd408ff9640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:29:57.749 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:29:57.747+0000 7fd4118fb640 5 monclient: authenticate success, global_id 4363 2024-03-25T18:29:57.749 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:29:57.747+0000 7fd4118fb640 10 monclient: _renew_subs 2024-03-25T18:29:57.749 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:29:57.747+0000 7fd4118fb640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.115:6789/0 2024-03-25T18:29:57.749 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:29:57.747+0000 7fd4118fb640 1 -- 172.21.15.1:0/1585760823 --> v1:172.21.15.115:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd40c145540 con 0x7fd40c0b4670 2024-03-25T18:29:57.749 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:29:57.747+0000 7fd4118fb640 10 client.4363.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T18:29:57.749 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:29:57.747+0000 7fd4118fb640 10 monclient: _renew_subs 2024-03-25T18:29:57.749 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:29:57.747+0000 7fd4118fb640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.115:6789/0 2024-03-25T18:29:57.749 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:29:57.747+0000 7fd4118fb640 1 -- 172.21.15.1:0/1585760823 --> v1:172.21.15.115:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7fd40c13f730 con 0x7fd40c0b4670 2024-03-25T18:29:57.749 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:29:57.747+0000 7fd4118fb640 1 librados: init done 2024-03-25T18:29:57.750 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:29:57.748+0000 7fd4118fb640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T18:29:57.750 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:29:57.748+0000 7fd4118fb640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:29:57.750 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:29:57.748+0000 7fd4118fb640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.115:6789/0 2024-03-25T18:29:57.750 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:29:57.748+0000 7fd4118fb640 1 -- 172.21.15.1:0/1585760823 --> v1:172.21.15.115:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd40c06dbd0 con 0x7fd40c0b4670 2024-03-25T18:29:58.042 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:29:58.040+0000 7fd408ff9640 1 -- 172.21.15.1:0/1585760823 <== mon.1 v1:172.21.15.115:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fd3fc003610 con 0x7fd40c0b4670 2024-03-25T18:29:58.042 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:29:58.041+0000 7fd408ff9640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:29:58.042 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:29:58.041+0000 7fd408ff9640 1 -- 172.21.15.1:0/1585760823 <== mon.1 v1:172.21.15.115:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1586965025 0 0) 0x7fd3fc003a20 con 0x7fd40c0b4670 2024-03-25T18:29:58.042 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:29:58.041+0000 7fd408ff9640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:29:58.042 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:29:58.041+0000 7fd408ff9640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-25T18:29:58.042 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:29:58.041+0000 7fd408ff9640 10 monclient: dump: 2024-03-25T18:29:58.042 INFO:teuthology.misc.health.smithi001.stderr:epoch 1 2024-03-25T18:29:58.042 INFO:teuthology.misc.health.smithi001.stderr:fsid c8ea72cf-9e99-4383-9b52-6c23d365fdfc 2024-03-25T18:29:58.042 INFO:teuthology.misc.health.smithi001.stderr:last_changed 2024-03-25T18:25:50.510519+0000 2024-03-25T18:29:58.042 INFO:teuthology.misc.health.smithi001.stderr:created 2024-03-25T18:25:50.510519+0000 2024-03-25T18:29:58.043 INFO:teuthology.misc.health.smithi001.stderr:min_mon_release 19 (squid) 2024-03-25T18:29:58.043 INFO:teuthology.misc.health.smithi001.stderr:election_strategy: 3 2024-03-25T18:29:58.043 INFO:teuthology.misc.health.smithi001.stderr:0: v1:172.21.15.1:6789/0 mon.a 2024-03-25T18:29:58.043 INFO:teuthology.misc.health.smithi001.stderr:1: v1:172.21.15.115:6789/0 mon.b 2024-03-25T18:29:58.043 INFO:teuthology.misc.health.smithi001.stderr:2: v1:172.21.15.1:6790/0 mon.c 2024-03-25T18:29:58.043 INFO:teuthology.misc.health.smithi001.stderr: 2024-03-25T18:29:58.043 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:29:58.041+0000 7fd409ffb640 4 set_mon_vals no callback set 2024-03-25T18:29:58.044 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:29:58.042+0000 7fd408ff9640 1 -- 172.21.15.1:0/1585760823 <== mon.1 v1:172.21.15.115:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 21787975 0 0) 0x7fd3fc003c40 con 0x7fd40c0b4670 2024-03-25T18:29:58.044 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:29:58.042+0000 7fd403fff640 10 monclient: get_auth_request con 0x7fd3ec06e0f0 auth_method 0 2024-03-25T18:29:58.044 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:29:58.043+0000 7fd408ff9640 10 client.4363.objecter ms_handle_connect 0x7fd3ec06e0f0 2024-03-25T18:29:58.046 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:29:58.045+0000 7fd408ff9640 1 -- 172.21.15.1:0/1585760823 <== mon.1 v1:172.21.15.115:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 996254230 0 0) 0x7fd3fc0547d0 con 0x7fd40c0b4670 2024-03-25T18:29:58.046 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:29:58.046+0000 7fd408ff9640 10 client.4363.objecter ms_dispatch 0x7fd40c000b90 osd_map(10..10 src has 1..10) v4 2024-03-25T18:29:58.046 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:29:58.046+0000 7fd408ff9640 3 client.4363.objecter handle_osd_map got epochs [10,10] > 0 2024-03-25T18:29:58.047 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:29:58.046+0000 7fd408ff9640 3 client.4363.objecter handle_osd_map decoding full epoch 10 2024-03-25T18:29:58.047 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:29:58.046+0000 7fd408ff9640 20 client.4363.objecter dump_active .. 0 homeless 2024-03-25T18:29:58.047 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:29:58.046+0000 7fd408ff9640 1 -- 172.21.15.1:0/1585760823 <== mon.1 v1:172.21.15.115:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7fd3fc0502e0 con 0x7fd40c0b4670 2024-03-25T18:29:58.047 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:29:58.046+0000 7fd408ff9640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:29:58.047 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:29:58.046+0000 7fd408ff9640 10 monclient: _finish_command 1 = system:0 2024-03-25T18:29:58.306 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:29:58.305+0000 7fd4118fb640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:29:58.307 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:29:58.305+0000 7fd4118fb640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:29:58.307 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:29:58.305+0000 7fd4118fb640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.115:6789/0 2024-03-25T18:29:58.307 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:29:58.305+0000 7fd4118fb640 1 -- 172.21.15.1:0/1585760823 --> v1:172.21.15.115:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fd40c0688c0 con 0x7fd40c0b4670 2024-03-25T18:29:58.351 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:29:58.350+0000 7fd408ff9640 1 -- 172.21.15.1:0/1585760823 <== mon.1 v1:172.21.15.115:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 4092213348) 0x7fd3fc052210 con 0x7fd40c0b4670 2024-03-25T18:29:58.351 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:29:58.350+0000 7fd408ff9640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:29:58.351 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:29:58.350+0000 7fd408ff9640 10 monclient: _finish_command 2 = system:0 2024-03-25T18:29:58.351 INFO:teuthology.misc.health.smithi001.stdout: 2024-03-25T18:29:58.351 INFO:teuthology.misc.health.smithi001.stdout:{"epoch":10,"fsid":"c8ea72cf-9e99-4383-9b52-6c23d365fdfc","created":"2024-03-25T18:26:38.856638+0000","modified":"2024-03-25T18:26:44.224208+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T18:26:44.224208+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"b9f34922-3a84-49be-8426-807c1eceaefc","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"42bf306f-55e3-4634-9874-8b4b73ac6644","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"292cb23a-18a9-4207-a7cc-4d5727feaec0","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"45e0e9ab-898d-4673-9590-b4117bc5e84a","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"831a5a09-73ab-4a61-9872-088e76bdbc66","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"88cd903d-3b55-4840-9396-6e3cfe005282","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"89db7355-bc9d-46f4-8ab0-e8d9e744e026","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"cac47bee-0914-4f08-896f-348838882866","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T18:29:58.353 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:29:58.352+0000 7fd4118fb640 10 librados: watch_flush enter 2024-03-25T18:29:58.354 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:29:58.352+0000 7fd4118fb640 10 librados: watch_flush exit 2024-03-25T18:29:58.354 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:29:58.352+0000 7fd4118fb640 20 client.4363.objecter shutdown clearing up homeless session... 2024-03-25T18:29:58.354 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:29:58.352+0000 7fd4118fb640 10 client.4363.objecter successfully canceled tick 2024-03-25T18:29:58.354 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:29:58.352+0000 7fd4118fb640 1 -- 172.21.15.1:0/1585760823 >> v1:172.21.15.115:6800/4237379176 conn(0x7fd3ec06e0f0 legacy=0x7fd3ec0705b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:29:58.354 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:29:58.352+0000 7fd4118fb640 10 monclient: shutdown 2024-03-25T18:29:58.354 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:29:58.352+0000 7fd4118fb640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:29:58.354 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:29:58.352+0000 7fd4118fb640 1 -- 172.21.15.1:0/1585760823 >> v1:172.21.15.115:6789/0 conn(0x7fd40c0b4670 legacy=0x7fd40c144e30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:29:58.354 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:29:58.352+0000 7fd4118fb640 1 -- 172.21.15.1:0/1585760823 shutdown_connections 2024-03-25T18:29:58.354 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:29:58.353+0000 7fd4118fb640 1 -- 172.21.15.1:0/1585760823 >> 172.21.15.1:0/1585760823 conn(0x7fd40c0617a0 msgr2=0x7fd40c140760 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:29:58.354 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:29:58.353+0000 7fd4118fb640 1 -- 172.21.15.1:0/1585760823 shutdown_connections 2024-03-25T18:29:58.355 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:29:58.353+0000 7fd4118fb640 1 -- 172.21.15.1:0/1585760823 wait complete. 2024-03-25T18:29:58.355 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:29:58.353+0000 7fd4118fb640 1 librados: shutdown 2024-03-25T18:29:58.365 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T18:30:05.172 DEBUG:teuthology.orchestra.run.smithi001:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T18:30:05.291 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:30:05.289+0000 7f73a7f8c640 10 monclient: get_monmap_and_config 2024-03-25T18:30:05.292 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:30:05.289+0000 7f73a7f8c640 10 monclient: build_initial_monmap 2024-03-25T18:30:05.292 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:30:05.289+0000 7f73a7f8c640 10 monclient: monmap: 2024-03-25T18:30:05.292 INFO:teuthology.misc.health.smithi001.stderr:epoch 0 2024-03-25T18:30:05.292 INFO:teuthology.misc.health.smithi001.stderr:fsid c8ea72cf-9e99-4383-9b52-6c23d365fdfc 2024-03-25T18:30:05.292 INFO:teuthology.misc.health.smithi001.stderr:last_changed 2024-03-25T18:30:05.291392+0000 2024-03-25T18:30:05.292 INFO:teuthology.misc.health.smithi001.stderr:created 2024-03-25T18:30:05.291392+0000 2024-03-25T18:30:05.292 INFO:teuthology.misc.health.smithi001.stderr:min_mon_release 0 (unknown) 2024-03-25T18:30:05.292 INFO:teuthology.misc.health.smithi001.stderr:election_strategy: 3 2024-03-25T18:30:05.292 INFO:teuthology.misc.health.smithi001.stderr:0: v1:172.21.15.1:6789/0 mon.noname-a 2024-03-25T18:30:05.292 INFO:teuthology.misc.health.smithi001.stderr:1: v1:172.21.15.115:6789/0 mon.noname-c 2024-03-25T18:30:05.292 INFO:teuthology.misc.health.smithi001.stderr:2: v1:172.21.15.1:6790/0 mon.noname-b 2024-03-25T18:30:05.292 INFO:teuthology.misc.health.smithi001.stderr: 2024-03-25T18:30:05.292 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:30:05.290+0000 7f73a7f8c640 1 Processor -- start 2024-03-25T18:30:05.293 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:30:05.290+0000 7f73a7f8c640 1 -- start start 2024-03-25T18:30:05.293 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:30:05.290+0000 7f73a7f8c640 10 monclient: init 2024-03-25T18:30:05.294 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:30:05.291+0000 7f73a7f8c640 10 monclient: _reopen_session rank -1 2024-03-25T18:30:05.294 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:30:05.291+0000 7f73a7f8c640 10 monclient: _add_conns ranks=[2,0,1] 2024-03-25T18:30:05.294 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:30:05.291+0000 7f73a7f8c640 10 monclient(hunting): picked mon.noname-b con 0x7f73a00b4ff0 addr v1:172.21.15.1:6790/0 2024-03-25T18:30:05.294 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:30:05.291+0000 7f73a7f8c640 10 monclient(hunting): picked mon.noname-a con 0x7f73a00b4670 addr v1:172.21.15.1:6789/0 2024-03-25T18:30:05.294 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:30:05.291+0000 7f73a7f8c640 10 monclient(hunting): picked mon.noname-c con 0x7f73a013c480 addr v1:172.21.15.115:6789/0 2024-03-25T18:30:05.294 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:30:05.291+0000 7f73a7f8c640 1 -- --> v1:172.21.15.1:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f73a00ab0e0 con 0x7f73a00b4670 2024-03-25T18:30:05.294 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:30:05.291+0000 7f73a7f8c640 1 -- --> v1:172.21.15.115:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f73a00ab3c0 con 0x7f73a013c480 2024-03-25T18:30:05.294 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:30:05.291+0000 7f73a7f8c640 1 -- --> v1:172.21.15.1:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f73a00a9f30 con 0x7f73a00b4ff0 2024-03-25T18:30:05.294 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:30:05.291+0000 7f73a7f8c640 10 monclient(hunting): _renew_subs 2024-03-25T18:30:05.294 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:30:05.291+0000 7f73a7f8c640 10 monclient(hunting): authenticate will time out at 7685.762227s 2024-03-25T18:30:05.296 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:30:05.294+0000 7f73a5527640 1 --1- >> v1:172.21.15.1:6789/0 conn(0x7f73a00b4670 0x7f73a01425c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.1:6789/0 says I am v1:172.21.15.1:54274/0 (socket says 172.21.15.1:54274) 2024-03-25T18:30:05.296 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:30:05.294+0000 7f73a5527640 1 -- 172.21.15.1:0/1774829857 learned_addr learned my addr 172.21.15.1:0/1774829857 (peer_addr_for_me v1:172.21.15.1:0/0) 2024-03-25T18:30:05.307 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:30:05.306+0000 7f73a4d26640 1 -- 172.21.15.1:0/1774829857 <== mon.2 v1:172.21.15.1:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3001188752 0 0) 0x7f73a00a9f30 con 0x7f73a00b4ff0 2024-03-25T18:30:05.308 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:30:05.306+0000 7f73a4d26640 10 monclient(hunting): _negotiate 2024-03-25T18:30:05.308 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:30:05.306+0000 7f73a4d26640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:30:05.308 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:30:05.306+0000 7f73a4d26640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:30:05.308 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:30:05.306+0000 7f73a4d26640 10 monclient(hunting): my global_id is 4373 2024-03-25T18:30:05.308 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:30:05.306+0000 7f73a4d26640 1 -- 172.21.15.1:0/1774829857 --> v1:172.21.15.1:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f738c003520 con 0x7f73a00b4ff0 2024-03-25T18:30:05.310 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:30:05.309+0000 7f73a4d26640 1 -- 172.21.15.1:0/1774829857 <== mon.0 v1:172.21.15.1:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1668556207 0 0) 0x7f73a00ab0e0 con 0x7f73a00b4670 2024-03-25T18:30:05.311 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:30:05.309+0000 7f73a4d26640 10 monclient(hunting): _negotiate 2024-03-25T18:30:05.311 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:30:05.309+0000 7f73a4d26640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:30:05.311 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:30:05.309+0000 7f73a4d26640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:30:05.311 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:30:05.309+0000 7f73a4d26640 10 monclient(hunting): my global_id is 4365 2024-03-25T18:30:05.311 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:30:05.309+0000 7f73a4d26640 1 -- 172.21.15.1:0/1774829857 --> v1:172.21.15.1:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f73a00a9f30 con 0x7f73a00b4670 2024-03-25T18:30:05.312 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:30:05.310+0000 7f73a4d26640 1 -- 172.21.15.1:0/1774829857 <== mon.1 v1:172.21.15.115:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2992765816 0 0) 0x7f73a00ab3c0 con 0x7f73a013c480 2024-03-25T18:30:05.312 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:30:05.310+0000 7f73a4d26640 10 monclient(hunting): _negotiate 2024-03-25T18:30:05.312 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:30:05.310+0000 7f73a4d26640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:30:05.312 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:30:05.310+0000 7f73a4d26640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:30:05.312 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:30:05.310+0000 7f73a4d26640 10 monclient(hunting): my global_id is 4366 2024-03-25T18:30:05.312 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:30:05.311+0000 7f73a4d26640 1 -- 172.21.15.1:0/1774829857 --> v1:172.21.15.115:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f73a00ab0e0 con 0x7f73a013c480 2024-03-25T18:30:05.320 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:30:05.318+0000 7f73a4d26640 1 -- 172.21.15.1:0/1774829857 <== mon.0 v1:172.21.15.1:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 36502141 0 0) 0x7f73a00a9f30 con 0x7f73a00b4670 2024-03-25T18:30:05.320 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:30:05.318+0000 7f73a4d26640 1 -- 172.21.15.1:0/1774829857 >> v1:172.21.15.1:6790/0 conn(0x7f73a00b4ff0 legacy=0x7f73a013bd70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:30:05.320 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:30:05.318+0000 7f73a4d26640 1 -- 172.21.15.1:0/1774829857 >> v1:172.21.15.115:6789/0 conn(0x7f73a013c480 legacy=0x7f73a014dc00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:30:05.320 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:30:05.318+0000 7f73a4d26640 10 monclient: _finish_hunting 0 2024-03-25T18:30:05.320 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:30:05.318+0000 7f73a4d26640 1 monclient: found mon.noname-a 2024-03-25T18:30:05.321 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:30:05.318+0000 7f73a4d26640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:30:05.321 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:30:05.318+0000 7f73a4d26640 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.1:6789/0 2024-03-25T18:30:05.321 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:30:05.318+0000 7f73a4d26640 1 -- 172.21.15.1:0/1774829857 --> v1:172.21.15.1:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f73a00a9b70 con 0x7f73a00b4670 2024-03-25T18:30:05.321 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:30:05.318+0000 7f73a4d26640 10 monclient: _finish_auth 0 2024-03-25T18:30:05.321 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:30:05.318+0000 7f73a4d26640 10 monclient: _check_auth_tickets 2024-03-25T18:30:05.321 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:30:05.318+0000 7f73a4d26640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:30:05.321 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:30:05.318+0000 7f73a4d26640 1 -- 172.21.15.1:0/1774829857 <== mon.0 v1:172.21.15.1:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1586965025 0 0) 0x7f7390001670 con 0x7f73a00b4670 2024-03-25T18:30:05.321 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:30:05.318+0000 7f73a7f8c640 5 monclient: authenticate success, global_id 4365 2024-03-25T18:30:05.321 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:30:05.318+0000 7f73a7f8c640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T18:30:05.321 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:30:05.318+0000 7f73a4d26640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:30:05.321 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:30:05.318+0000 7f73a4d26640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-03-25T18:30:05.321 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:30:05.318+0000 7f73a4d26640 10 monclient: dump: 2024-03-25T18:30:05.321 INFO:teuthology.misc.health.smithi001.stderr:epoch 1 2024-03-25T18:30:05.321 INFO:teuthology.misc.health.smithi001.stderr:fsid c8ea72cf-9e99-4383-9b52-6c23d365fdfc 2024-03-25T18:30:05.322 INFO:teuthology.misc.health.smithi001.stderr:last_changed 2024-03-25T18:25:50.510519+0000 2024-03-25T18:30:05.322 INFO:teuthology.misc.health.smithi001.stderr:created 2024-03-25T18:25:50.510519+0000 2024-03-25T18:30:05.322 INFO:teuthology.misc.health.smithi001.stderr:min_mon_release 19 (squid) 2024-03-25T18:30:05.322 INFO:teuthology.misc.health.smithi001.stderr:election_strategy: 3 2024-03-25T18:30:05.322 INFO:teuthology.misc.health.smithi001.stderr:0: v1:172.21.15.1:6789/0 mon.a 2024-03-25T18:30:05.322 INFO:teuthology.misc.health.smithi001.stderr:1: v1:172.21.15.115:6789/0 mon.b 2024-03-25T18:30:05.322 INFO:teuthology.misc.health.smithi001.stderr:2: v1:172.21.15.1:6790/0 mon.c 2024-03-25T18:30:05.322 INFO:teuthology.misc.health.smithi001.stderr: 2024-03-25T18:30:05.322 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:30:05.318+0000 7f73a7f8c640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T18:30:05.332 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:30:05.330+0000 7f73a4d26640 1 -- 172.21.15.1:0/1774829857 <== mon.0 v1:172.21.15.1:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f73900017d0 con 0x7f73a00b4670 2024-03-25T18:30:05.332 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:30:05.330+0000 7f73a4d26640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:30:05.332 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:30:05.330+0000 7f73a4d26640 1 -- 172.21.15.1:0/1774829857 <== mon.0 v1:172.21.15.1:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1586965025 0 0) 0x7f7390001be0 con 0x7f73a00b4670 2024-03-25T18:30:05.332 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:30:05.330+0000 7f73a7f8c640 10 monclient: get_monmap_and_config success 2024-03-25T18:30:05.332 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:30:05.330+0000 7f73a7f8c640 4 set_mon_vals no callback set 2024-03-25T18:30:05.332 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:30:05.330+0000 7f73a4d26640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:30:05.332 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:30:05.330+0000 7f73a4d26640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-25T18:30:05.332 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:30:05.330+0000 7f73a4d26640 10 monclient: dump: 2024-03-25T18:30:05.332 INFO:teuthology.misc.health.smithi001.stderr:epoch 1 2024-03-25T18:30:05.332 INFO:teuthology.misc.health.smithi001.stderr:fsid c8ea72cf-9e99-4383-9b52-6c23d365fdfc 2024-03-25T18:30:05.332 INFO:teuthology.misc.health.smithi001.stderr:last_changed 2024-03-25T18:25:50.510519+0000 2024-03-25T18:30:05.332 INFO:teuthology.misc.health.smithi001.stderr:created 2024-03-25T18:25:50.510519+0000 2024-03-25T18:30:05.332 INFO:teuthology.misc.health.smithi001.stderr:min_mon_release 19 (squid) 2024-03-25T18:30:05.333 INFO:teuthology.misc.health.smithi001.stderr:election_strategy: 3 2024-03-25T18:30:05.333 INFO:teuthology.misc.health.smithi001.stderr:0: v1:172.21.15.1:6789/0 mon.a 2024-03-25T18:30:05.333 INFO:teuthology.misc.health.smithi001.stderr:1: v1:172.21.15.115:6789/0 mon.b 2024-03-25T18:30:05.333 INFO:teuthology.misc.health.smithi001.stderr:2: v1:172.21.15.1:6790/0 mon.c 2024-03-25T18:30:05.333 INFO:teuthology.misc.health.smithi001.stderr: 2024-03-25T18:30:05.334 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:30:05.331+0000 7f73a7f8c640 10 monclient: shutdown 2024-03-25T18:30:05.334 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:30:05.331+0000 7f73a7f8c640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:30:05.334 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:30:05.331+0000 7f73a7f8c640 1 -- 172.21.15.1:0/1774829857 >> v1:172.21.15.1:6789/0 conn(0x7f73a00b4670 legacy=0x7f73a01425c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:30:05.334 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:30:05.331+0000 7f73a7f8c640 1 -- 172.21.15.1:0/1774829857 shutdown_connections 2024-03-25T18:30:05.334 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:30:05.331+0000 7f73a7f8c640 1 -- 172.21.15.1:0/1774829857 >> 172.21.15.1:0/1774829857 conn(0x7f73a00617a0 msgr2=0x7f73a0061bb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:30:05.334 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:30:05.331+0000 7f73a7f8c640 1 -- 172.21.15.1:0/1774829857 shutdown_connections 2024-03-25T18:30:05.334 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:30:05.331+0000 7f73a7f8c640 1 -- 172.21.15.1:0/1774829857 wait complete. 2024-03-25T18:30:05.335 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:30:05.332+0000 7f73a7f8c640 10 monclient: build_initial_monmap 2024-03-25T18:30:05.335 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:30:05.332+0000 7f73a7f8c640 10 monclient: monmap: 2024-03-25T18:30:05.335 INFO:teuthology.misc.health.smithi001.stderr:epoch 0 2024-03-25T18:30:05.335 INFO:teuthology.misc.health.smithi001.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T18:30:05.335 INFO:teuthology.misc.health.smithi001.stderr:last_changed 0.000000 2024-03-25T18:30:05.335 INFO:teuthology.misc.health.smithi001.stderr:created 0.000000 2024-03-25T18:30:05.335 INFO:teuthology.misc.health.smithi001.stderr:min_mon_release 0 (unknown) 2024-03-25T18:30:05.335 INFO:teuthology.misc.health.smithi001.stderr:election_strategy: 1 2024-03-25T18:30:05.335 INFO:teuthology.misc.health.smithi001.stderr:0: v1:172.21.15.1:6789/0 mon.noname-a 2024-03-25T18:30:05.335 INFO:teuthology.misc.health.smithi001.stderr:1: v1:172.21.15.115:6789/0 mon.noname-b 2024-03-25T18:30:05.335 INFO:teuthology.misc.health.smithi001.stderr:2: v1:172.21.15.1:6790/0 mon.noname-c 2024-03-25T18:30:05.335 INFO:teuthology.misc.health.smithi001.stderr: 2024-03-25T18:30:05.335 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:30:05.332+0000 7f73a7f8c640 1 librados: starting msgr at 2024-03-25T18:30:05.335 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:30:05.332+0000 7f73a7f8c640 1 librados: starting objecter 2024-03-25T18:30:05.335 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:30:05.332+0000 7f73a7f8c640 1 Processor -- start 2024-03-25T18:30:05.336 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:30:05.332+0000 7f73a7f8c640 1 -- start start 2024-03-25T18:30:05.336 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:30:05.332+0000 7f73a7f8c640 1 librados: setting wanted keys 2024-03-25T18:30:05.336 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:30:05.332+0000 7f73a7f8c640 1 librados: calling monclient init 2024-03-25T18:30:05.336 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:30:05.332+0000 7f73a7f8c640 10 monclient: init 2024-03-25T18:30:05.337 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:30:05.333+0000 7f73a7f8c640 10 monclient: _reopen_session rank -1 2024-03-25T18:30:05.337 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:30:05.333+0000 7f73a7f8c640 10 monclient: _add_conns ranks=[1,2,0] 2024-03-25T18:30:05.337 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:30:05.333+0000 7f73a7f8c640 10 monclient(hunting): picked mon.noname-b con 0x7f73a00b4670 addr v1:172.21.15.115:6789/0 2024-03-25T18:30:05.337 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:30:05.333+0000 7f73a7f8c640 10 monclient(hunting): picked mon.noname-c con 0x7f73a00b4ff0 addr v1:172.21.15.1:6790/0 2024-03-25T18:30:05.337 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:30:05.333+0000 7f73a7f8c640 10 monclient(hunting): picked mon.noname-a con 0x7f73a013c480 addr v1:172.21.15.1:6789/0 2024-03-25T18:30:05.337 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:30:05.333+0000 7f73a7f8c640 1 -- --> v1:172.21.15.1:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f73a01df920 con 0x7f73a013c480 2024-03-25T18:30:05.337 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:30:05.333+0000 7f73a7f8c640 1 -- --> v1:172.21.15.115:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f73a01467e0 con 0x7f73a00b4670 2024-03-25T18:30:05.337 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:30:05.333+0000 7f73a7f8c640 1 -- --> v1:172.21.15.1:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f73a01382d0 con 0x7f73a00b4ff0 2024-03-25T18:30:05.337 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:30:05.333+0000 7f73a7f8c640 10 monclient(hunting): _renew_subs 2024-03-25T18:30:05.337 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:30:05.333+0000 7f73a7f8c640 10 monclient(hunting): authenticate will time out at 7685.804471s 2024-03-25T18:30:05.339 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:30:05.337+0000 7f73a6529640 1 --1- >> v1:172.21.15.1:6789/0 conn(0x7f73a013c480 0x7f73a0137bc0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.1:6789/0 says I am v1:172.21.15.1:54310/0 (socket says 172.21.15.1:54310) 2024-03-25T18:30:05.339 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:30:05.337+0000 7f73a6529640 1 -- 172.21.15.1:0/993969202 learned_addr learned my addr 172.21.15.1:0/993969202 (peer_addr_for_me v1:172.21.15.1:0/0) 2024-03-25T18:30:05.347 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:30:05.346+0000 7f73967fc640 10 client.?.objecter ms_handle_connect 0x7f73a013c480 2024-03-25T18:30:05.347 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:30:05.346+0000 7f73967fc640 10 client.?.objecter resend_mon_ops 2024-03-25T18:30:05.354 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:30:05.352+0000 7f73967fc640 10 client.?.objecter ms_handle_connect 0x7f73a00b4670 2024-03-25T18:30:05.354 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:30:05.352+0000 7f73967fc640 10 client.?.objecter resend_mon_ops 2024-03-25T18:30:05.360 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:30:05.358+0000 7f73967fc640 10 client.?.objecter ms_handle_connect 0x7f73a00b4ff0 2024-03-25T18:30:05.360 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:30:05.358+0000 7f73967fc640 10 client.?.objecter resend_mon_ops 2024-03-25T18:30:05.363 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:30:05.361+0000 7f73967fc640 1 -- 172.21.15.1:0/993969202 <== mon.1 v1:172.21.15.115:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1307822040 0 0) 0x7f73a01467e0 con 0x7f73a00b4670 2024-03-25T18:30:05.363 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:30:05.361+0000 7f73967fc640 10 monclient(hunting): _negotiate 2024-03-25T18:30:05.363 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:30:05.361+0000 7f73967fc640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:30:05.363 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:30:05.361+0000 7f73967fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:30:05.363 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:30:05.361+0000 7f73967fc640 10 monclient(hunting): my global_id is 4369 2024-03-25T18:30:05.363 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:30:05.361+0000 7f73967fc640 1 -- 172.21.15.1:0/993969202 --> v1:172.21.15.115:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f7384004eb0 con 0x7f73a00b4670 2024-03-25T18:30:05.364 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:30:05.362+0000 7f73967fc640 1 -- 172.21.15.1:0/993969202 <== mon.0 v1:172.21.15.1:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1688743666 0 0) 0x7f73a01df920 con 0x7f73a013c480 2024-03-25T18:30:05.364 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:30:05.362+0000 7f73967fc640 10 monclient(hunting): _negotiate 2024-03-25T18:30:05.364 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:30:05.362+0000 7f73967fc640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:30:05.364 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:30:05.362+0000 7f73967fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:30:05.364 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:30:05.362+0000 7f73967fc640 10 monclient(hunting): my global_id is 4368 2024-03-25T18:30:05.364 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:30:05.362+0000 7f73967fc640 1 -- 172.21.15.1:0/993969202 --> v1:172.21.15.1:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f73a01467e0 con 0x7f73a013c480 2024-03-25T18:30:05.372 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:30:05.371+0000 7f73967fc640 1 -- 172.21.15.1:0/993969202 <== mon.2 v1:172.21.15.1:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2771617473 0 0) 0x7f73a01382d0 con 0x7f73a00b4ff0 2024-03-25T18:30:05.372 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:30:05.371+0000 7f73967fc640 10 monclient(hunting): _negotiate 2024-03-25T18:30:05.373 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:30:05.371+0000 7f73967fc640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:30:05.373 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:30:05.371+0000 7f73967fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:30:05.373 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:30:05.371+0000 7f73967fc640 10 monclient(hunting): my global_id is 4376 2024-03-25T18:30:05.373 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:30:05.371+0000 7f73967fc640 1 -- 172.21.15.1:0/993969202 --> v1:172.21.15.1:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f73a01df920 con 0x7f73a00b4ff0 2024-03-25T18:30:05.374 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:30:05.373+0000 7f73967fc640 1 -- 172.21.15.1:0/993969202 <== mon.0 v1:172.21.15.1:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 152573099 0 0) 0x7f73a01467e0 con 0x7f73a013c480 2024-03-25T18:30:05.374 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:30:05.373+0000 7f73967fc640 1 -- 172.21.15.1:0/993969202 --> v1:172.21.15.1:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f73a01382d0 con 0x7f73a013c480 2024-03-25T18:30:05.375 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:30:05.373+0000 7f73967fc640 1 -- 172.21.15.1:0/993969202 <== mon.0 v1:172.21.15.1:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1586965025 0 0) 0x7f7398001c50 con 0x7f73a013c480 2024-03-25T18:30:05.375 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:30:05.373+0000 7f73967fc640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:30:05.375 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:30:05.373+0000 7f73967fc640 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2024-03-25T18:30:05.375 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:30:05.373+0000 7f73967fc640 10 monclient(hunting): dump: 2024-03-25T18:30:05.375 INFO:teuthology.misc.health.smithi001.stderr:epoch 1 2024-03-25T18:30:05.375 INFO:teuthology.misc.health.smithi001.stderr:fsid c8ea72cf-9e99-4383-9b52-6c23d365fdfc 2024-03-25T18:30:05.375 INFO:teuthology.misc.health.smithi001.stderr:last_changed 2024-03-25T18:25:50.510519+0000 2024-03-25T18:30:05.375 INFO:teuthology.misc.health.smithi001.stderr:created 2024-03-25T18:25:50.510519+0000 2024-03-25T18:30:05.375 INFO:teuthology.misc.health.smithi001.stderr:min_mon_release 19 (squid) 2024-03-25T18:30:05.375 INFO:teuthology.misc.health.smithi001.stderr:election_strategy: 3 2024-03-25T18:30:05.375 INFO:teuthology.misc.health.smithi001.stderr:0: v1:172.21.15.1:6789/0 mon.a 2024-03-25T18:30:05.375 INFO:teuthology.misc.health.smithi001.stderr:1: v1:172.21.15.115:6789/0 mon.b 2024-03-25T18:30:05.375 INFO:teuthology.misc.health.smithi001.stderr:2: v1:172.21.15.1:6790/0 mon.c 2024-03-25T18:30:05.375 INFO:teuthology.misc.health.smithi001.stderr: 2024-03-25T18:30:05.375 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:30:05.373+0000 7f73967fc640 10 monclient(hunting): _finish_auth 0 2024-03-25T18:30:05.379 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:30:05.377+0000 7f73967fc640 1 -- 172.21.15.1:0/993969202 <== mon.2 v1:172.21.15.1:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3594455419 0 0) 0x7f73a01df920 con 0x7f73a00b4ff0 2024-03-25T18:30:05.379 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:30:05.377+0000 7f73967fc640 1 -- 172.21.15.1:0/993969202 --> v1:172.21.15.1:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f73a01467e0 con 0x7f73a00b4ff0 2024-03-25T18:30:05.379 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:30:05.377+0000 7f73967fc640 1 -- 172.21.15.1:0/993969202 <== mon.2 v1:172.21.15.1:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1586965025 0 0) 0x7f7390001ae0 con 0x7f73a00b4ff0 2024-03-25T18:30:05.379 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:30:05.377+0000 7f73967fc640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:30:05.379 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:30:05.377+0000 7f73967fc640 10 monclient(hunting): got monmap 1 from mon.c (according to old e1) 2024-03-25T18:30:05.379 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:30:05.377+0000 7f73967fc640 10 monclient(hunting): dump: 2024-03-25T18:30:05.379 INFO:teuthology.misc.health.smithi001.stderr:epoch 1 2024-03-25T18:30:05.380 INFO:teuthology.misc.health.smithi001.stderr:fsid c8ea72cf-9e99-4383-9b52-6c23d365fdfc 2024-03-25T18:30:05.380 INFO:teuthology.misc.health.smithi001.stderr:last_changed 2024-03-25T18:25:50.510519+0000 2024-03-25T18:30:05.380 INFO:teuthology.misc.health.smithi001.stderr:created 2024-03-25T18:25:50.510519+0000 2024-03-25T18:30:05.380 INFO:teuthology.misc.health.smithi001.stderr:min_mon_release 19 (squid) 2024-03-25T18:30:05.380 INFO:teuthology.misc.health.smithi001.stderr:election_strategy: 3 2024-03-25T18:30:05.380 INFO:teuthology.misc.health.smithi001.stderr:0: v1:172.21.15.1:6789/0 mon.a 2024-03-25T18:30:05.380 INFO:teuthology.misc.health.smithi001.stderr:1: v1:172.21.15.115:6789/0 mon.b 2024-03-25T18:30:05.380 INFO:teuthology.misc.health.smithi001.stderr:2: v1:172.21.15.1:6790/0 mon.c 2024-03-25T18:30:05.380 INFO:teuthology.misc.health.smithi001.stderr: 2024-03-25T18:30:05.383 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:30:05.379+0000 7f73967fc640 1 -- 172.21.15.1:0/993969202 <== mon.1 v1:172.21.15.115:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1955509452 0 0) 0x7f7384004eb0 con 0x7f73a00b4670 2024-03-25T18:30:05.383 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:30:05.379+0000 7f73967fc640 1 -- 172.21.15.1:0/993969202 --> v1:172.21.15.115:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f73a01df920 con 0x7f73a00b4670 2024-03-25T18:30:05.383 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:30:05.379+0000 7f73967fc640 1 -- 172.21.15.1:0/993969202 <== mon.1 v1:172.21.15.115:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1586965025 0 0) 0x7f739c001950 con 0x7f73a00b4670 2024-03-25T18:30:05.383 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:30:05.379+0000 7f73967fc640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:30:05.383 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:30:05.379+0000 7f73967fc640 10 monclient(hunting): got monmap 1 from mon.b (according to old e1) 2024-03-25T18:30:05.383 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:30:05.379+0000 7f73967fc640 10 monclient(hunting): dump: 2024-03-25T18:30:05.383 INFO:teuthology.misc.health.smithi001.stderr:epoch 1 2024-03-25T18:30:05.383 INFO:teuthology.misc.health.smithi001.stderr:fsid c8ea72cf-9e99-4383-9b52-6c23d365fdfc 2024-03-25T18:30:05.383 INFO:teuthology.misc.health.smithi001.stderr:last_changed 2024-03-25T18:25:50.510519+0000 2024-03-25T18:30:05.383 INFO:teuthology.misc.health.smithi001.stderr:created 2024-03-25T18:25:50.510519+0000 2024-03-25T18:30:05.383 INFO:teuthology.misc.health.smithi001.stderr:min_mon_release 19 (squid) 2024-03-25T18:30:05.383 INFO:teuthology.misc.health.smithi001.stderr:election_strategy: 3 2024-03-25T18:30:05.383 INFO:teuthology.misc.health.smithi001.stderr:0: v1:172.21.15.1:6789/0 mon.a 2024-03-25T18:30:05.384 INFO:teuthology.misc.health.smithi001.stderr:1: v1:172.21.15.115:6789/0 mon.b 2024-03-25T18:30:05.384 INFO:teuthology.misc.health.smithi001.stderr:2: v1:172.21.15.1:6790/0 mon.c 2024-03-25T18:30:05.384 INFO:teuthology.misc.health.smithi001.stderr: 2024-03-25T18:30:05.384 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:30:05.380+0000 7f73967fc640 1 -- 172.21.15.1:0/993969202 <== mon.0 v1:172.21.15.1:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 1167486806 0 0) 0x7f73a01382d0 con 0x7f73a013c480 2024-03-25T18:30:05.384 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:30:05.380+0000 7f73967fc640 1 -- 172.21.15.1:0/993969202 >> v1:172.21.15.1:6790/0 conn(0x7f73a00b4ff0 legacy=0x7f73a01374b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:30:05.384 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:30:05.380+0000 7f73967fc640 1 -- 172.21.15.1:0/993969202 >> v1:172.21.15.115:6789/0 conn(0x7f73a00b4670 legacy=0x7f73a0145290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:30:05.384 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:30:05.380+0000 7f73967fc640 10 monclient: _finish_hunting 0 2024-03-25T18:30:05.384 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:30:05.380+0000 7f73967fc640 1 monclient: found mon.a 2024-03-25T18:30:05.384 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:30:05.380+0000 7f73967fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:30:05.384 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:30:05.380+0000 7f73967fc640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.1:6789/0 2024-03-25T18:30:05.384 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:30:05.380+0000 7f73967fc640 1 -- 172.21.15.1:0/993969202 --> v1:172.21.15.1:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f73a006e890 con 0x7f73a013c480 2024-03-25T18:30:05.384 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:30:05.380+0000 7f73967fc640 10 monclient: _finish_auth 0 2024-03-25T18:30:05.385 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:30:05.380+0000 7f73967fc640 10 monclient: _check_auth_tickets 2024-03-25T18:30:05.385 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:30:05.380+0000 7f73967fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:30:05.385 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:30:05.380+0000 7f73a7f8c640 5 monclient: authenticate success, global_id 4368 2024-03-25T18:30:05.385 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:30:05.380+0000 7f73a7f8c640 10 monclient: _renew_subs 2024-03-25T18:30:05.385 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:30:05.380+0000 7f73a7f8c640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.1:6789/0 2024-03-25T18:30:05.385 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:30:05.380+0000 7f73a7f8c640 1 -- 172.21.15.1:0/993969202 --> v1:172.21.15.1:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f73a006eb70 con 0x7f73a013c480 2024-03-25T18:30:05.385 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:30:05.380+0000 7f73a7f8c640 10 client.4368.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T18:30:05.385 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:30:05.380+0000 7f73a7f8c640 10 monclient: _renew_subs 2024-03-25T18:30:05.385 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:30:05.380+0000 7f73a7f8c640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.1:6789/0 2024-03-25T18:30:05.385 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:30:05.380+0000 7f73a7f8c640 1 -- 172.21.15.1:0/993969202 --> v1:172.21.15.1:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f73a01df2e0 con 0x7f73a013c480 2024-03-25T18:30:05.385 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:30:05.380+0000 7f73a7f8c640 1 librados: init done 2024-03-25T18:30:05.385 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:30:05.381+0000 7f73a7f8c640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T18:30:05.385 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:30:05.381+0000 7f73a7f8c640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:30:05.385 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:30:05.381+0000 7f73a7f8c640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.1:6789/0 2024-03-25T18:30:05.386 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:30:05.381+0000 7f73a7f8c640 1 -- 172.21.15.1:0/993969202 --> v1:172.21.15.1:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f73a006dbd0 con 0x7f73a013c480 2024-03-25T18:30:05.706 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:30:05.704+0000 7f73967fc640 1 -- 172.21.15.1:0/993969202 <== mon.0 v1:172.21.15.1:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f73980020e0 con 0x7f73a013c480 2024-03-25T18:30:05.706 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:30:05.704+0000 7f73967fc640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:30:05.706 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:30:05.704+0000 7f73967fc640 1 -- 172.21.15.1:0/993969202 <== mon.0 v1:172.21.15.1:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1586965025 0 0) 0x7f7398003bb0 con 0x7f73a013c480 2024-03-25T18:30:05.706 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:30:05.704+0000 7f73967fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:30:05.706 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:30:05.704+0000 7f73967fc640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-25T18:30:05.706 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:30:05.704+0000 7f73967fc640 10 monclient: dump: 2024-03-25T18:30:05.706 INFO:teuthology.misc.health.smithi001.stderr:epoch 1 2024-03-25T18:30:05.706 INFO:teuthology.misc.health.smithi001.stderr:fsid c8ea72cf-9e99-4383-9b52-6c23d365fdfc 2024-03-25T18:30:05.706 INFO:teuthology.misc.health.smithi001.stderr:last_changed 2024-03-25T18:25:50.510519+0000 2024-03-25T18:30:05.706 INFO:teuthology.misc.health.smithi001.stderr:created 2024-03-25T18:25:50.510519+0000 2024-03-25T18:30:05.706 INFO:teuthology.misc.health.smithi001.stderr:min_mon_release 19 (squid) 2024-03-25T18:30:05.707 INFO:teuthology.misc.health.smithi001.stderr:election_strategy: 3 2024-03-25T18:30:05.707 INFO:teuthology.misc.health.smithi001.stderr:0: v1:172.21.15.1:6789/0 mon.a 2024-03-25T18:30:05.707 INFO:teuthology.misc.health.smithi001.stderr:1: v1:172.21.15.115:6789/0 mon.b 2024-03-25T18:30:05.707 INFO:teuthology.misc.health.smithi001.stderr:2: v1:172.21.15.1:6790/0 mon.c 2024-03-25T18:30:05.707 INFO:teuthology.misc.health.smithi001.stderr: 2024-03-25T18:30:05.707 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:30:05.704+0000 7f7397fff640 4 set_mon_vals no callback set 2024-03-25T18:30:05.707 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:30:05.705+0000 7f73967fc640 1 -- 172.21.15.1:0/993969202 <== mon.0 v1:172.21.15.1:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 21787975 0 0) 0x7f7398003dd0 con 0x7f73a013c480 2024-03-25T18:30:05.708 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:30:05.705+0000 7f73a5d28640 10 monclient: get_auth_request con 0x7f738406e0f0 auth_method 0 2024-03-25T18:30:05.708 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:30:05.706+0000 7f73967fc640 10 client.4368.objecter ms_handle_connect 0x7f738406e0f0 2024-03-25T18:30:05.713 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:30:05.712+0000 7f73967fc640 1 -- 172.21.15.1:0/993969202 <== mon.0 v1:172.21.15.1:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 996254230 0 0) 0x7f73980047d0 con 0x7f73a013c480 2024-03-25T18:30:05.714 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:30:05.712+0000 7f73967fc640 10 client.4368.objecter ms_dispatch 0x7f73a0000b90 osd_map(10..10 src has 1..10) v4 2024-03-25T18:30:05.714 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:30:05.712+0000 7f73967fc640 3 client.4368.objecter handle_osd_map got epochs [10,10] > 0 2024-03-25T18:30:05.714 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:30:05.712+0000 7f73967fc640 3 client.4368.objecter handle_osd_map decoding full epoch 10 2024-03-25T18:30:05.714 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:30:05.712+0000 7f73967fc640 20 client.4368.objecter dump_active .. 0 homeless 2024-03-25T18:30:05.714 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:30:05.712+0000 7f73967fc640 1 -- 172.21.15.1:0/993969202 <== mon.0 v1:172.21.15.1:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f73980505a0 con 0x7f73a013c480 2024-03-25T18:30:05.714 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:30:05.712+0000 7f73967fc640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:30:05.714 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:30:05.712+0000 7f73967fc640 10 monclient: _finish_command 1 = system:0 2024-03-25T18:30:05.938 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:30:05.936+0000 7f73a7f8c640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:30:05.939 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:30:05.936+0000 7f73a7f8c640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:30:05.939 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:30:05.936+0000 7f73a7f8c640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.1:6789/0 2024-03-25T18:30:05.939 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:30:05.936+0000 7f73a7f8c640 1 -- 172.21.15.1:0/993969202 --> v1:172.21.15.1:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f73a01dfad0 con 0x7f73a013c480 2024-03-25T18:30:05.962 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:30:05.960+0000 7f73967fc640 1 -- 172.21.15.1:0/993969202 <== mon.0 v1:172.21.15.1:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 4092213348) 0x7f73a01dfad0 con 0x7f73a013c480 2024-03-25T18:30:05.963 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:30:05.960+0000 7f73967fc640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:30:05.963 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:30:05.960+0000 7f73967fc640 10 monclient: _finish_command 2 = system:0 2024-03-25T18:30:05.963 INFO:teuthology.misc.health.smithi001.stdout: 2024-03-25T18:30:05.963 INFO:teuthology.misc.health.smithi001.stdout:{"epoch":10,"fsid":"c8ea72cf-9e99-4383-9b52-6c23d365fdfc","created":"2024-03-25T18:26:38.856638+0000","modified":"2024-03-25T18:26:44.224208+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T18:26:44.224208+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"b9f34922-3a84-49be-8426-807c1eceaefc","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"42bf306f-55e3-4634-9874-8b4b73ac6644","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"292cb23a-18a9-4207-a7cc-4d5727feaec0","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"45e0e9ab-898d-4673-9590-b4117bc5e84a","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"831a5a09-73ab-4a61-9872-088e76bdbc66","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"88cd903d-3b55-4840-9396-6e3cfe005282","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"89db7355-bc9d-46f4-8ab0-e8d9e744e026","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"cac47bee-0914-4f08-896f-348838882866","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T18:30:05.965 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:30:05.963+0000 7f73a7f8c640 10 librados: watch_flush enter 2024-03-25T18:30:05.965 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:30:05.963+0000 7f73a7f8c640 10 librados: watch_flush exit 2024-03-25T18:30:05.965 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:30:05.963+0000 7f73a7f8c640 20 client.4368.objecter shutdown clearing up homeless session... 2024-03-25T18:30:05.965 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:30:05.963+0000 7f73a7f8c640 10 client.4368.objecter successfully canceled tick 2024-03-25T18:30:05.965 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:30:05.963+0000 7f73a7f8c640 1 -- 172.21.15.1:0/993969202 >> v1:172.21.15.115:6800/4237379176 conn(0x7f738406e0f0 legacy=0x7f73840705b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:30:05.965 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:30:05.963+0000 7f73a7f8c640 10 monclient: shutdown 2024-03-25T18:30:05.965 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:30:05.963+0000 7f73a7f8c640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:30:05.966 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:30:05.963+0000 7f73a7f8c640 1 -- 172.21.15.1:0/993969202 >> v1:172.21.15.1:6789/0 conn(0x7f73a013c480 legacy=0x7f73a0137bc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:30:05.966 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:30:05.963+0000 7f73a7f8c640 1 -- 172.21.15.1:0/993969202 shutdown_connections 2024-03-25T18:30:05.966 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:30:05.963+0000 7f73a7f8c640 1 -- 172.21.15.1:0/993969202 >> 172.21.15.1:0/993969202 conn(0x7f73a00617a0 msgr2=0x7f73a00b2a00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:30:05.966 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:30:05.963+0000 7f73a7f8c640 1 -- 172.21.15.1:0/993969202 shutdown_connections 2024-03-25T18:30:05.966 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:30:05.963+0000 7f73a7f8c640 1 -- 172.21.15.1:0/993969202 wait complete. 2024-03-25T18:30:05.966 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:30:05.963+0000 7f73a7f8c640 1 librados: shutdown 2024-03-25T18:30:05.977 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T18:30:12.785 DEBUG:teuthology.orchestra.run.smithi001:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T18:30:12.898 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:30:12.896+0000 7f51ca554640 10 monclient: get_monmap_and_config 2024-03-25T18:30:12.898 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:30:12.896+0000 7f51ca554640 10 monclient: build_initial_monmap 2024-03-25T18:30:12.898 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:30:12.896+0000 7f51ca554640 10 monclient: monmap: 2024-03-25T18:30:12.898 INFO:teuthology.misc.health.smithi001.stderr:epoch 0 2024-03-25T18:30:12.898 INFO:teuthology.misc.health.smithi001.stderr:fsid c8ea72cf-9e99-4383-9b52-6c23d365fdfc 2024-03-25T18:30:12.898 INFO:teuthology.misc.health.smithi001.stderr:last_changed 2024-03-25T18:30:12.898220+0000 2024-03-25T18:30:12.899 INFO:teuthology.misc.health.smithi001.stderr:created 2024-03-25T18:30:12.898220+0000 2024-03-25T18:30:12.899 INFO:teuthology.misc.health.smithi001.stderr:min_mon_release 0 (unknown) 2024-03-25T18:30:12.899 INFO:teuthology.misc.health.smithi001.stderr:election_strategy: 3 2024-03-25T18:30:12.899 INFO:teuthology.misc.health.smithi001.stderr:0: v1:172.21.15.1:6789/0 mon.noname-a 2024-03-25T18:30:12.899 INFO:teuthology.misc.health.smithi001.stderr:1: v1:172.21.15.115:6789/0 mon.noname-c 2024-03-25T18:30:12.899 INFO:teuthology.misc.health.smithi001.stderr:2: v1:172.21.15.1:6790/0 mon.noname-b 2024-03-25T18:30:12.899 INFO:teuthology.misc.health.smithi001.stderr: 2024-03-25T18:30:12.900 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:30:12.897+0000 7f51ca554640 1 Processor -- start 2024-03-25T18:30:12.900 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:30:12.897+0000 7f51ca554640 1 -- start start 2024-03-25T18:30:12.900 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:30:12.897+0000 7f51ca554640 10 monclient: init 2024-03-25T18:30:12.900 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:30:12.898+0000 7f51ca554640 10 monclient: _reopen_session rank -1 2024-03-25T18:30:12.900 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:30:12.898+0000 7f51ca554640 10 monclient: _add_conns ranks=[1,2,0] 2024-03-25T18:30:12.900 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:30:12.898+0000 7f51ca554640 10 monclient(hunting): picked mon.noname-c con 0x7f51c40b4e70 addr v1:172.21.15.115:6789/0 2024-03-25T18:30:12.900 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:30:12.898+0000 7f51ca554640 10 monclient(hunting): picked mon.noname-b con 0x7f51c40b44b0 addr v1:172.21.15.1:6790/0 2024-03-25T18:30:12.901 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:30:12.898+0000 7f51ca554640 10 monclient(hunting): picked mon.noname-a con 0x7f51c40b1b00 addr v1:172.21.15.1:6789/0 2024-03-25T18:30:12.901 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:30:12.898+0000 7f51ca554640 1 -- --> v1:172.21.15.1:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f51c40a96f0 con 0x7f51c40b1b00 2024-03-25T18:30:12.901 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:30:12.898+0000 7f51ca554640 1 -- --> v1:172.21.15.115:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f51c40ab0e0 con 0x7f51c40b4e70 2024-03-25T18:30:12.901 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:30:12.898+0000 7f51ca554640 1 -- --> v1:172.21.15.1:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f51c4145f70 con 0x7f51c40b44b0 2024-03-25T18:30:12.901 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:30:12.898+0000 7f51ca554640 10 monclient(hunting): _renew_subs 2024-03-25T18:30:12.901 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:30:12.898+0000 7f51ca554640 10 monclient(hunting): authenticate will time out at 7693.369016s 2024-03-25T18:30:12.907 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:30:12.905+0000 7f51c3fff640 1 --1- >> v1:172.21.15.115:6789/0 conn(0x7f51c40b4e70 0x7f51c413f940 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.115:6789/0 says I am v1:172.21.15.1:40830/0 (socket says 172.21.15.1:40830) 2024-03-25T18:30:12.907 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:30:12.905+0000 7f51c3fff640 1 -- 172.21.15.1:0/3437571483 learned_addr learned my addr 172.21.15.1:0/3437571483 (peer_addr_for_me v1:172.21.15.1:0/0) 2024-03-25T18:30:12.922 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:30:12.920+0000 7f51c2ffd640 1 -- 172.21.15.1:0/3437571483 <== mon.0 v1:172.21.15.1:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2787326920 0 0) 0x7f51c40a96f0 con 0x7f51c40b1b00 2024-03-25T18:30:12.922 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:30:12.920+0000 7f51c2ffd640 10 monclient(hunting): _negotiate 2024-03-25T18:30:12.922 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:30:12.920+0000 7f51c2ffd640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:30:12.922 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:30:12.920+0000 7f51c2ffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:30:12.922 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:30:12.920+0000 7f51c2ffd640 10 monclient(hunting): my global_id is 4371 2024-03-25T18:30:12.922 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:30:12.920+0000 7f51c2ffd640 1 -- 172.21.15.1:0/3437571483 --> v1:172.21.15.1:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f51ac003520 con 0x7f51c40b1b00 2024-03-25T18:30:12.923 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:30:12.921+0000 7f51c2ffd640 1 -- 172.21.15.1:0/3437571483 <== mon.1 v1:172.21.15.115:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1183219881 0 0) 0x7f51c40ab0e0 con 0x7f51c40b4e70 2024-03-25T18:30:12.923 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:30:12.921+0000 7f51c2ffd640 10 monclient(hunting): _negotiate 2024-03-25T18:30:12.923 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:30:12.921+0000 7f51c2ffd640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:30:12.923 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:30:12.921+0000 7f51c2ffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:30:12.923 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:30:12.921+0000 7f51c2ffd640 10 monclient(hunting): my global_id is 4372 2024-03-25T18:30:12.923 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:30:12.921+0000 7f51c2ffd640 1 -- 172.21.15.1:0/3437571483 --> v1:172.21.15.115:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f51c40a96f0 con 0x7f51c40b4e70 2024-03-25T18:30:12.928 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:30:12.926+0000 7f51c2ffd640 1 -- 172.21.15.1:0/3437571483 <== mon.2 v1:172.21.15.1:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2840070122 0 0) 0x7f51c4145f70 con 0x7f51c40b44b0 2024-03-25T18:30:12.928 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:30:12.926+0000 7f51c2ffd640 10 monclient(hunting): _negotiate 2024-03-25T18:30:12.928 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:30:12.926+0000 7f51c2ffd640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:30:12.928 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:30:12.926+0000 7f51c2ffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:30:12.928 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:30:12.926+0000 7f51c2ffd640 10 monclient(hunting): my global_id is 4379 2024-03-25T18:30:12.928 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:30:12.926+0000 7f51c2ffd640 1 -- 172.21.15.1:0/3437571483 --> v1:172.21.15.1:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f51c40ab0e0 con 0x7f51c40b44b0 2024-03-25T18:30:12.929 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:30:12.927+0000 7f51c2ffd640 1 -- 172.21.15.1:0/3437571483 <== mon.1 v1:172.21.15.115:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3740929263 0 0) 0x7f51c40a96f0 con 0x7f51c40b4e70 2024-03-25T18:30:12.929 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:30:12.927+0000 7f51c2ffd640 1 -- 172.21.15.1:0/3437571483 >> v1:172.21.15.1:6790/0 conn(0x7f51c40b44b0 legacy=0x7f51c4145860 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:30:12.929 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:30:12.927+0000 7f51c2ffd640 1 -- 172.21.15.1:0/3437571483 >> v1:172.21.15.1:6789/0 conn(0x7f51c40b1b00 legacy=0x7f51c414dc00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:30:12.929 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:30:12.927+0000 7f51c2ffd640 10 monclient: _finish_hunting 0 2024-03-25T18:30:12.930 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:30:12.927+0000 7f51c2ffd640 1 monclient: found mon.noname-c 2024-03-25T18:30:12.930 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:30:12.927+0000 7f51c2ffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:30:12.930 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:30:12.927+0000 7f51c2ffd640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.115:6789/0 2024-03-25T18:30:12.930 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:30:12.927+0000 7f51c2ffd640 1 -- 172.21.15.1:0/3437571483 --> v1:172.21.15.115:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f51c40a9b80 con 0x7f51c40b4e70 2024-03-25T18:30:12.930 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:30:12.927+0000 7f51c2ffd640 10 monclient: _finish_auth 0 2024-03-25T18:30:12.930 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:30:12.927+0000 7f51c2ffd640 10 monclient: _check_auth_tickets 2024-03-25T18:30:12.930 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:30:12.927+0000 7f51c2ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:30:12.930 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:30:12.927+0000 7f51ca554640 5 monclient: authenticate success, global_id 4372 2024-03-25T18:30:12.930 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:30:12.927+0000 7f51ca554640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T18:30:12.932 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:30:12.930+0000 7f51c2ffd640 1 -- 172.21.15.1:0/3437571483 <== mon.1 v1:172.21.15.115:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1586965025 0 0) 0x7f51b4001630 con 0x7f51c40b4e70 2024-03-25T18:30:12.932 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:30:12.930+0000 7f51c2ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:30:12.932 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:30:12.930+0000 7f51c2ffd640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-03-25T18:30:12.932 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:30:12.930+0000 7f51c2ffd640 10 monclient: dump: 2024-03-25T18:30:12.932 INFO:teuthology.misc.health.smithi001.stderr:epoch 1 2024-03-25T18:30:12.932 INFO:teuthology.misc.health.smithi001.stderr:fsid c8ea72cf-9e99-4383-9b52-6c23d365fdfc 2024-03-25T18:30:12.932 INFO:teuthology.misc.health.smithi001.stderr:last_changed 2024-03-25T18:25:50.510519+0000 2024-03-25T18:30:12.932 INFO:teuthology.misc.health.smithi001.stderr:created 2024-03-25T18:25:50.510519+0000 2024-03-25T18:30:12.933 INFO:teuthology.misc.health.smithi001.stderr:min_mon_release 19 (squid) 2024-03-25T18:30:12.933 INFO:teuthology.misc.health.smithi001.stderr:election_strategy: 3 2024-03-25T18:30:12.933 INFO:teuthology.misc.health.smithi001.stderr:0: v1:172.21.15.1:6789/0 mon.a 2024-03-25T18:30:12.933 INFO:teuthology.misc.health.smithi001.stderr:1: v1:172.21.15.115:6789/0 mon.b 2024-03-25T18:30:12.933 INFO:teuthology.misc.health.smithi001.stderr:2: v1:172.21.15.1:6790/0 mon.c 2024-03-25T18:30:12.933 INFO:teuthology.misc.health.smithi001.stderr: 2024-03-25T18:30:12.933 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:30:12.930+0000 7f51ca554640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T18:30:12.933 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:30:12.930+0000 7f51c2ffd640 1 -- 172.21.15.1:0/3437571483 <== mon.1 v1:172.21.15.115:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f51b4001880 con 0x7f51c40b4e70 2024-03-25T18:30:12.933 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:30:12.930+0000 7f51c2ffd640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:30:12.933 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:30:12.930+0000 7f51c2ffd640 1 -- 172.21.15.1:0/3437571483 <== mon.1 v1:172.21.15.115:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1586965025 0 0) 0x7f51b4001c10 con 0x7f51c40b4e70 2024-03-25T18:30:12.933 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:30:12.930+0000 7f51ca554640 10 monclient: get_monmap_and_config success 2024-03-25T18:30:12.933 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:30:12.930+0000 7f51ca554640 4 set_mon_vals no callback set 2024-03-25T18:30:12.933 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:30:12.930+0000 7f51c2ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:30:12.933 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:30:12.930+0000 7f51c2ffd640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-25T18:30:12.933 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:30:12.930+0000 7f51c2ffd640 10 monclient: dump: 2024-03-25T18:30:12.934 INFO:teuthology.misc.health.smithi001.stderr:epoch 1 2024-03-25T18:30:12.934 INFO:teuthology.misc.health.smithi001.stderr:fsid c8ea72cf-9e99-4383-9b52-6c23d365fdfc 2024-03-25T18:30:12.934 INFO:teuthology.misc.health.smithi001.stderr:last_changed 2024-03-25T18:25:50.510519+0000 2024-03-25T18:30:12.934 INFO:teuthology.misc.health.smithi001.stderr:created 2024-03-25T18:25:50.510519+0000 2024-03-25T18:30:12.934 INFO:teuthology.misc.health.smithi001.stderr:min_mon_release 19 (squid) 2024-03-25T18:30:12.934 INFO:teuthology.misc.health.smithi001.stderr:election_strategy: 3 2024-03-25T18:30:12.934 INFO:teuthology.misc.health.smithi001.stderr:0: v1:172.21.15.1:6789/0 mon.a 2024-03-25T18:30:12.934 INFO:teuthology.misc.health.smithi001.stderr:1: v1:172.21.15.115:6789/0 mon.b 2024-03-25T18:30:12.934 INFO:teuthology.misc.health.smithi001.stderr:2: v1:172.21.15.1:6790/0 mon.c 2024-03-25T18:30:12.934 INFO:teuthology.misc.health.smithi001.stderr: 2024-03-25T18:30:12.936 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:30:12.931+0000 7f51ca554640 10 monclient: shutdown 2024-03-25T18:30:12.936 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:30:12.931+0000 7f51ca554640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:30:12.936 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:30:12.931+0000 7f51ca554640 1 -- 172.21.15.1:0/3437571483 >> v1:172.21.15.115:6789/0 conn(0x7f51c40b4e70 legacy=0x7f51c413f940 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:30:12.936 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:30:12.931+0000 7f51ca554640 1 -- 172.21.15.1:0/3437571483 shutdown_connections 2024-03-25T18:30:12.936 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:30:12.931+0000 7f51ca554640 1 -- 172.21.15.1:0/3437571483 >> 172.21.15.1:0/3437571483 conn(0x7f51c40617a0 msgr2=0x7f51c4061bb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:30:12.936 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:30:12.931+0000 7f51ca554640 1 -- 172.21.15.1:0/3437571483 shutdown_connections 2024-03-25T18:30:12.937 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:30:12.931+0000 7f51ca554640 1 -- 172.21.15.1:0/3437571483 wait complete. 2024-03-25T18:30:12.937 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:30:12.932+0000 7f51ca554640 10 monclient: build_initial_monmap 2024-03-25T18:30:12.937 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:30:12.932+0000 7f51ca554640 10 monclient: monmap: 2024-03-25T18:30:12.937 INFO:teuthology.misc.health.smithi001.stderr:epoch 0 2024-03-25T18:30:12.937 INFO:teuthology.misc.health.smithi001.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T18:30:12.937 INFO:teuthology.misc.health.smithi001.stderr:last_changed 0.000000 2024-03-25T18:30:12.937 INFO:teuthology.misc.health.smithi001.stderr:created 0.000000 2024-03-25T18:30:12.937 INFO:teuthology.misc.health.smithi001.stderr:min_mon_release 0 (unknown) 2024-03-25T18:30:12.937 INFO:teuthology.misc.health.smithi001.stderr:election_strategy: 1 2024-03-25T18:30:12.937 INFO:teuthology.misc.health.smithi001.stderr:0: v1:172.21.15.1:6789/0 mon.noname-a 2024-03-25T18:30:12.937 INFO:teuthology.misc.health.smithi001.stderr:1: v1:172.21.15.115:6789/0 mon.noname-b 2024-03-25T18:30:12.937 INFO:teuthology.misc.health.smithi001.stderr:2: v1:172.21.15.1:6790/0 mon.noname-c 2024-03-25T18:30:12.937 INFO:teuthology.misc.health.smithi001.stderr: 2024-03-25T18:30:12.937 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:30:12.933+0000 7f51ca554640 1 librados: starting msgr at 2024-03-25T18:30:12.937 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:30:12.933+0000 7f51ca554640 1 librados: starting objecter 2024-03-25T18:30:12.938 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:30:12.933+0000 7f51ca554640 1 Processor -- start 2024-03-25T18:30:12.938 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:30:12.933+0000 7f51ca554640 1 -- start start 2024-03-25T18:30:12.938 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:30:12.933+0000 7f51ca554640 1 librados: setting wanted keys 2024-03-25T18:30:12.938 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:30:12.933+0000 7f51ca554640 1 librados: calling monclient init 2024-03-25T18:30:12.938 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:30:12.933+0000 7f51ca554640 10 monclient: init 2024-03-25T18:30:12.938 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:30:12.934+0000 7f51ca554640 10 monclient: _reopen_session rank -1 2024-03-25T18:30:12.938 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:30:12.934+0000 7f51ca554640 10 monclient: _add_conns ranks=[2,1,0] 2024-03-25T18:30:12.938 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:30:12.934+0000 7f51ca554640 10 monclient(hunting): picked mon.noname-c con 0x7f51c40b1b00 addr v1:172.21.15.1:6790/0 2024-03-25T18:30:12.938 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:30:12.934+0000 7f51ca554640 10 monclient(hunting): picked mon.noname-b con 0x7f51c40b44b0 addr v1:172.21.15.115:6789/0 2024-03-25T18:30:12.938 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:30:12.934+0000 7f51ca554640 10 monclient(hunting): picked mon.noname-a con 0x7f51c40b4e70 addr v1:172.21.15.1:6789/0 2024-03-25T18:30:12.938 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:30:12.934+0000 7f51ca554640 1 -- --> v1:172.21.15.1:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f51c413c5c0 con 0x7f51c40b4e70 2024-03-25T18:30:12.938 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:30:12.934+0000 7f51ca554640 1 -- --> v1:172.21.15.115:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f51c413c8a0 con 0x7f51c40b44b0 2024-03-25T18:30:12.938 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:30:12.934+0000 7f51ca554640 1 -- --> v1:172.21.15.1:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f51c413b520 con 0x7f51c40b1b00 2024-03-25T18:30:12.938 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:30:12.934+0000 7f51ca554640 10 monclient(hunting): _renew_subs 2024-03-25T18:30:12.939 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:30:12.934+0000 7f51ca554640 10 monclient(hunting): authenticate will time out at 7693.404847s 2024-03-25T18:30:12.948 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:30:12.946+0000 7f51c37fe640 1 --1- >> v1:172.21.15.115:6789/0 conn(0x7f51c40b44b0 0x7f51c413dc90 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.115:6789/0 says I am v1:172.21.15.1:40860/0 (socket says 172.21.15.1:40860) 2024-03-25T18:30:12.948 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:30:12.946+0000 7f51c37fe640 1 -- 172.21.15.1:0/3632331008 learned_addr learned my addr 172.21.15.1:0/3632331008 (peer_addr_for_me v1:172.21.15.1:0/0) 2024-03-25T18:30:12.953 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:30:12.950+0000 7f51c0ff9640 10 client.?.objecter ms_handle_connect 0x7f51c40b44b0 2024-03-25T18:30:12.953 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:30:12.950+0000 7f51c0ff9640 10 client.?.objecter resend_mon_ops 2024-03-25T18:30:12.953 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:30:12.951+0000 7f51c0ff9640 10 client.?.objecter ms_handle_connect 0x7f51c40b4e70 2024-03-25T18:30:12.953 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:30:12.951+0000 7f51c0ff9640 10 client.?.objecter resend_mon_ops 2024-03-25T18:30:12.957 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:30:12.955+0000 7f51c0ff9640 1 -- 172.21.15.1:0/3632331008 <== mon.0 v1:172.21.15.1:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2507840994 0 0) 0x7f51c413c5c0 con 0x7f51c40b4e70 2024-03-25T18:30:12.957 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:30:12.955+0000 7f51c0ff9640 10 monclient(hunting): _negotiate 2024-03-25T18:30:12.957 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:30:12.955+0000 7f51c0ff9640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:30:12.957 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:30:12.955+0000 7f51c0ff9640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:30:12.957 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:30:12.955+0000 7f51c0ff9640 10 monclient(hunting): my global_id is 4374 2024-03-25T18:30:12.957 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:30:12.955+0000 7f51c0ff9640 1 -- 172.21.15.1:0/3632331008 --> v1:172.21.15.1:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f51b0004eb0 con 0x7f51c40b4e70 2024-03-25T18:30:12.962 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:30:12.960+0000 7f51c0ff9640 1 -- 172.21.15.1:0/3632331008 <== mon.1 v1:172.21.15.115:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3670239467 0 0) 0x7f51c413c8a0 con 0x7f51c40b44b0 2024-03-25T18:30:12.962 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:30:12.960+0000 7f51c0ff9640 10 monclient(hunting): _negotiate 2024-03-25T18:30:12.962 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:30:12.960+0000 7f51c0ff9640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:30:12.963 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:30:12.960+0000 7f51c0ff9640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:30:12.963 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:30:12.960+0000 7f51c0ff9640 10 monclient(hunting): my global_id is 4375 2024-03-25T18:30:12.963 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:30:12.960+0000 7f51c0ff9640 1 -- 172.21.15.1:0/3632331008 --> v1:172.21.15.115:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f51c413c5c0 con 0x7f51c40b44b0 2024-03-25T18:30:12.964 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:30:12.963+0000 7f51c0ff9640 10 client.?.objecter ms_handle_connect 0x7f51c40b1b00 2024-03-25T18:30:12.965 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:30:12.963+0000 7f51c0ff9640 10 client.?.objecter resend_mon_ops 2024-03-25T18:30:12.973 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:30:12.971+0000 7f51c0ff9640 1 -- 172.21.15.1:0/3632331008 <== mon.0 v1:172.21.15.1:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3754450104 0 0) 0x7f51b0004eb0 con 0x7f51c40b4e70 2024-03-25T18:30:12.973 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:30:12.971+0000 7f51c0ff9640 1 -- 172.21.15.1:0/3632331008 --> v1:172.21.15.1:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f51c413c8a0 con 0x7f51c40b4e70 2024-03-25T18:30:12.973 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:30:12.971+0000 7f51c0ff9640 1 -- 172.21.15.1:0/3632331008 <== mon.0 v1:172.21.15.1:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1586965025 0 0) 0x7f51bc001cd0 con 0x7f51c40b4e70 2024-03-25T18:30:12.973 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:30:12.971+0000 7f51c0ff9640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:30:12.973 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:30:12.971+0000 7f51c0ff9640 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2024-03-25T18:30:12.973 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:30:12.971+0000 7f51c0ff9640 10 monclient(hunting): dump: 2024-03-25T18:30:12.973 INFO:teuthology.misc.health.smithi001.stderr:epoch 1 2024-03-25T18:30:12.973 INFO:teuthology.misc.health.smithi001.stderr:fsid c8ea72cf-9e99-4383-9b52-6c23d365fdfc 2024-03-25T18:30:12.974 INFO:teuthology.misc.health.smithi001.stderr:last_changed 2024-03-25T18:25:50.510519+0000 2024-03-25T18:30:12.974 INFO:teuthology.misc.health.smithi001.stderr:created 2024-03-25T18:25:50.510519+0000 2024-03-25T18:30:12.974 INFO:teuthology.misc.health.smithi001.stderr:min_mon_release 19 (squid) 2024-03-25T18:30:12.974 INFO:teuthology.misc.health.smithi001.stderr:election_strategy: 3 2024-03-25T18:30:12.974 INFO:teuthology.misc.health.smithi001.stderr:0: v1:172.21.15.1:6789/0 mon.a 2024-03-25T18:30:12.974 INFO:teuthology.misc.health.smithi001.stderr:1: v1:172.21.15.115:6789/0 mon.b 2024-03-25T18:30:12.974 INFO:teuthology.misc.health.smithi001.stderr:2: v1:172.21.15.1:6790/0 mon.c 2024-03-25T18:30:12.974 INFO:teuthology.misc.health.smithi001.stderr: 2024-03-25T18:30:12.974 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:30:12.971+0000 7f51c0ff9640 10 monclient(hunting): _finish_auth 0 2024-03-25T18:30:12.976 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:30:12.974+0000 7f51c0ff9640 1 -- 172.21.15.1:0/3632331008 <== mon.1 v1:172.21.15.115:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 549281080 0 0) 0x7f51c413c5c0 con 0x7f51c40b44b0 2024-03-25T18:30:12.976 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:30:12.974+0000 7f51c0ff9640 1 -- 172.21.15.1:0/3632331008 --> v1:172.21.15.115:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f51b0004eb0 con 0x7f51c40b44b0 2024-03-25T18:30:12.976 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:30:12.974+0000 7f51c0ff9640 1 -- 172.21.15.1:0/3632331008 <== mon.1 v1:172.21.15.115:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1586965025 0 0) 0x7f51b80019b0 con 0x7f51c40b44b0 2024-03-25T18:30:12.976 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:30:12.974+0000 7f51c0ff9640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:30:12.976 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:30:12.974+0000 7f51c0ff9640 10 monclient(hunting): got monmap 1 from mon.b (according to old e1) 2024-03-25T18:30:12.976 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:30:12.974+0000 7f51c0ff9640 10 monclient(hunting): dump: 2024-03-25T18:30:12.976 INFO:teuthology.misc.health.smithi001.stderr:epoch 1 2024-03-25T18:30:12.976 INFO:teuthology.misc.health.smithi001.stderr:fsid c8ea72cf-9e99-4383-9b52-6c23d365fdfc 2024-03-25T18:30:12.977 INFO:teuthology.misc.health.smithi001.stderr:last_changed 2024-03-25T18:25:50.510519+0000 2024-03-25T18:30:12.977 INFO:teuthology.misc.health.smithi001.stderr:created 2024-03-25T18:25:50.510519+0000 2024-03-25T18:30:12.977 INFO:teuthology.misc.health.smithi001.stderr:min_mon_release 19 (squid) 2024-03-25T18:30:12.977 INFO:teuthology.misc.health.smithi001.stderr:election_strategy: 3 2024-03-25T18:30:12.977 INFO:teuthology.misc.health.smithi001.stderr:0: v1:172.21.15.1:6789/0 mon.a 2024-03-25T18:30:12.977 INFO:teuthology.misc.health.smithi001.stderr:1: v1:172.21.15.115:6789/0 mon.b 2024-03-25T18:30:12.977 INFO:teuthology.misc.health.smithi001.stderr:2: v1:172.21.15.1:6790/0 mon.c 2024-03-25T18:30:12.977 INFO:teuthology.misc.health.smithi001.stderr: 2024-03-25T18:30:12.983 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:30:12.981+0000 7f51c0ff9640 1 -- 172.21.15.1:0/3632331008 <== mon.2 v1:172.21.15.1:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1262565409 0 0) 0x7f51c413b520 con 0x7f51c40b1b00 2024-03-25T18:30:12.983 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:30:12.981+0000 7f51c0ff9640 10 monclient(hunting): _negotiate 2024-03-25T18:30:12.983 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:30:12.981+0000 7f51c0ff9640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:30:12.983 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:30:12.981+0000 7f51c0ff9640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:30:12.983 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:30:12.981+0000 7f51c0ff9640 10 monclient(hunting): my global_id is 4382 2024-03-25T18:30:12.983 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:30:12.981+0000 7f51c0ff9640 1 -- 172.21.15.1:0/3632331008 --> v1:172.21.15.1:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f51c413c5c0 con 0x7f51c40b1b00 2024-03-25T18:30:12.984 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:30:12.981+0000 7f51c0ff9640 1 -- 172.21.15.1:0/3632331008 <== mon.1 v1:172.21.15.115:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 732174943 0 0) 0x7f51b0004eb0 con 0x7f51c40b44b0 2024-03-25T18:30:12.984 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:30:12.981+0000 7f51c0ff9640 1 -- 172.21.15.1:0/3632331008 >> v1:172.21.15.1:6790/0 conn(0x7f51c40b1b00 legacy=0x7f51c413d580 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:30:12.984 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:30:12.981+0000 7f51c0ff9640 1 -- 172.21.15.1:0/3632331008 >> v1:172.21.15.1:6789/0 conn(0x7f51c40b4e70 legacy=0x7f51c41e1f40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:30:12.984 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:30:12.981+0000 7f51c0ff9640 10 monclient: _finish_hunting 0 2024-03-25T18:30:12.984 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:30:12.981+0000 7f51c0ff9640 1 monclient: found mon.b 2024-03-25T18:30:12.984 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:30:12.981+0000 7f51c0ff9640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:30:12.984 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:30:12.981+0000 7f51c0ff9640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.115:6789/0 2024-03-25T18:30:12.984 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:30:12.981+0000 7f51c0ff9640 1 -- 172.21.15.1:0/3632331008 --> v1:172.21.15.115:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f51c413a1d0 con 0x7f51c40b44b0 2024-03-25T18:30:12.984 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:30:12.981+0000 7f51c0ff9640 10 monclient: _finish_auth 0 2024-03-25T18:30:12.984 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:30:12.981+0000 7f51c0ff9640 10 monclient: _check_auth_tickets 2024-03-25T18:30:12.984 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:30:12.981+0000 7f51c0ff9640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:30:12.985 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:30:12.981+0000 7f51ca554640 5 monclient: authenticate success, global_id 4375 2024-03-25T18:30:12.985 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:30:12.981+0000 7f51ca554640 10 monclient: _renew_subs 2024-03-25T18:30:12.985 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:30:12.981+0000 7f51ca554640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.115:6789/0 2024-03-25T18:30:12.985 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:30:12.981+0000 7f51ca554640 1 -- 172.21.15.1:0/3632331008 --> v1:172.21.15.115:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f51c413ccc0 con 0x7f51c40b44b0 2024-03-25T18:30:12.985 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:30:12.981+0000 7f51ca554640 10 client.4375.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T18:30:12.985 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:30:12.981+0000 7f51ca554640 10 monclient: _renew_subs 2024-03-25T18:30:12.986 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:30:12.981+0000 7f51ca554640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.115:6789/0 2024-03-25T18:30:12.986 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:30:12.981+0000 7f51ca554640 1 -- 172.21.15.1:0/3632331008 --> v1:172.21.15.115:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f51c413afb0 con 0x7f51c40b44b0 2024-03-25T18:30:12.986 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:30:12.981+0000 7f51ca554640 1 librados: init done 2024-03-25T18:30:12.986 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:30:12.982+0000 7f51ca554640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T18:30:12.986 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:30:12.982+0000 7f51ca554640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:30:12.986 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:30:12.982+0000 7f51ca554640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.115:6789/0 2024-03-25T18:30:12.987 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:30:12.982+0000 7f51ca554640 1 -- 172.21.15.1:0/3632331008 --> v1:172.21.15.115:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f51c413b6b0 con 0x7f51c40b44b0 2024-03-25T18:30:13.271 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:30:13.269+0000 7f51c0ff9640 1 -- 172.21.15.1:0/3632331008 <== mon.1 v1:172.21.15.115:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f51b8003610 con 0x7f51c40b44b0 2024-03-25T18:30:13.271 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:30:13.269+0000 7f51c0ff9640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:30:13.272 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:30:13.269+0000 7f51c0ff9640 1 -- 172.21.15.1:0/3632331008 <== mon.1 v1:172.21.15.115:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1586965025 0 0) 0x7f51b8003a20 con 0x7f51c40b44b0 2024-03-25T18:30:13.272 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:30:13.269+0000 7f51c0ff9640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:30:13.272 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:30:13.269+0000 7f51c0ff9640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-25T18:30:13.272 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:30:13.269+0000 7f51c0ff9640 10 monclient: dump: 2024-03-25T18:30:13.272 INFO:teuthology.misc.health.smithi001.stderr:epoch 1 2024-03-25T18:30:13.273 INFO:teuthology.misc.health.smithi001.stderr:fsid c8ea72cf-9e99-4383-9b52-6c23d365fdfc 2024-03-25T18:30:13.273 INFO:teuthology.misc.health.smithi001.stderr:last_changed 2024-03-25T18:25:50.510519+0000 2024-03-25T18:30:13.273 INFO:teuthology.misc.health.smithi001.stderr:created 2024-03-25T18:25:50.510519+0000 2024-03-25T18:30:13.273 INFO:teuthology.misc.health.smithi001.stderr:min_mon_release 19 (squid) 2024-03-25T18:30:13.273 INFO:teuthology.misc.health.smithi001.stderr:election_strategy: 3 2024-03-25T18:30:13.273 INFO:teuthology.misc.health.smithi001.stderr:0: v1:172.21.15.1:6789/0 mon.a 2024-03-25T18:30:13.274 INFO:teuthology.misc.health.smithi001.stderr:1: v1:172.21.15.115:6789/0 mon.b 2024-03-25T18:30:13.274 INFO:teuthology.misc.health.smithi001.stderr:2: v1:172.21.15.1:6790/0 mon.c 2024-03-25T18:30:13.274 INFO:teuthology.misc.health.smithi001.stderr: 2024-03-25T18:30:13.274 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:30:13.269+0000 7f51c27fc640 4 set_mon_vals no callback set 2024-03-25T18:30:13.274 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:30:13.270+0000 7f51c0ff9640 1 -- 172.21.15.1:0/3632331008 <== mon.1 v1:172.21.15.115:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 21787975 0 0) 0x7f51b8003c40 con 0x7f51c40b44b0 2024-03-25T18:30:13.275 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:30:13.270+0000 7f51c3fff640 10 monclient: get_auth_request con 0x7f51b006df50 auth_method 0 2024-03-25T18:30:13.275 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:30:13.270+0000 7f51c0ff9640 10 client.4375.objecter ms_handle_connect 0x7f51b006df50 2024-03-25T18:30:13.277 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:30:13.275+0000 7f51c0ff9640 1 -- 172.21.15.1:0/3632331008 <== mon.1 v1:172.21.15.115:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 996254230 0 0) 0x7f51b8054800 con 0x7f51c40b44b0 2024-03-25T18:30:13.277 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:30:13.275+0000 7f51c0ff9640 10 client.4375.objecter ms_dispatch 0x7f51c4000b90 osd_map(10..10 src has 1..10) v4 2024-03-25T18:30:13.277 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:30:13.275+0000 7f51c0ff9640 3 client.4375.objecter handle_osd_map got epochs [10,10] > 0 2024-03-25T18:30:13.277 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:30:13.275+0000 7f51c0ff9640 3 client.4375.objecter handle_osd_map decoding full epoch 10 2024-03-25T18:30:13.277 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:30:13.275+0000 7f51c0ff9640 20 client.4375.objecter dump_active .. 0 homeless 2024-03-25T18:30:13.279 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:30:13.277+0000 7f51c0ff9640 1 -- 172.21.15.1:0/3632331008 <== mon.1 v1:172.21.15.115:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f51b8050310 con 0x7f51c40b44b0 2024-03-25T18:30:13.279 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:30:13.277+0000 7f51c0ff9640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:30:13.279 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:30:13.277+0000 7f51c0ff9640 10 monclient: _finish_command 1 = system:0 2024-03-25T18:30:13.535 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:30:13.533+0000 7f51ca554640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:30:13.535 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:30:13.533+0000 7f51ca554640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:30:13.535 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:30:13.533+0000 7f51ca554640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.115:6789/0 2024-03-25T18:30:13.535 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:30:13.533+0000 7f51ca554640 1 -- 172.21.15.1:0/3632331008 --> v1:172.21.15.115:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f51c404f730 con 0x7f51c40b44b0 2024-03-25T18:30:13.558 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:30:13.556+0000 7f51c0ff9640 1 -- 172.21.15.1:0/3632331008 <== mon.1 v1:172.21.15.115:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 4092213348) 0x7f51b8052240 con 0x7f51c40b44b0 2024-03-25T18:30:13.558 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:30:13.556+0000 7f51c0ff9640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:30:13.558 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:30:13.556+0000 7f51c0ff9640 10 monclient: _finish_command 2 = system:0 2024-03-25T18:30:13.559 INFO:teuthology.misc.health.smithi001.stdout: 2024-03-25T18:30:13.559 INFO:teuthology.misc.health.smithi001.stdout:{"epoch":10,"fsid":"c8ea72cf-9e99-4383-9b52-6c23d365fdfc","created":"2024-03-25T18:26:38.856638+0000","modified":"2024-03-25T18:26:44.224208+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T18:26:44.224208+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"b9f34922-3a84-49be-8426-807c1eceaefc","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"42bf306f-55e3-4634-9874-8b4b73ac6644","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"292cb23a-18a9-4207-a7cc-4d5727feaec0","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"45e0e9ab-898d-4673-9590-b4117bc5e84a","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"831a5a09-73ab-4a61-9872-088e76bdbc66","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"88cd903d-3b55-4840-9396-6e3cfe005282","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"89db7355-bc9d-46f4-8ab0-e8d9e744e026","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"cac47bee-0914-4f08-896f-348838882866","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T18:30:13.560 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:30:13.559+0000 7f51ca554640 10 librados: watch_flush enter 2024-03-25T18:30:13.561 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:30:13.559+0000 7f51ca554640 10 librados: watch_flush exit 2024-03-25T18:30:13.561 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:30:13.559+0000 7f51ca554640 20 client.4375.objecter shutdown clearing up homeless session... 2024-03-25T18:30:13.562 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:30:13.559+0000 7f51ca554640 10 client.4375.objecter successfully canceled tick 2024-03-25T18:30:13.562 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:30:13.559+0000 7f51ca554640 1 -- 172.21.15.1:0/3632331008 >> v1:172.21.15.115:6800/4237379176 conn(0x7f51b006df50 legacy=0x7f51b0070410 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:30:13.562 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:30:13.559+0000 7f51ca554640 10 monclient: shutdown 2024-03-25T18:30:13.562 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:30:13.559+0000 7f51ca554640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:30:13.562 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:30:13.559+0000 7f51ca554640 1 -- 172.21.15.1:0/3632331008 >> v1:172.21.15.115:6789/0 conn(0x7f51c40b44b0 legacy=0x7f51c413dc90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:30:13.562 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:30:13.559+0000 7f51ca554640 1 -- 172.21.15.1:0/3632331008 shutdown_connections 2024-03-25T18:30:13.562 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:30:13.559+0000 7f51ca554640 1 -- 172.21.15.1:0/3632331008 >> 172.21.15.1:0/3632331008 conn(0x7f51c40617a0 msgr2=0x7f51c40b2710 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:30:13.562 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:30:13.560+0000 7f51ca554640 1 -- 172.21.15.1:0/3632331008 shutdown_connections 2024-03-25T18:30:13.562 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:30:13.560+0000 7f51ca554640 1 -- 172.21.15.1:0/3632331008 wait complete. 2024-03-25T18:30:13.562 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:30:13.560+0000 7f51ca554640 1 librados: shutdown 2024-03-25T18:30:13.574 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T18:30:16.496 INFO:tasks.ceph.mon.a.smithi001.stderr:==00:00:03:55.054 34876== Warning: unimplemented fcntl command: 1036 2024-03-25T18:30:16.502 INFO:tasks.ceph.mon.a.smithi001.stderr:==00:00:03:55.060 34876== Warning: unimplemented fcntl command: 1036 2024-03-25T18:30:16.528 INFO:tasks.ceph.mon.a.smithi001.stderr:==00:00:03:55.087 34876== Warning: unimplemented fcntl command: 1036 2024-03-25T18:30:16.554 INFO:tasks.ceph.mon.b.smithi115.stderr:==00:00:03:55.038 34836== Warning: unimplemented fcntl command: 1036 2024-03-25T18:30:16.560 INFO:tasks.ceph.mon.c.smithi001.stderr:==00:00:03:55.118 34877== Warning: unimplemented fcntl command: 1036 2024-03-25T18:30:16.562 INFO:tasks.ceph.mon.b.smithi115.stderr:==00:00:03:55.046 34836== Warning: unimplemented fcntl command: 1036 2024-03-25T18:30:16.565 INFO:tasks.ceph.mon.c.smithi001.stderr:==00:00:03:55.124 34877== Warning: unimplemented fcntl command: 1036 2024-03-25T18:30:16.599 INFO:tasks.ceph.mon.b.smithi115.stderr:==00:00:03:55.083 34836== Warning: unimplemented fcntl command: 1036 2024-03-25T18:30:16.615 INFO:tasks.ceph.mon.c.smithi001.stderr:==00:00:03:55.173 34877== Warning: unimplemented fcntl command: 1036 2024-03-25T18:30:20.380 DEBUG:teuthology.orchestra.run.smithi001:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T18:30:20.493 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:30:20.491+0000 7f4528d96640 10 monclient: get_monmap_and_config 2024-03-25T18:30:20.493 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:30:20.491+0000 7f4528d96640 10 monclient: build_initial_monmap 2024-03-25T18:30:20.493 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:30:20.491+0000 7f4528d96640 10 monclient: monmap: 2024-03-25T18:30:20.493 INFO:teuthology.misc.health.smithi001.stderr:epoch 0 2024-03-25T18:30:20.494 INFO:teuthology.misc.health.smithi001.stderr:fsid c8ea72cf-9e99-4383-9b52-6c23d365fdfc 2024-03-25T18:30:20.494 INFO:teuthology.misc.health.smithi001.stderr:last_changed 2024-03-25T18:30:20.493382+0000 2024-03-25T18:30:20.494 INFO:teuthology.misc.health.smithi001.stderr:created 2024-03-25T18:30:20.493382+0000 2024-03-25T18:30:20.494 INFO:teuthology.misc.health.smithi001.stderr:min_mon_release 0 (unknown) 2024-03-25T18:30:20.494 INFO:teuthology.misc.health.smithi001.stderr:election_strategy: 3 2024-03-25T18:30:20.494 INFO:teuthology.misc.health.smithi001.stderr:0: v1:172.21.15.1:6789/0 mon.noname-a 2024-03-25T18:30:20.494 INFO:teuthology.misc.health.smithi001.stderr:1: v1:172.21.15.115:6789/0 mon.noname-c 2024-03-25T18:30:20.494 INFO:teuthology.misc.health.smithi001.stderr:2: v1:172.21.15.1:6790/0 mon.noname-b 2024-03-25T18:30:20.494 INFO:teuthology.misc.health.smithi001.stderr: 2024-03-25T18:30:20.495 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:30:20.492+0000 7f4528d96640 1 Processor -- start 2024-03-25T18:30:20.495 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:30:20.492+0000 7f4528d96640 1 -- start start 2024-03-25T18:30:20.496 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:30:20.492+0000 7f4528d96640 10 monclient: init 2024-03-25T18:30:20.496 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:30:20.493+0000 7f4528d96640 10 monclient: _reopen_session rank -1 2024-03-25T18:30:20.496 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:30:20.493+0000 7f4528d96640 10 monclient: _add_conns ranks=[2,0,1] 2024-03-25T18:30:20.496 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:30:20.493+0000 7f4528d96640 10 monclient(hunting): picked mon.noname-b con 0x7f452406f460 addr v1:172.21.15.1:6790/0 2024-03-25T18:30:20.496 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:30:20.493+0000 7f4528d96640 10 monclient(hunting): picked mon.noname-a con 0x7f452406f840 addr v1:172.21.15.1:6789/0 2024-03-25T18:30:20.496 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:30:20.493+0000 7f4528d96640 10 monclient(hunting): picked mon.noname-c con 0x7f45240728f0 addr v1:172.21.15.115:6789/0 2024-03-25T18:30:20.496 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:30:20.493+0000 7f4528d96640 1 -- --> v1:172.21.15.1:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4524071a40 con 0x7f452406f840 2024-03-25T18:30:20.496 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:30:20.493+0000 7f4528d96640 1 -- --> v1:172.21.15.115:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4524071db0 con 0x7f45240728f0 2024-03-25T18:30:20.496 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:30:20.493+0000 7f4528d96640 1 -- --> v1:172.21.15.1:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4524070650 con 0x7f452406f460 2024-03-25T18:30:20.496 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:30:20.493+0000 7f4528d96640 10 monclient(hunting): _renew_subs 2024-03-25T18:30:20.496 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:30:20.493+0000 7f4528d96640 10 monclient(hunting): authenticate will time out at 7700.964266s 2024-03-25T18:30:20.497 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:30:20.495+0000 7f4522d9d640 1 --1- >> v1:172.21.15.115:6789/0 conn(0x7f45240728f0 0x7f452414a8f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.115:6789/0 says I am v1:172.21.15.1:45724/0 (socket says 172.21.15.1:45724) 2024-03-25T18:30:20.497 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:30:20.495+0000 7f4522d9d640 1 -- 172.21.15.1:0/1950452578 learned_addr learned my addr 172.21.15.1:0/1950452578 (peer_addr_for_me v1:172.21.15.1:0/0) 2024-03-25T18:30:20.507 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:30:20.506+0000 7f452159a640 1 -- 172.21.15.1:0/1950452578 <== mon.1 v1:172.21.15.115:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3424407364 0 0) 0x7f4524071db0 con 0x7f45240728f0 2024-03-25T18:30:20.508 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:30:20.506+0000 7f452159a640 10 monclient(hunting): _negotiate 2024-03-25T18:30:20.508 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:30:20.506+0000 7f452159a640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:30:20.508 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:30:20.506+0000 7f452159a640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:30:20.508 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:30:20.506+0000 7f452159a640 10 monclient(hunting): my global_id is 4378 2024-03-25T18:30:20.508 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:30:20.506+0000 7f452159a640 1 -- 172.21.15.1:0/1950452578 --> v1:172.21.15.115:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f450c003520 con 0x7f45240728f0 2024-03-25T18:30:20.515 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:30:20.513+0000 7f452159a640 1 -- 172.21.15.1:0/1950452578 <== mon.1 v1:172.21.15.115:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 818070890 0 0) 0x7f450c003520 con 0x7f45240728f0 2024-03-25T18:30:20.515 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:30:20.513+0000 7f452159a640 1 -- 172.21.15.1:0/1950452578 >> v1:172.21.15.1:6790/0 conn(0x7f452406f460 legacy=0x7f4524142cb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-25T18:30:20.515 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:30:20.513+0000 7f452159a640 1 -- 172.21.15.1:0/1950452578 >> v1:172.21.15.1:6789/0 conn(0x7f452406f840 legacy=0x7f45240721e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-25T18:30:20.515 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:30:20.513+0000 7f452159a640 10 monclient: _finish_hunting 0 2024-03-25T18:30:20.515 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:30:20.513+0000 7f452159a640 1 monclient: found mon.noname-c 2024-03-25T18:30:20.515 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:30:20.513+0000 7f452159a640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:30:20.515 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:30:20.513+0000 7f452159a640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.115:6789/0 2024-03-25T18:30:20.515 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:30:20.513+0000 7f452159a640 1 -- 172.21.15.1:0/1950452578 --> v1:172.21.15.115:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4524070200 con 0x7f45240728f0 2024-03-25T18:30:20.515 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:30:20.513+0000 7f452159a640 10 monclient: _finish_auth 0 2024-03-25T18:30:20.515 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:30:20.513+0000 7f452159a640 10 monclient: _check_auth_tickets 2024-03-25T18:30:20.516 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:30:20.513+0000 7f452159a640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:30:20.516 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:30:20.513+0000 7f4528d96640 5 monclient: authenticate success, global_id 4378 2024-03-25T18:30:20.516 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:30:20.513+0000 7f4528d96640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T18:30:20.526 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:30:20.524+0000 7f452159a640 1 -- 172.21.15.1:0/1950452578 <== mon.1 v1:172.21.15.115:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1586965025 0 0) 0x7f451c001a10 con 0x7f45240728f0 2024-03-25T18:30:20.526 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:30:20.524+0000 7f452159a640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:30:20.526 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:30:20.524+0000 7f452159a640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-03-25T18:30:20.526 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:30:20.524+0000 7f452159a640 10 monclient: dump: 2024-03-25T18:30:20.526 INFO:teuthology.misc.health.smithi001.stderr:epoch 1 2024-03-25T18:30:20.526 INFO:teuthology.misc.health.smithi001.stderr:fsid c8ea72cf-9e99-4383-9b52-6c23d365fdfc 2024-03-25T18:30:20.527 INFO:teuthology.misc.health.smithi001.stderr:last_changed 2024-03-25T18:25:50.510519+0000 2024-03-25T18:30:20.527 INFO:teuthology.misc.health.smithi001.stderr:created 2024-03-25T18:25:50.510519+0000 2024-03-25T18:30:20.527 INFO:teuthology.misc.health.smithi001.stderr:min_mon_release 19 (squid) 2024-03-25T18:30:20.527 INFO:teuthology.misc.health.smithi001.stderr:election_strategy: 3 2024-03-25T18:30:20.528 INFO:teuthology.misc.health.smithi001.stderr:0: v1:172.21.15.1:6789/0 mon.a 2024-03-25T18:30:20.528 INFO:teuthology.misc.health.smithi001.stderr:1: v1:172.21.15.115:6789/0 mon.b 2024-03-25T18:30:20.528 INFO:teuthology.misc.health.smithi001.stderr:2: v1:172.21.15.1:6790/0 mon.c 2024-03-25T18:30:20.528 INFO:teuthology.misc.health.smithi001.stderr: 2024-03-25T18:30:20.528 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:30:20.524+0000 7f452159a640 1 -- 172.21.15.1:0/1950452578 <== mon.1 v1:172.21.15.115:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f451c001c70 con 0x7f45240728f0 2024-03-25T18:30:20.528 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:30:20.524+0000 7f4528d96640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T18:30:20.528 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:30:20.524+0000 7f452159a640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:30:20.528 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:30:20.524+0000 7f452159a640 1 -- 172.21.15.1:0/1950452578 <== mon.1 v1:172.21.15.115:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1586965025 0 0) 0x7f451c002000 con 0x7f45240728f0 2024-03-25T18:30:20.528 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:30:20.524+0000 7f452159a640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:30:20.528 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:30:20.524+0000 7f452159a640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-25T18:30:20.528 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:30:20.524+0000 7f452159a640 10 monclient: dump: 2024-03-25T18:30:20.528 INFO:teuthology.misc.health.smithi001.stderr:epoch 1 2024-03-25T18:30:20.528 INFO:teuthology.misc.health.smithi001.stderr:fsid c8ea72cf-9e99-4383-9b52-6c23d365fdfc 2024-03-25T18:30:20.529 INFO:teuthology.misc.health.smithi001.stderr:last_changed 2024-03-25T18:25:50.510519+0000 2024-03-25T18:30:20.529 INFO:teuthology.misc.health.smithi001.stderr:created 2024-03-25T18:25:50.510519+0000 2024-03-25T18:30:20.529 INFO:teuthology.misc.health.smithi001.stderr:min_mon_release 19 (squid) 2024-03-25T18:30:20.529 INFO:teuthology.misc.health.smithi001.stderr:election_strategy: 3 2024-03-25T18:30:20.529 INFO:teuthology.misc.health.smithi001.stderr:0: v1:172.21.15.1:6789/0 mon.a 2024-03-25T18:30:20.529 INFO:teuthology.misc.health.smithi001.stderr:1: v1:172.21.15.115:6789/0 mon.b 2024-03-25T18:30:20.529 INFO:teuthology.misc.health.smithi001.stderr:2: v1:172.21.15.1:6790/0 mon.c 2024-03-25T18:30:20.529 INFO:teuthology.misc.health.smithi001.stderr: 2024-03-25T18:30:20.529 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:30:20.524+0000 7f4528d96640 10 monclient: get_monmap_and_config success 2024-03-25T18:30:20.529 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:30:20.524+0000 7f4528d96640 4 set_mon_vals no callback set 2024-03-25T18:30:20.531 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:30:20.525+0000 7f4528d96640 10 monclient: shutdown 2024-03-25T18:30:20.531 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:30:20.525+0000 7f4528d96640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:30:20.531 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:30:20.525+0000 7f4528d96640 1 -- 172.21.15.1:0/1950452578 >> v1:172.21.15.115:6789/0 conn(0x7f45240728f0 legacy=0x7f452414a8f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:30:20.531 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:30:20.525+0000 7f4528d96640 1 -- 172.21.15.1:0/1950452578 shutdown_connections 2024-03-25T18:30:20.531 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:30:20.525+0000 7f4528d96640 1 -- 172.21.15.1:0/1950452578 >> 172.21.15.1:0/1950452578 conn(0x7f45240b4bd0 msgr2=0x7f45240b4fe0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:30:20.531 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:30:20.525+0000 7f4528d96640 1 -- 172.21.15.1:0/1950452578 shutdown_connections 2024-03-25T18:30:20.531 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:30:20.525+0000 7f4528d96640 1 -- 172.21.15.1:0/1950452578 wait complete. 2024-03-25T18:30:20.531 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:30:20.526+0000 7f4528d96640 10 monclient: build_initial_monmap 2024-03-25T18:30:20.533 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:30:20.526+0000 7f4528d96640 10 monclient: monmap: 2024-03-25T18:30:20.533 INFO:teuthology.misc.health.smithi001.stderr:epoch 0 2024-03-25T18:30:20.533 INFO:teuthology.misc.health.smithi001.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T18:30:20.533 INFO:teuthology.misc.health.smithi001.stderr:last_changed 0.000000 2024-03-25T18:30:20.533 INFO:teuthology.misc.health.smithi001.stderr:created 0.000000 2024-03-25T18:30:20.533 INFO:teuthology.misc.health.smithi001.stderr:min_mon_release 0 (unknown) 2024-03-25T18:30:20.534 INFO:teuthology.misc.health.smithi001.stderr:election_strategy: 1 2024-03-25T18:30:20.534 INFO:teuthology.misc.health.smithi001.stderr:0: v1:172.21.15.1:6789/0 mon.noname-a 2024-03-25T18:30:20.534 INFO:teuthology.misc.health.smithi001.stderr:1: v1:172.21.15.115:6789/0 mon.noname-b 2024-03-25T18:30:20.534 INFO:teuthology.misc.health.smithi001.stderr:2: v1:172.21.15.1:6790/0 mon.noname-c 2024-03-25T18:30:20.534 INFO:teuthology.misc.health.smithi001.stderr: 2024-03-25T18:30:20.534 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:30:20.526+0000 7f4528d96640 1 librados: starting msgr at 2024-03-25T18:30:20.534 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:30:20.526+0000 7f4528d96640 1 librados: starting objecter 2024-03-25T18:30:20.534 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:30:20.526+0000 7f4528d96640 1 Processor -- start 2024-03-25T18:30:20.534 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:30:20.526+0000 7f4528d96640 1 -- start start 2024-03-25T18:30:20.534 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:30:20.526+0000 7f4528d96640 1 librados: setting wanted keys 2024-03-25T18:30:20.534 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:30:20.526+0000 7f4528d96640 1 librados: calling monclient init 2024-03-25T18:30:20.535 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:30:20.526+0000 7f4528d96640 10 monclient: init 2024-03-25T18:30:20.535 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:30:20.527+0000 7f4528d96640 10 monclient: _reopen_session rank -1 2024-03-25T18:30:20.535 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:30:20.527+0000 7f4528d96640 10 monclient: _add_conns ranks=[1,0,2] 2024-03-25T18:30:20.535 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:30:20.527+0000 7f4528d96640 10 monclient(hunting): picked mon.noname-b con 0x7f452406f460 addr v1:172.21.15.115:6789/0 2024-03-25T18:30:20.535 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:30:20.527+0000 7f4528d96640 10 monclient(hunting): picked mon.noname-a con 0x7f452406f840 addr v1:172.21.15.1:6789/0 2024-03-25T18:30:20.535 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:30:20.527+0000 7f4528d96640 10 monclient(hunting): picked mon.noname-c con 0x7f45240728f0 addr v1:172.21.15.1:6790/0 2024-03-25T18:30:20.535 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:30:20.527+0000 7f4528d96640 1 -- --> v1:172.21.15.1:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f45240adae0 con 0x7f452406f840 2024-03-25T18:30:20.535 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:30:20.527+0000 7f4528d96640 1 -- --> v1:172.21.15.115:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f45240ad770 con 0x7f452406f460 2024-03-25T18:30:20.535 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:30:20.527+0000 7f4528d96640 1 -- --> v1:172.21.15.1:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f452413a690 con 0x7f45240728f0 2024-03-25T18:30:20.535 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:30:20.527+0000 7f4528d96640 10 monclient(hunting): _renew_subs 2024-03-25T18:30:20.535 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:30:20.527+0000 7f4528d96640 10 monclient(hunting): authenticate will time out at 7700.998362s 2024-03-25T18:30:20.537 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:30:20.536+0000 7f4522d9d640 1 --1- >> v1:172.21.15.1:6790/0 conn(0x7f45240728f0 0x7f45241dfc70 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.1:6790/0 says I am v1:172.21.15.1:41874/0 (socket says 172.21.15.1:41874) 2024-03-25T18:30:20.537 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:30:20.536+0000 7f4522d9d640 1 -- 172.21.15.1:0/3195241094 learned_addr learned my addr 172.21.15.1:0/3195241094 (peer_addr_for_me v1:172.21.15.1:0/0) 2024-03-25T18:30:20.538 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:30:20.537+0000 7f4512ffd640 10 client.?.objecter ms_handle_connect 0x7f452406f460 2024-03-25T18:30:20.538 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:30:20.537+0000 7f4512ffd640 10 client.?.objecter resend_mon_ops 2024-03-25T18:30:20.539 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:30:20.538+0000 7f4512ffd640 10 client.?.objecter ms_handle_connect 0x7f45240728f0 2024-03-25T18:30:20.539 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:30:20.538+0000 7f4512ffd640 10 client.?.objecter resend_mon_ops 2024-03-25T18:30:20.547 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:30:20.546+0000 7f4512ffd640 1 -- 172.21.15.1:0/3195241094 <== mon.1 v1:172.21.15.115:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3285894336 0 0) 0x7f45240ad770 con 0x7f452406f460 2024-03-25T18:30:20.547 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:30:20.546+0000 7f4512ffd640 10 monclient(hunting): _negotiate 2024-03-25T18:30:20.547 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:30:20.546+0000 7f4512ffd640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:30:20.548 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:30:20.546+0000 7f4512ffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:30:20.548 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:30:20.546+0000 7f4512ffd640 10 monclient(hunting): my global_id is 4381 2024-03-25T18:30:20.548 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:30:20.546+0000 7f4512ffd640 1 -- 172.21.15.1:0/3195241094 --> v1:172.21.15.115:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f4504004eb0 con 0x7f452406f460 2024-03-25T18:30:20.551 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:30:20.550+0000 7f4512ffd640 10 client.?.objecter ms_handle_connect 0x7f452406f840 2024-03-25T18:30:20.551 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:30:20.550+0000 7f4512ffd640 10 client.?.objecter resend_mon_ops 2024-03-25T18:30:20.554 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:30:20.552+0000 7f4512ffd640 1 -- 172.21.15.1:0/3195241094 <== mon.2 v1:172.21.15.1:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 126605409 0 0) 0x7f452413a690 con 0x7f45240728f0 2024-03-25T18:30:20.554 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:30:20.552+0000 7f4512ffd640 10 monclient(hunting): _negotiate 2024-03-25T18:30:20.554 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:30:20.552+0000 7f4512ffd640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:30:20.554 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:30:20.552+0000 7f4512ffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:30:20.554 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:30:20.552+0000 7f4512ffd640 10 monclient(hunting): my global_id is 4385 2024-03-25T18:30:20.554 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:30:20.552+0000 7f4512ffd640 1 -- 172.21.15.1:0/3195241094 --> v1:172.21.15.1:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f45240ad770 con 0x7f45240728f0 2024-03-25T18:30:20.560 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:30:20.559+0000 7f4512ffd640 1 -- 172.21.15.1:0/3195241094 <== mon.0 v1:172.21.15.1:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 784125610 0 0) 0x7f45240adae0 con 0x7f452406f840 2024-03-25T18:30:20.560 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:30:20.559+0000 7f4512ffd640 10 monclient(hunting): _negotiate 2024-03-25T18:30:20.560 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:30:20.559+0000 7f4512ffd640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:30:20.561 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:30:20.559+0000 7f4512ffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:30:20.561 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:30:20.559+0000 7f4512ffd640 10 monclient(hunting): my global_id is 4377 2024-03-25T18:30:20.561 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:30:20.559+0000 7f4512ffd640 1 -- 172.21.15.1:0/3195241094 --> v1:172.21.15.1:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f452413a690 con 0x7f452406f840 2024-03-25T18:30:20.562 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:30:20.560+0000 7f4512ffd640 1 -- 172.21.15.1:0/3195241094 <== mon.1 v1:172.21.15.115:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 4171496828 0 0) 0x7f4504004eb0 con 0x7f452406f460 2024-03-25T18:30:20.562 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:30:20.560+0000 7f4512ffd640 1 -- 172.21.15.1:0/3195241094 --> v1:172.21.15.115:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f45240adae0 con 0x7f452406f460 2024-03-25T18:30:20.562 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:30:20.560+0000 7f4512ffd640 1 -- 172.21.15.1:0/3195241094 <== mon.1 v1:172.21.15.115:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1586965025 0 0) 0x7f4514001870 con 0x7f452406f460 2024-03-25T18:30:20.562 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:30:20.560+0000 7f4512ffd640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:30:20.562 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:30:20.560+0000 7f4512ffd640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-03-25T18:30:20.563 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:30:20.560+0000 7f4512ffd640 10 monclient(hunting): dump: 2024-03-25T18:30:20.563 INFO:teuthology.misc.health.smithi001.stderr:epoch 1 2024-03-25T18:30:20.563 INFO:teuthology.misc.health.smithi001.stderr:fsid c8ea72cf-9e99-4383-9b52-6c23d365fdfc 2024-03-25T18:30:20.563 INFO:teuthology.misc.health.smithi001.stderr:last_changed 2024-03-25T18:25:50.510519+0000 2024-03-25T18:30:20.563 INFO:teuthology.misc.health.smithi001.stderr:created 2024-03-25T18:25:50.510519+0000 2024-03-25T18:30:20.563 INFO:teuthology.misc.health.smithi001.stderr:min_mon_release 19 (squid) 2024-03-25T18:30:20.563 INFO:teuthology.misc.health.smithi001.stderr:election_strategy: 3 2024-03-25T18:30:20.563 INFO:teuthology.misc.health.smithi001.stderr:0: v1:172.21.15.1:6789/0 mon.a 2024-03-25T18:30:20.563 INFO:teuthology.misc.health.smithi001.stderr:1: v1:172.21.15.115:6789/0 mon.b 2024-03-25T18:30:20.563 INFO:teuthology.misc.health.smithi001.stderr:2: v1:172.21.15.1:6790/0 mon.c 2024-03-25T18:30:20.563 INFO:teuthology.misc.health.smithi001.stderr: 2024-03-25T18:30:20.563 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:30:20.560+0000 7f4512ffd640 10 monclient(hunting): _finish_auth 0 2024-03-25T18:30:20.582 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:30:20.578+0000 7f4512ffd640 1 -- 172.21.15.1:0/3195241094 <== mon.1 v1:172.21.15.115:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 3787199537 0 0) 0x7f45240adae0 con 0x7f452406f460 2024-03-25T18:30:20.582 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:30:20.578+0000 7f4512ffd640 1 -- 172.21.15.1:0/3195241094 >> v1:172.21.15.1:6790/0 conn(0x7f45240728f0 legacy=0x7f45241dfc70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:30:20.583 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:30:20.578+0000 7f4512ffd640 1 -- 172.21.15.1:0/3195241094 >> v1:172.21.15.1:6789/0 conn(0x7f452406f840 legacy=0x7f45241d5100 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:30:20.583 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:30:20.578+0000 7f4512ffd640 10 monclient: _finish_hunting 0 2024-03-25T18:30:20.583 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:30:20.578+0000 7f4512ffd640 1 monclient: found mon.b 2024-03-25T18:30:20.583 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:30:20.578+0000 7f4512ffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:30:20.583 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:30:20.578+0000 7f4512ffd640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.115:6789/0 2024-03-25T18:30:20.583 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:30:20.578+0000 7f4512ffd640 1 -- 172.21.15.1:0/3195241094 --> v1:172.21.15.115:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f452413a820 con 0x7f452406f460 2024-03-25T18:30:20.583 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:30:20.578+0000 7f4512ffd640 10 monclient: _finish_auth 0 2024-03-25T18:30:20.583 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:30:20.578+0000 7f4512ffd640 10 monclient: _check_auth_tickets 2024-03-25T18:30:20.583 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:30:20.578+0000 7f4512ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:30:20.583 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:30:20.578+0000 7f4528d96640 5 monclient: authenticate success, global_id 4381 2024-03-25T18:30:20.583 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:30:20.578+0000 7f4528d96640 10 monclient: _renew_subs 2024-03-25T18:30:20.583 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:30:20.578+0000 7f4528d96640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.115:6789/0 2024-03-25T18:30:20.583 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:30:20.578+0000 7f4528d96640 1 -- 172.21.15.1:0/3195241094 --> v1:172.21.15.115:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f45241d5810 con 0x7f452406f460 2024-03-25T18:30:20.583 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:30:20.579+0000 7f4528d96640 10 client.4381.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T18:30:20.584 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:30:20.579+0000 7f4528d96640 10 monclient: _renew_subs 2024-03-25T18:30:20.584 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:30:20.579+0000 7f4528d96640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.115:6789/0 2024-03-25T18:30:20.584 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:30:20.579+0000 7f4528d96640 1 -- 172.21.15.1:0/3195241094 --> v1:172.21.15.115:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f452406b2b0 con 0x7f452406f460 2024-03-25T18:30:20.584 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:30:20.579+0000 7f4528d96640 1 librados: init done 2024-03-25T18:30:20.584 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:30:20.580+0000 7f4528d96640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T18:30:20.584 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:30:20.580+0000 7f4528d96640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:30:20.584 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:30:20.580+0000 7f4528d96640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.115:6789/0 2024-03-25T18:30:20.584 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:30:20.580+0000 7f4528d96640 1 -- 172.21.15.1:0/3195241094 --> v1:172.21.15.115:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4524072d20 con 0x7f452406f460 2024-03-25T18:30:20.847 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:30:20.845+0000 7f4512ffd640 1 -- 172.21.15.1:0/3195241094 <== mon.1 v1:172.21.15.115:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f45140035f0 con 0x7f452406f460 2024-03-25T18:30:20.847 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:30:20.845+0000 7f4512ffd640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:30:20.847 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:30:20.845+0000 7f4512ffd640 1 -- 172.21.15.1:0/3195241094 <== mon.1 v1:172.21.15.115:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1586965025 0 0) 0x7f4514001d80 con 0x7f452406f460 2024-03-25T18:30:20.848 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:30:20.845+0000 7f4512ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:30:20.848 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:30:20.845+0000 7f4512ffd640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-25T18:30:20.848 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:30:20.845+0000 7f4512ffd640 10 monclient: dump: 2024-03-25T18:30:20.848 INFO:teuthology.misc.health.smithi001.stderr:epoch 1 2024-03-25T18:30:20.848 INFO:teuthology.misc.health.smithi001.stderr:fsid c8ea72cf-9e99-4383-9b52-6c23d365fdfc 2024-03-25T18:30:20.848 INFO:teuthology.misc.health.smithi001.stderr:last_changed 2024-03-25T18:25:50.510519+0000 2024-03-25T18:30:20.848 INFO:teuthology.misc.health.smithi001.stderr:created 2024-03-25T18:25:50.510519+0000 2024-03-25T18:30:20.848 INFO:teuthology.misc.health.smithi001.stderr:min_mon_release 19 (squid) 2024-03-25T18:30:20.848 INFO:teuthology.misc.health.smithi001.stderr:election_strategy: 3 2024-03-25T18:30:20.848 INFO:teuthology.misc.health.smithi001.stderr:0: v1:172.21.15.1:6789/0 mon.a 2024-03-25T18:30:20.848 INFO:teuthology.misc.health.smithi001.stderr:1: v1:172.21.15.115:6789/0 mon.b 2024-03-25T18:30:20.848 INFO:teuthology.misc.health.smithi001.stderr:2: v1:172.21.15.1:6790/0 mon.c 2024-03-25T18:30:20.848 INFO:teuthology.misc.health.smithi001.stderr: 2024-03-25T18:30:20.849 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:30:20.845+0000 7f4520d99640 4 set_mon_vals no callback set 2024-03-25T18:30:20.849 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:30:20.846+0000 7f4512ffd640 1 -- 172.21.15.1:0/3195241094 <== mon.1 v1:172.21.15.115:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 21787975 0 0) 0x7f4514019830 con 0x7f452406f460 2024-03-25T18:30:20.849 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:30:20.847+0000 7f4521d9b640 10 monclient: get_auth_request con 0x7f450406de20 auth_method 0 2024-03-25T18:30:20.849 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:30:20.847+0000 7f4512ffd640 10 client.4381.objecter ms_handle_connect 0x7f450406de20 2024-03-25T18:30:20.852 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:30:20.850+0000 7f4512ffd640 1 -- 172.21.15.1:0/3195241094 <== mon.1 v1:172.21.15.115:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 996254230 0 0) 0x7f45140535b0 con 0x7f452406f460 2024-03-25T18:30:20.852 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:30:20.850+0000 7f4512ffd640 10 client.4381.objecter ms_dispatch 0x7f4524000b90 osd_map(10..10 src has 1..10) v4 2024-03-25T18:30:20.852 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:30:20.850+0000 7f4512ffd640 3 client.4381.objecter handle_osd_map got epochs [10,10] > 0 2024-03-25T18:30:20.852 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:30:20.850+0000 7f4512ffd640 3 client.4381.objecter handle_osd_map decoding full epoch 10 2024-03-25T18:30:20.852 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:30:20.850+0000 7f4512ffd640 20 client.4381.objecter dump_active .. 0 homeless 2024-03-25T18:30:20.853 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:30:20.851+0000 7f4512ffd640 1 -- 172.21.15.1:0/3195241094 <== mon.1 v1:172.21.15.115:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f451404eef0 con 0x7f452406f460 2024-03-25T18:30:20.853 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:30:20.851+0000 7f4512ffd640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:30:20.853 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:30:20.851+0000 7f4512ffd640 10 monclient: _finish_command 1 = system:0 2024-03-25T18:30:21.080 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:30:21.078+0000 7f4528d96640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:30:21.080 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:30:21.078+0000 7f4528d96640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:30:21.080 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:30:21.078+0000 7f4528d96640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.115:6789/0 2024-03-25T18:30:21.080 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:30:21.078+0000 7f4528d96640 1 -- 172.21.15.1:0/3195241094 --> v1:172.21.15.115:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f45240b39c0 con 0x7f452406f460 2024-03-25T18:30:21.104 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:30:21.102+0000 7f4512ffd640 1 -- 172.21.15.1:0/3195241094 <== mon.1 v1:172.21.15.115:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 4092213348) 0x7f4514050ff0 con 0x7f452406f460 2024-03-25T18:30:21.104 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:30:21.102+0000 7f4512ffd640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:30:21.104 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:30:21.102+0000 7f4512ffd640 10 monclient: _finish_command 2 = system:0 2024-03-25T18:30:21.105 INFO:teuthology.misc.health.smithi001.stdout: 2024-03-25T18:30:21.105 INFO:teuthology.misc.health.smithi001.stdout:{"epoch":10,"fsid":"c8ea72cf-9e99-4383-9b52-6c23d365fdfc","created":"2024-03-25T18:26:38.856638+0000","modified":"2024-03-25T18:26:44.224208+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T18:26:44.224208+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"b9f34922-3a84-49be-8426-807c1eceaefc","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"42bf306f-55e3-4634-9874-8b4b73ac6644","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"292cb23a-18a9-4207-a7cc-4d5727feaec0","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"45e0e9ab-898d-4673-9590-b4117bc5e84a","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"831a5a09-73ab-4a61-9872-088e76bdbc66","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"88cd903d-3b55-4840-9396-6e3cfe005282","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"89db7355-bc9d-46f4-8ab0-e8d9e744e026","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"cac47bee-0914-4f08-896f-348838882866","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T18:30:21.107 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:30:21.104+0000 7f4528d96640 10 librados: watch_flush enter 2024-03-25T18:30:21.107 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:30:21.104+0000 7f4528d96640 10 librados: watch_flush exit 2024-03-25T18:30:21.107 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:30:21.104+0000 7f4528d96640 20 client.4381.objecter shutdown clearing up homeless session... 2024-03-25T18:30:21.107 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:30:21.104+0000 7f4528d96640 10 client.4381.objecter successfully canceled tick 2024-03-25T18:30:21.107 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:30:21.105+0000 7f4528d96640 1 -- 172.21.15.1:0/3195241094 >> v1:172.21.15.115:6800/4237379176 conn(0x7f450406de20 legacy=0x7f45040702e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:30:21.107 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:30:21.105+0000 7f4528d96640 10 monclient: shutdown 2024-03-25T18:30:21.107 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:30:21.105+0000 7f4528d96640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:30:21.107 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:30:21.105+0000 7f4528d96640 1 -- 172.21.15.1:0/3195241094 >> v1:172.21.15.115:6789/0 conn(0x7f452406f460 legacy=0x7f45240adf70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:30:21.107 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:30:21.105+0000 7f4528d96640 1 -- 172.21.15.1:0/3195241094 shutdown_connections 2024-03-25T18:30:21.107 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:30:21.105+0000 7f4528d96640 1 -- 172.21.15.1:0/3195241094 >> 172.21.15.1:0/3195241094 conn(0x7f45240b4bd0 msgr2=0x7f452413b280 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:30:21.107 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:30:21.105+0000 7f4528d96640 1 -- 172.21.15.1:0/3195241094 shutdown_connections 2024-03-25T18:30:21.107 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:30:21.105+0000 7f4528d96640 1 -- 172.21.15.1:0/3195241094 wait complete. 2024-03-25T18:30:21.107 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:30:21.105+0000 7f4528d96640 1 librados: shutdown 2024-03-25T18:30:21.117 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T18:30:27.924 DEBUG:teuthology.orchestra.run.smithi001:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T18:30:28.039 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:30:28.037+0000 7f4204633640 10 monclient: get_monmap_and_config 2024-03-25T18:30:28.039 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:30:28.037+0000 7f4204633640 10 monclient: build_initial_monmap 2024-03-25T18:30:28.039 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:30:28.037+0000 7f4204633640 10 monclient: monmap: 2024-03-25T18:30:28.039 INFO:teuthology.misc.health.smithi001.stderr:epoch 0 2024-03-25T18:30:28.039 INFO:teuthology.misc.health.smithi001.stderr:fsid c8ea72cf-9e99-4383-9b52-6c23d365fdfc 2024-03-25T18:30:28.039 INFO:teuthology.misc.health.smithi001.stderr:last_changed 2024-03-25T18:30:28.038832+0000 2024-03-25T18:30:28.039 INFO:teuthology.misc.health.smithi001.stderr:created 2024-03-25T18:30:28.038832+0000 2024-03-25T18:30:28.039 INFO:teuthology.misc.health.smithi001.stderr:min_mon_release 0 (unknown) 2024-03-25T18:30:28.039 INFO:teuthology.misc.health.smithi001.stderr:election_strategy: 3 2024-03-25T18:30:28.039 INFO:teuthology.misc.health.smithi001.stderr:0: v1:172.21.15.1:6789/0 mon.noname-a 2024-03-25T18:30:28.039 INFO:teuthology.misc.health.smithi001.stderr:1: v1:172.21.15.115:6789/0 mon.noname-c 2024-03-25T18:30:28.040 INFO:teuthology.misc.health.smithi001.stderr:2: v1:172.21.15.1:6790/0 mon.noname-b 2024-03-25T18:30:28.040 INFO:teuthology.misc.health.smithi001.stderr: 2024-03-25T18:30:28.040 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:30:28.038+0000 7f4204633640 1 Processor -- start 2024-03-25T18:30:28.041 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:30:28.038+0000 7f4204633640 1 -- start start 2024-03-25T18:30:28.041 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:30:28.038+0000 7f4204633640 10 monclient: init 2024-03-25T18:30:28.041 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:30:28.039+0000 7f4204633640 10 monclient: _reopen_session rank -1 2024-03-25T18:30:28.041 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:30:28.039+0000 7f4204633640 10 monclient: _add_conns ranks=[2,1,0] 2024-03-25T18:30:28.041 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:30:28.039+0000 7f4204633640 10 monclient(hunting): picked mon.noname-b con 0x7f41fc0b4ff0 addr v1:172.21.15.1:6790/0 2024-03-25T18:30:28.041 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:30:28.039+0000 7f4204633640 10 monclient(hunting): picked mon.noname-c con 0x7f41fc0b4670 addr v1:172.21.15.115:6789/0 2024-03-25T18:30:28.041 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:30:28.039+0000 7f4204633640 10 monclient(hunting): picked mon.noname-a con 0x7f41fc13c480 addr v1:172.21.15.1:6789/0 2024-03-25T18:30:28.041 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:30:28.039+0000 7f4204633640 1 -- --> v1:172.21.15.1:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f41fc0ab0e0 con 0x7f41fc13c480 2024-03-25T18:30:28.041 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:30:28.039+0000 7f4204633640 1 -- --> v1:172.21.15.115:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f41fc0ab3c0 con 0x7f41fc0b4670 2024-03-25T18:30:28.041 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:30:28.039+0000 7f4204633640 1 -- --> v1:172.21.15.1:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f41fc0a9f30 con 0x7f41fc0b4ff0 2024-03-25T18:30:28.041 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:30:28.039+0000 7f4204633640 10 monclient(hunting): _renew_subs 2024-03-25T18:30:28.041 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:30:28.039+0000 7f4204633640 10 monclient(hunting): authenticate will time out at 7708.509837s 2024-03-25T18:30:28.057 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:30:28.055+0000 7f4201bce640 1 --1- >> v1:172.21.15.115:6789/0 conn(0x7f41fc0b4670 0x7f41fc1425c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.115:6789/0 says I am v1:172.21.15.1:34130/0 (socket says 172.21.15.1:34130) 2024-03-25T18:30:28.057 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:30:28.055+0000 7f4201bce640 1 -- 172.21.15.1:0/1214760001 learned_addr learned my addr 172.21.15.1:0/1214760001 (peer_addr_for_me v1:172.21.15.1:0/0) 2024-03-25T18:30:28.069 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:30:28.068+0000 7f42013cd640 1 -- 172.21.15.1:0/1214760001 <== mon.1 v1:172.21.15.115:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4120184379 0 0) 0x7f41fc0ab3c0 con 0x7f41fc0b4670 2024-03-25T18:30:28.069 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:30:28.068+0000 7f42013cd640 10 monclient(hunting): _negotiate 2024-03-25T18:30:28.070 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:30:28.068+0000 7f42013cd640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:30:28.070 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:30:28.068+0000 7f42013cd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:30:28.070 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:30:28.068+0000 7f42013cd640 10 monclient(hunting): my global_id is 4384 2024-03-25T18:30:28.070 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:30:28.068+0000 7f42013cd640 1 -- 172.21.15.1:0/1214760001 --> v1:172.21.15.115:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f41e8003520 con 0x7f41fc0b4670 2024-03-25T18:30:28.077 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:30:28.074+0000 7f42013cd640 1 -- 172.21.15.1:0/1214760001 <== mon.1 v1:172.21.15.115:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3549108264 0 0) 0x7f41e8003520 con 0x7f41fc0b4670 2024-03-25T18:30:28.077 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:30:28.074+0000 7f42013cd640 1 -- 172.21.15.1:0/1214760001 >> v1:172.21.15.1:6790/0 conn(0x7f41fc0b4ff0 legacy=0x7f41fc13bd70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:30:28.077 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:30:28.075+0000 7f42013cd640 1 -- 172.21.15.1:0/1214760001 >> v1:172.21.15.1:6789/0 conn(0x7f41fc13c480 legacy=0x7f41fc14dc00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:30:28.077 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:30:28.075+0000 7f42013cd640 10 monclient: _finish_hunting 0 2024-03-25T18:30:28.077 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:30:28.075+0000 7f42013cd640 1 monclient: found mon.noname-c 2024-03-25T18:30:28.077 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:30:28.075+0000 7f42013cd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:30:28.077 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:30:28.075+0000 7f42013cd640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.115:6789/0 2024-03-25T18:30:28.077 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:30:28.075+0000 7f42013cd640 1 -- 172.21.15.1:0/1214760001 --> v1:172.21.15.115:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f41fc0a9b70 con 0x7f41fc0b4670 2024-03-25T18:30:28.077 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:30:28.075+0000 7f42013cd640 10 monclient: _finish_auth 0 2024-03-25T18:30:28.077 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:30:28.075+0000 7f42013cd640 10 monclient: _check_auth_tickets 2024-03-25T18:30:28.077 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:30:28.075+0000 7f42013cd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:30:28.077 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:30:28.075+0000 7f4204633640 5 monclient: authenticate success, global_id 4384 2024-03-25T18:30:28.077 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:30:28.075+0000 7f4204633640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T18:30:28.079 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:30:28.077+0000 7f42013cd640 1 -- 172.21.15.1:0/1214760001 <== mon.1 v1:172.21.15.115:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1586965025 0 0) 0x7f41f8001690 con 0x7f41fc0b4670 2024-03-25T18:30:28.079 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:30:28.077+0000 7f42013cd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:30:28.079 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:30:28.077+0000 7f42013cd640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-03-25T18:30:28.079 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:30:28.077+0000 7f42013cd640 10 monclient: dump: 2024-03-25T18:30:28.079 INFO:teuthology.misc.health.smithi001.stderr:epoch 1 2024-03-25T18:30:28.079 INFO:teuthology.misc.health.smithi001.stderr:fsid c8ea72cf-9e99-4383-9b52-6c23d365fdfc 2024-03-25T18:30:28.079 INFO:teuthology.misc.health.smithi001.stderr:last_changed 2024-03-25T18:25:50.510519+0000 2024-03-25T18:30:28.079 INFO:teuthology.misc.health.smithi001.stderr:created 2024-03-25T18:25:50.510519+0000 2024-03-25T18:30:28.079 INFO:teuthology.misc.health.smithi001.stderr:min_mon_release 19 (squid) 2024-03-25T18:30:28.079 INFO:teuthology.misc.health.smithi001.stderr:election_strategy: 3 2024-03-25T18:30:28.079 INFO:teuthology.misc.health.smithi001.stderr:0: v1:172.21.15.1:6789/0 mon.a 2024-03-25T18:30:28.079 INFO:teuthology.misc.health.smithi001.stderr:1: v1:172.21.15.115:6789/0 mon.b 2024-03-25T18:30:28.079 INFO:teuthology.misc.health.smithi001.stderr:2: v1:172.21.15.1:6790/0 mon.c 2024-03-25T18:30:28.080 INFO:teuthology.misc.health.smithi001.stderr: 2024-03-25T18:30:28.080 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:30:28.077+0000 7f4204633640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T18:30:28.082 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:30:28.081+0000 7f42013cd640 1 -- 172.21.15.1:0/1214760001 <== mon.1 v1:172.21.15.115:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f41f8001920 con 0x7f41fc0b4670 2024-03-25T18:30:28.082 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:30:28.081+0000 7f42013cd640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:30:28.082 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:30:28.081+0000 7f42013cd640 1 -- 172.21.15.1:0/1214760001 <== mon.1 v1:172.21.15.115:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1586965025 0 0) 0x7f41f8001c30 con 0x7f41fc0b4670 2024-03-25T18:30:28.083 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:30:28.081+0000 7f4204633640 10 monclient: get_monmap_and_config success 2024-03-25T18:30:28.083 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:30:28.081+0000 7f4204633640 4 set_mon_vals no callback set 2024-03-25T18:30:28.083 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:30:28.081+0000 7f42013cd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:30:28.083 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:30:28.081+0000 7f42013cd640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-25T18:30:28.083 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:30:28.081+0000 7f42013cd640 10 monclient: dump: 2024-03-25T18:30:28.083 INFO:teuthology.misc.health.smithi001.stderr:epoch 1 2024-03-25T18:30:28.083 INFO:teuthology.misc.health.smithi001.stderr:fsid c8ea72cf-9e99-4383-9b52-6c23d365fdfc 2024-03-25T18:30:28.083 INFO:teuthology.misc.health.smithi001.stderr:last_changed 2024-03-25T18:25:50.510519+0000 2024-03-25T18:30:28.083 INFO:teuthology.misc.health.smithi001.stderr:created 2024-03-25T18:25:50.510519+0000 2024-03-25T18:30:28.083 INFO:teuthology.misc.health.smithi001.stderr:min_mon_release 19 (squid) 2024-03-25T18:30:28.083 INFO:teuthology.misc.health.smithi001.stderr:election_strategy: 3 2024-03-25T18:30:28.083 INFO:teuthology.misc.health.smithi001.stderr:0: v1:172.21.15.1:6789/0 mon.a 2024-03-25T18:30:28.083 INFO:teuthology.misc.health.smithi001.stderr:1: v1:172.21.15.115:6789/0 mon.b 2024-03-25T18:30:28.083 INFO:teuthology.misc.health.smithi001.stderr:2: v1:172.21.15.1:6790/0 mon.c 2024-03-25T18:30:28.083 INFO:teuthology.misc.health.smithi001.stderr: 2024-03-25T18:30:28.085 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:30:28.081+0000 7f4204633640 10 monclient: shutdown 2024-03-25T18:30:28.085 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:30:28.081+0000 7f4204633640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:30:28.085 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:30:28.081+0000 7f4204633640 1 -- 172.21.15.1:0/1214760001 >> v1:172.21.15.115:6789/0 conn(0x7f41fc0b4670 legacy=0x7f41fc1425c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:30:28.085 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:30:28.082+0000 7f4204633640 1 -- 172.21.15.1:0/1214760001 shutdown_connections 2024-03-25T18:30:28.085 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:30:28.082+0000 7f4204633640 1 -- 172.21.15.1:0/1214760001 >> 172.21.15.1:0/1214760001 conn(0x7f41fc0617a0 msgr2=0x7f41fc061bb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:30:28.085 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:30:28.082+0000 7f4204633640 1 -- 172.21.15.1:0/1214760001 shutdown_connections 2024-03-25T18:30:28.085 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:30:28.082+0000 7f4204633640 1 -- 172.21.15.1:0/1214760001 wait complete. 2024-03-25T18:30:28.085 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:30:28.083+0000 7f4204633640 10 monclient: build_initial_monmap 2024-03-25T18:30:28.085 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:30:28.083+0000 7f4204633640 10 monclient: monmap: 2024-03-25T18:30:28.085 INFO:teuthology.misc.health.smithi001.stderr:epoch 0 2024-03-25T18:30:28.085 INFO:teuthology.misc.health.smithi001.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T18:30:28.086 INFO:teuthology.misc.health.smithi001.stderr:last_changed 0.000000 2024-03-25T18:30:28.086 INFO:teuthology.misc.health.smithi001.stderr:created 0.000000 2024-03-25T18:30:28.086 INFO:teuthology.misc.health.smithi001.stderr:min_mon_release 0 (unknown) 2024-03-25T18:30:28.086 INFO:teuthology.misc.health.smithi001.stderr:election_strategy: 1 2024-03-25T18:30:28.086 INFO:teuthology.misc.health.smithi001.stderr:0: v1:172.21.15.1:6789/0 mon.noname-a 2024-03-25T18:30:28.086 INFO:teuthology.misc.health.smithi001.stderr:1: v1:172.21.15.115:6789/0 mon.noname-b 2024-03-25T18:30:28.086 INFO:teuthology.misc.health.smithi001.stderr:2: v1:172.21.15.1:6790/0 mon.noname-c 2024-03-25T18:30:28.086 INFO:teuthology.misc.health.smithi001.stderr: 2024-03-25T18:30:28.086 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:30:28.083+0000 7f4204633640 1 librados: starting msgr at 2024-03-25T18:30:28.086 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:30:28.083+0000 7f4204633640 1 librados: starting objecter 2024-03-25T18:30:28.086 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:30:28.083+0000 7f4204633640 1 Processor -- start 2024-03-25T18:30:28.086 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:30:28.083+0000 7f4204633640 1 -- start start 2024-03-25T18:30:28.086 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:30:28.083+0000 7f4204633640 1 librados: setting wanted keys 2024-03-25T18:30:28.086 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:30:28.083+0000 7f4204633640 1 librados: calling monclient init 2024-03-25T18:30:28.087 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:30:28.083+0000 7f4204633640 10 monclient: init 2024-03-25T18:30:28.087 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:30:28.084+0000 7f4204633640 10 monclient: _reopen_session rank -1 2024-03-25T18:30:28.088 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:30:28.084+0000 7f4204633640 10 monclient: _add_conns ranks=[1,0,2] 2024-03-25T18:30:28.088 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:30:28.084+0000 7f4204633640 10 monclient(hunting): picked mon.noname-b con 0x7f41fc0b4670 addr v1:172.21.15.115:6789/0 2024-03-25T18:30:28.088 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:30:28.084+0000 7f4204633640 10 monclient(hunting): picked mon.noname-a con 0x7f41fc0b4ff0 addr v1:172.21.15.1:6789/0 2024-03-25T18:30:28.088 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:30:28.084+0000 7f4204633640 10 monclient(hunting): picked mon.noname-c con 0x7f41fc13c480 addr v1:172.21.15.1:6790/0 2024-03-25T18:30:28.088 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:30:28.084+0000 7f4204633640 1 -- --> v1:172.21.15.1:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f41fc0b0470 con 0x7f41fc0b4ff0 2024-03-25T18:30:28.088 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:30:28.084+0000 7f4204633640 1 -- --> v1:172.21.15.115:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f41fc0b0750 con 0x7f41fc0b4670 2024-03-25T18:30:28.088 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:30:28.084+0000 7f4204633640 1 -- --> v1:172.21.15.1:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f41fc0aeb90 con 0x7f41fc13c480 2024-03-25T18:30:28.088 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:30:28.084+0000 7f4204633640 10 monclient(hunting): _renew_subs 2024-03-25T18:30:28.088 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:30:28.084+0000 7f4204633640 10 monclient(hunting): authenticate will time out at 7708.554998s 2024-03-25T18:30:28.103 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:30:28.101+0000 7f42023cf640 1 --1- >> v1:172.21.15.115:6789/0 conn(0x7f41fc0b4670 0x7f41fc0ae0f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.115:6789/0 says I am v1:172.21.15.1:34164/0 (socket says 172.21.15.1:34164) 2024-03-25T18:30:28.103 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:30:28.101+0000 7f42023cf640 1 -- 172.21.15.1:0/3835490513 learned_addr learned my addr 172.21.15.1:0/3835490513 (peer_addr_for_me v1:172.21.15.1:0/0) 2024-03-25T18:30:28.116 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:30:28.114+0000 7f41f2ffd640 10 client.?.objecter ms_handle_connect 0x7f41fc0b4670 2024-03-25T18:30:28.116 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:30:28.114+0000 7f41f2ffd640 10 client.?.objecter resend_mon_ops 2024-03-25T18:30:28.119 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:30:28.118+0000 7f41f2ffd640 10 client.?.objecter ms_handle_connect 0x7f41fc0b4ff0 2024-03-25T18:30:28.119 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:30:28.118+0000 7f41f2ffd640 10 client.?.objecter resend_mon_ops 2024-03-25T18:30:28.120 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:30:28.119+0000 7f41f2ffd640 10 client.?.objecter ms_handle_connect 0x7f41fc13c480 2024-03-25T18:30:28.120 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:30:28.119+0000 7f41f2ffd640 10 client.?.objecter resend_mon_ops 2024-03-25T18:30:28.127 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:30:28.125+0000 7f41f2ffd640 1 -- 172.21.15.1:0/3835490513 <== mon.1 v1:172.21.15.115:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1744669385 0 0) 0x7f41fc0b0750 con 0x7f41fc0b4670 2024-03-25T18:30:28.127 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:30:28.125+0000 7f41f2ffd640 10 monclient(hunting): _negotiate 2024-03-25T18:30:28.127 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:30:28.125+0000 7f41f2ffd640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:30:28.127 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:30:28.125+0000 7f41f2ffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:30:28.127 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:30:28.125+0000 7f41f2ffd640 10 monclient(hunting): my global_id is 4387 2024-03-25T18:30:28.127 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:30:28.125+0000 7f41f2ffd640 1 -- 172.21.15.1:0/3835490513 --> v1:172.21.15.115:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f41e0004eb0 con 0x7f41fc0b4670 2024-03-25T18:30:28.138 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:30:28.136+0000 7f41f2ffd640 1 -- 172.21.15.1:0/3835490513 <== mon.2 v1:172.21.15.1:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 876189138 0 0) 0x7f41fc0aeb90 con 0x7f41fc13c480 2024-03-25T18:30:28.138 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:30:28.136+0000 7f41f2ffd640 10 monclient(hunting): _negotiate 2024-03-25T18:30:28.138 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:30:28.136+0000 7f41f2ffd640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:30:28.138 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:30:28.136+0000 7f41f2ffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:30:28.138 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:30:28.136+0000 7f41f2ffd640 10 monclient(hunting): my global_id is 4391 2024-03-25T18:30:28.139 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:30:28.136+0000 7f41f2ffd640 1 -- 172.21.15.1:0/3835490513 --> v1:172.21.15.1:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f41fc0b0750 con 0x7f41fc13c480 2024-03-25T18:30:28.147 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:30:28.145+0000 7f41f2ffd640 1 -- 172.21.15.1:0/3835490513 <== mon.0 v1:172.21.15.1:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1279048657 0 0) 0x7f41fc0b0470 con 0x7f41fc0b4ff0 2024-03-25T18:30:28.147 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:30:28.145+0000 7f41f2ffd640 10 monclient(hunting): _negotiate 2024-03-25T18:30:28.147 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:30:28.145+0000 7f41f2ffd640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:30:28.147 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:30:28.145+0000 7f41f2ffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:30:28.147 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:30:28.145+0000 7f41f2ffd640 10 monclient(hunting): my global_id is 4383 2024-03-25T18:30:28.147 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:30:28.145+0000 7f41f2ffd640 1 -- 172.21.15.1:0/3835490513 --> v1:172.21.15.1:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f41fc0aeb90 con 0x7f41fc0b4ff0 2024-03-25T18:30:28.162 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:30:28.160+0000 7f41f2ffd640 1 -- 172.21.15.1:0/3835490513 <== mon.1 v1:172.21.15.115:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3084034911 0 0) 0x7f41e0004eb0 con 0x7f41fc0b4670 2024-03-25T18:30:28.162 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:30:28.160+0000 7f41f2ffd640 1 -- 172.21.15.1:0/3835490513 --> v1:172.21.15.115:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f41fc0b0470 con 0x7f41fc0b4670 2024-03-25T18:30:28.162 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:30:28.160+0000 7f41f2ffd640 1 -- 172.21.15.1:0/3835490513 <== mon.1 v1:172.21.15.115:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1586965025 0 0) 0x7f41ec0019d0 con 0x7f41fc0b4670 2024-03-25T18:30:28.162 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:30:28.160+0000 7f41f2ffd640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:30:28.162 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:30:28.160+0000 7f41f2ffd640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-03-25T18:30:28.162 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:30:28.160+0000 7f41f2ffd640 10 monclient(hunting): dump: 2024-03-25T18:30:28.163 INFO:teuthology.misc.health.smithi001.stderr:epoch 1 2024-03-25T18:30:28.163 INFO:teuthology.misc.health.smithi001.stderr:fsid c8ea72cf-9e99-4383-9b52-6c23d365fdfc 2024-03-25T18:30:28.163 INFO:teuthology.misc.health.smithi001.stderr:last_changed 2024-03-25T18:25:50.510519+0000 2024-03-25T18:30:28.163 INFO:teuthology.misc.health.smithi001.stderr:created 2024-03-25T18:25:50.510519+0000 2024-03-25T18:30:28.163 INFO:teuthology.misc.health.smithi001.stderr:min_mon_release 19 (squid) 2024-03-25T18:30:28.163 INFO:teuthology.misc.health.smithi001.stderr:election_strategy: 3 2024-03-25T18:30:28.163 INFO:teuthology.misc.health.smithi001.stderr:0: v1:172.21.15.1:6789/0 mon.a 2024-03-25T18:30:28.163 INFO:teuthology.misc.health.smithi001.stderr:1: v1:172.21.15.115:6789/0 mon.b 2024-03-25T18:30:28.163 INFO:teuthology.misc.health.smithi001.stderr:2: v1:172.21.15.1:6790/0 mon.c 2024-03-25T18:30:28.163 INFO:teuthology.misc.health.smithi001.stderr: 2024-03-25T18:30:28.163 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:30:28.160+0000 7f41f2ffd640 10 monclient(hunting): _finish_auth 0 2024-03-25T18:30:28.176 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:30:28.174+0000 7f41f2ffd640 1 -- 172.21.15.1:0/3835490513 <== mon.2 v1:172.21.15.1:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 34817903 0 0) 0x7f41fc0b0750 con 0x7f41fc13c480 2024-03-25T18:30:28.176 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:30:28.174+0000 7f41f2ffd640 1 -- 172.21.15.1:0/3835490513 --> v1:172.21.15.1:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f41e0004eb0 con 0x7f41fc13c480 2024-03-25T18:30:28.176 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:30:28.174+0000 7f41f2ffd640 1 -- 172.21.15.1:0/3835490513 <== mon.2 v1:172.21.15.1:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1586965025 0 0) 0x7f41f4001b10 con 0x7f41fc13c480 2024-03-25T18:30:28.176 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:30:28.174+0000 7f41f2ffd640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:30:28.176 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:30:28.174+0000 7f41f2ffd640 10 monclient(hunting): got monmap 1 from mon.c (according to old e1) 2024-03-25T18:30:28.176 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:30:28.174+0000 7f41f2ffd640 10 monclient(hunting): dump: 2024-03-25T18:30:28.176 INFO:teuthology.misc.health.smithi001.stderr:epoch 1 2024-03-25T18:30:28.176 INFO:teuthology.misc.health.smithi001.stderr:fsid c8ea72cf-9e99-4383-9b52-6c23d365fdfc 2024-03-25T18:30:28.176 INFO:teuthology.misc.health.smithi001.stderr:last_changed 2024-03-25T18:25:50.510519+0000 2024-03-25T18:30:28.176 INFO:teuthology.misc.health.smithi001.stderr:created 2024-03-25T18:25:50.510519+0000 2024-03-25T18:30:28.177 INFO:teuthology.misc.health.smithi001.stderr:min_mon_release 19 (squid) 2024-03-25T18:30:28.177 INFO:teuthology.misc.health.smithi001.stderr:election_strategy: 3 2024-03-25T18:30:28.177 INFO:teuthology.misc.health.smithi001.stderr:0: v1:172.21.15.1:6789/0 mon.a 2024-03-25T18:30:28.177 INFO:teuthology.misc.health.smithi001.stderr:1: v1:172.21.15.115:6789/0 mon.b 2024-03-25T18:30:28.177 INFO:teuthology.misc.health.smithi001.stderr:2: v1:172.21.15.1:6790/0 mon.c 2024-03-25T18:30:28.177 INFO:teuthology.misc.health.smithi001.stderr: 2024-03-25T18:30:28.178 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:30:28.175+0000 7f41f2ffd640 1 -- 172.21.15.1:0/3835490513 <== mon.1 v1:172.21.15.115:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 4190927504 0 0) 0x7f41fc0b0470 con 0x7f41fc0b4670 2024-03-25T18:30:28.178 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:30:28.175+0000 7f41f2ffd640 1 -- 172.21.15.1:0/3835490513 >> v1:172.21.15.1:6790/0 conn(0x7f41fc13c480 legacy=0x7f41fc1e2380 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:30:28.178 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:30:28.175+0000 7f41f2ffd640 1 -- 172.21.15.1:0/3835490513 >> v1:172.21.15.1:6789/0 conn(0x7f41fc0b4ff0 legacy=0x7f41fc0b0be0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:30:28.178 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:30:28.175+0000 7f41f2ffd640 10 monclient: _finish_hunting 0 2024-03-25T18:30:28.178 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:30:28.175+0000 7f41f2ffd640 1 monclient: found mon.b 2024-03-25T18:30:28.178 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:30:28.175+0000 7f41f2ffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:30:28.178 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:30:28.175+0000 7f41f2ffd640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.115:6789/0 2024-03-25T18:30:28.179 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:30:28.175+0000 7f41f2ffd640 1 -- 172.21.15.1:0/3835490513 --> v1:172.21.15.115:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f41fc0ae800 con 0x7f41fc0b4670 2024-03-25T18:30:28.179 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:30:28.176+0000 7f41f2ffd640 10 monclient: _finish_auth 0 2024-03-25T18:30:28.179 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:30:28.176+0000 7f41f2ffd640 10 monclient: _check_auth_tickets 2024-03-25T18:30:28.179 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:30:28.176+0000 7f41f2ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:30:28.179 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:30:28.176+0000 7f4204633640 5 monclient: authenticate success, global_id 4387 2024-03-25T18:30:28.179 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:30:28.176+0000 7f4204633640 10 monclient: _renew_subs 2024-03-25T18:30:28.179 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:30:28.176+0000 7f4204633640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.115:6789/0 2024-03-25T18:30:28.179 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:30:28.176+0000 7f4204633640 1 -- 172.21.15.1:0/3835490513 --> v1:172.21.15.115:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f41fc0b12f0 con 0x7f41fc0b4670 2024-03-25T18:30:28.179 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:30:28.176+0000 7f4204633640 10 client.4387.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T18:30:28.179 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:30:28.176+0000 7f4204633640 10 monclient: _renew_subs 2024-03-25T18:30:28.179 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:30:28.176+0000 7f4204633640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.115:6789/0 2024-03-25T18:30:28.179 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:30:28.176+0000 7f4204633640 1 -- 172.21.15.1:0/3835490513 --> v1:172.21.15.115:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f41fc0af600 con 0x7f41fc0b4670 2024-03-25T18:30:28.179 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:30:28.176+0000 7f4204633640 1 librados: init done 2024-03-25T18:30:28.179 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:30:28.176+0000 7f4204633640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T18:30:28.179 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:30:28.176+0000 7f4204633640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:30:28.179 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:30:28.176+0000 7f4204633640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.115:6789/0 2024-03-25T18:30:28.180 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:30:28.176+0000 7f4204633640 1 -- 172.21.15.1:0/3835490513 --> v1:172.21.15.115:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f41fc0aed20 con 0x7f41fc0b4670 2024-03-25T18:30:28.475 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:30:28.473+0000 7f41f2ffd640 1 -- 172.21.15.1:0/3835490513 <== mon.1 v1:172.21.15.115:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f41ec003610 con 0x7f41fc0b4670 2024-03-25T18:30:28.475 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:30:28.473+0000 7f41f2ffd640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:30:28.475 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:30:28.473+0000 7f41f2ffd640 1 -- 172.21.15.1:0/3835490513 <== mon.1 v1:172.21.15.115:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1586965025 0 0) 0x7f41ec0039a0 con 0x7f41fc0b4670 2024-03-25T18:30:28.475 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:30:28.473+0000 7f41f2ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:30:28.475 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:30:28.473+0000 7f41f2ffd640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-25T18:30:28.475 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:30:28.473+0000 7f41f2ffd640 10 monclient: dump: 2024-03-25T18:30:28.475 INFO:teuthology.misc.health.smithi001.stderr:epoch 1 2024-03-25T18:30:28.476 INFO:teuthology.misc.health.smithi001.stderr:fsid c8ea72cf-9e99-4383-9b52-6c23d365fdfc 2024-03-25T18:30:28.476 INFO:teuthology.misc.health.smithi001.stderr:last_changed 2024-03-25T18:25:50.510519+0000 2024-03-25T18:30:28.476 INFO:teuthology.misc.health.smithi001.stderr:created 2024-03-25T18:25:50.510519+0000 2024-03-25T18:30:28.476 INFO:teuthology.misc.health.smithi001.stderr:min_mon_release 19 (squid) 2024-03-25T18:30:28.476 INFO:teuthology.misc.health.smithi001.stderr:election_strategy: 3 2024-03-25T18:30:28.476 INFO:teuthology.misc.health.smithi001.stderr:0: v1:172.21.15.1:6789/0 mon.a 2024-03-25T18:30:28.476 INFO:teuthology.misc.health.smithi001.stderr:1: v1:172.21.15.115:6789/0 mon.b 2024-03-25T18:30:28.476 INFO:teuthology.misc.health.smithi001.stderr:2: v1:172.21.15.1:6790/0 mon.c 2024-03-25T18:30:28.476 INFO:teuthology.misc.health.smithi001.stderr: 2024-03-25T18:30:28.476 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:30:28.473+0000 7f4200bcc640 4 set_mon_vals no callback set 2024-03-25T18:30:28.477 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:30:28.474+0000 7f41f2ffd640 1 -- 172.21.15.1:0/3835490513 <== mon.1 v1:172.21.15.115:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 21787975 0 0) 0x7f41ec003bc0 con 0x7f41fc0b4670 2024-03-25T18:30:28.477 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:30:28.474+0000 7f4201bce640 10 monclient: get_auth_request con 0x7f41e006df70 auth_method 0 2024-03-25T18:30:28.477 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:30:28.475+0000 7f41f2ffd640 10 client.4387.objecter ms_handle_connect 0x7f41e006df70 2024-03-25T18:30:28.480 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:30:28.478+0000 7f41f2ffd640 1 -- 172.21.15.1:0/3835490513 <== mon.1 v1:172.21.15.115:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 996254230 0 0) 0x7f41ec0547f0 con 0x7f41fc0b4670 2024-03-25T18:30:28.480 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:30:28.478+0000 7f41f2ffd640 10 client.4387.objecter ms_dispatch 0x7f41fc000b90 osd_map(10..10 src has 1..10) v4 2024-03-25T18:30:28.480 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:30:28.478+0000 7f41f2ffd640 3 client.4387.objecter handle_osd_map got epochs [10,10] > 0 2024-03-25T18:30:28.480 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:30:28.478+0000 7f41f2ffd640 3 client.4387.objecter handle_osd_map decoding full epoch 10 2024-03-25T18:30:28.480 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:30:28.478+0000 7f41f2ffd640 20 client.4387.objecter dump_active .. 0 homeless 2024-03-25T18:30:28.481 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:30:28.479+0000 7f41f2ffd640 1 -- 172.21.15.1:0/3835490513 <== mon.1 v1:172.21.15.115:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f41ec050300 con 0x7f41fc0b4670 2024-03-25T18:30:28.481 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:30:28.479+0000 7f41f2ffd640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:30:28.481 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:30:28.479+0000 7f41f2ffd640 10 monclient: _finish_command 1 = system:0 2024-03-25T18:30:28.744 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:30:28.742+0000 7f4204633640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:30:28.744 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:30:28.742+0000 7f4204633640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:30:28.744 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:30:28.742+0000 7f4204633640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.115:6789/0 2024-03-25T18:30:28.744 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:30:28.742+0000 7f4204633640 1 -- 172.21.15.1:0/3835490513 --> v1:172.21.15.115:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f41fc0b1530 con 0x7f41fc0b4670 2024-03-25T18:30:28.894 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:30:28.892+0000 7f41f2ffd640 1 -- 172.21.15.1:0/3835490513 <== mon.1 v1:172.21.15.115:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 4092213348) 0x7f41ec052230 con 0x7f41fc0b4670 2024-03-25T18:30:28.894 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:30:28.892+0000 7f41f2ffd640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:30:28.894 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:30:28.892+0000 7f41f2ffd640 10 monclient: _finish_command 2 = system:0 2024-03-25T18:30:28.894 INFO:teuthology.misc.health.smithi001.stdout: 2024-03-25T18:30:28.894 INFO:teuthology.misc.health.smithi001.stdout:{"epoch":10,"fsid":"c8ea72cf-9e99-4383-9b52-6c23d365fdfc","created":"2024-03-25T18:26:38.856638+0000","modified":"2024-03-25T18:26:44.224208+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T18:26:44.224208+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"b9f34922-3a84-49be-8426-807c1eceaefc","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"42bf306f-55e3-4634-9874-8b4b73ac6644","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"292cb23a-18a9-4207-a7cc-4d5727feaec0","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"45e0e9ab-898d-4673-9590-b4117bc5e84a","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"831a5a09-73ab-4a61-9872-088e76bdbc66","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"88cd903d-3b55-4840-9396-6e3cfe005282","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"89db7355-bc9d-46f4-8ab0-e8d9e744e026","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"cac47bee-0914-4f08-896f-348838882866","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T18:30:28.896 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:30:28.894+0000 7f4204633640 10 librados: watch_flush enter 2024-03-25T18:30:28.896 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:30:28.894+0000 7f4204633640 10 librados: watch_flush exit 2024-03-25T18:30:28.896 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:30:28.894+0000 7f4204633640 20 client.4387.objecter shutdown clearing up homeless session... 2024-03-25T18:30:28.896 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:30:28.894+0000 7f4204633640 10 client.4387.objecter successfully canceled tick 2024-03-25T18:30:28.896 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:30:28.894+0000 7f4204633640 1 -- 172.21.15.1:0/3835490513 >> v1:172.21.15.115:6800/4237379176 conn(0x7f41e006df70 legacy=0x7f41e0070430 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:30:28.897 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:30:28.894+0000 7f4204633640 10 monclient: shutdown 2024-03-25T18:30:28.897 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:30:28.894+0000 7f4204633640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:30:28.897 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:30:28.894+0000 7f4204633640 1 -- 172.21.15.1:0/3835490513 >> v1:172.21.15.115:6789/0 conn(0x7f41fc0b4670 legacy=0x7f41fc0ae0f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:30:28.897 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:30:28.894+0000 7f4204633640 1 -- 172.21.15.1:0/3835490513 shutdown_connections 2024-03-25T18:30:28.897 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:30:28.894+0000 7f4204633640 1 -- 172.21.15.1:0/3835490513 >> 172.21.15.1:0/3835490513 conn(0x7f41fc0617a0 msgr2=0x7f41fc145420 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:30:28.897 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:30:28.894+0000 7f4204633640 1 -- 172.21.15.1:0/3835490513 shutdown_connections 2024-03-25T18:30:28.897 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:30:28.894+0000 7f4204633640 1 -- 172.21.15.1:0/3835490513 wait complete. 2024-03-25T18:30:28.897 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:30:28.894+0000 7f4204633640 1 librados: shutdown 2024-03-25T18:30:28.908 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T18:30:35.715 DEBUG:teuthology.orchestra.run.smithi001:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T18:30:35.814 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:30:35.812+0000 7f30a89e8640 10 monclient: get_monmap_and_config 2024-03-25T18:30:35.814 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:30:35.812+0000 7f30a89e8640 10 monclient: build_initial_monmap 2024-03-25T18:30:35.814 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:30:35.812+0000 7f30a89e8640 10 monclient: monmap: 2024-03-25T18:30:35.814 INFO:teuthology.misc.health.smithi001.stderr:epoch 0 2024-03-25T18:30:35.814 INFO:teuthology.misc.health.smithi001.stderr:fsid c8ea72cf-9e99-4383-9b52-6c23d365fdfc 2024-03-25T18:30:35.814 INFO:teuthology.misc.health.smithi001.stderr:last_changed 2024-03-25T18:30:35.814049+0000 2024-03-25T18:30:35.815 INFO:teuthology.misc.health.smithi001.stderr:created 2024-03-25T18:30:35.814049+0000 2024-03-25T18:30:35.815 INFO:teuthology.misc.health.smithi001.stderr:min_mon_release 0 (unknown) 2024-03-25T18:30:35.815 INFO:teuthology.misc.health.smithi001.stderr:election_strategy: 3 2024-03-25T18:30:35.815 INFO:teuthology.misc.health.smithi001.stderr:0: v1:172.21.15.1:6789/0 mon.noname-a 2024-03-25T18:30:35.815 INFO:teuthology.misc.health.smithi001.stderr:1: v1:172.21.15.115:6789/0 mon.noname-c 2024-03-25T18:30:35.815 INFO:teuthology.misc.health.smithi001.stderr:2: v1:172.21.15.1:6790/0 mon.noname-b 2024-03-25T18:30:35.815 INFO:teuthology.misc.health.smithi001.stderr: 2024-03-25T18:30:35.816 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:30:35.813+0000 7f30a89e8640 1 Processor -- start 2024-03-25T18:30:35.816 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:30:35.813+0000 7f30a89e8640 1 -- start start 2024-03-25T18:30:35.816 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:30:35.813+0000 7f30a89e8640 10 monclient: init 2024-03-25T18:30:35.816 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:30:35.813+0000 7f30a89e8640 10 monclient: _reopen_session rank -1 2024-03-25T18:30:35.817 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:30:35.813+0000 7f30a89e8640 10 monclient: _add_conns ranks=[2,0,1] 2024-03-25T18:30:35.817 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:30:35.813+0000 7f30a89e8640 10 monclient(hunting): picked mon.noname-b con 0x7f30a40b4e70 addr v1:172.21.15.1:6790/0 2024-03-25T18:30:35.817 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:30:35.813+0000 7f30a89e8640 10 monclient(hunting): picked mon.noname-a con 0x7f30a40b44b0 addr v1:172.21.15.1:6789/0 2024-03-25T18:30:35.817 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:30:35.814+0000 7f30a89e8640 10 monclient(hunting): picked mon.noname-c con 0x7f30a40b1b00 addr v1:172.21.15.115:6789/0 2024-03-25T18:30:35.817 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:30:35.814+0000 7f30a89e8640 1 -- --> v1:172.21.15.1:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f30a40a96f0 con 0x7f30a40b44b0 2024-03-25T18:30:35.817 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:30:35.814+0000 7f30a89e8640 1 -- --> v1:172.21.15.115:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f30a40ab0e0 con 0x7f30a40b1b00 2024-03-25T18:30:35.817 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:30:35.814+0000 7f30a89e8640 1 -- --> v1:172.21.15.1:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f30a4145f70 con 0x7f30a40b4e70 2024-03-25T18:30:35.817 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:30:35.814+0000 7f30a89e8640 10 monclient(hunting): _renew_subs 2024-03-25T18:30:35.817 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:30:35.814+0000 7f30a89e8640 10 monclient(hunting): authenticate will time out at 7716.284840s 2024-03-25T18:30:35.830 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:30:35.829+0000 7f30a1d9b640 1 --1- >> v1:172.21.15.1:6789/0 conn(0x7f30a40b44b0 0x7f30a4145860 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.1:6789/0 says I am v1:172.21.15.1:43272/0 (socket says 172.21.15.1:43272) 2024-03-25T18:30:35.830 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:30:35.829+0000 7f30a1d9b640 1 -- 172.21.15.1:0/19105090 learned_addr learned my addr 172.21.15.1:0/19105090 (peer_addr_for_me v1:172.21.15.1:0/0) 2024-03-25T18:30:35.840 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:30:35.839+0000 7f30a159a640 1 -- 172.21.15.1:0/19105090 <== mon.0 v1:172.21.15.1:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2848107000 0 0) 0x7f30a40a96f0 con 0x7f30a40b44b0 2024-03-25T18:30:35.840 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:30:35.839+0000 7f30a159a640 10 monclient(hunting): _negotiate 2024-03-25T18:30:35.841 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:30:35.839+0000 7f30a159a640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:30:35.841 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:30:35.839+0000 7f30a159a640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:30:35.841 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:30:35.839+0000 7f30a159a640 10 monclient(hunting): my global_id is 4386 2024-03-25T18:30:35.841 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:30:35.839+0000 7f30a159a640 1 -- 172.21.15.1:0/19105090 --> v1:172.21.15.1:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f308c003520 con 0x7f30a40b44b0 2024-03-25T18:30:35.843 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:30:35.841+0000 7f30a159a640 1 -- 172.21.15.1:0/19105090 <== mon.1 v1:172.21.15.115:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2917813768 0 0) 0x7f30a40ab0e0 con 0x7f30a40b1b00 2024-03-25T18:30:35.843 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:30:35.841+0000 7f30a159a640 10 monclient(hunting): _negotiate 2024-03-25T18:30:35.843 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:30:35.841+0000 7f30a159a640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:30:35.843 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:30:35.841+0000 7f30a159a640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:30:35.843 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:30:35.841+0000 7f30a159a640 10 monclient(hunting): my global_id is 4390 2024-03-25T18:30:35.843 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:30:35.841+0000 7f30a159a640 1 -- 172.21.15.1:0/19105090 --> v1:172.21.15.115:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f30a40a96f0 con 0x7f30a40b1b00 2024-03-25T18:30:35.848 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:30:35.845+0000 7f30a159a640 1 -- 172.21.15.1:0/19105090 <== mon.0 v1:172.21.15.1:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2364861235 0 0) 0x7f308c003520 con 0x7f30a40b44b0 2024-03-25T18:30:35.848 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:30:35.845+0000 7f30a159a640 1 -- 172.21.15.1:0/19105090 >> v1:172.21.15.1:6790/0 conn(0x7f30a40b4e70 legacy=0x7f30a413f940 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:30:35.848 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:30:35.845+0000 7f30a159a640 1 -- 172.21.15.1:0/19105090 >> v1:172.21.15.115:6789/0 conn(0x7f30a40b1b00 legacy=0x7f30a414dc00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:30:35.848 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:30:35.845+0000 7f30a159a640 10 monclient: _finish_hunting 0 2024-03-25T18:30:35.848 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:30:35.845+0000 7f30a159a640 1 monclient: found mon.noname-a 2024-03-25T18:30:35.848 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:30:35.845+0000 7f30a159a640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:30:35.848 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:30:35.845+0000 7f30a159a640 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.1:6789/0 2024-03-25T18:30:35.848 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:30:35.845+0000 7f30a159a640 1 -- 172.21.15.1:0/19105090 --> v1:172.21.15.1:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f30a40a9b80 con 0x7f30a40b44b0 2024-03-25T18:30:35.848 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:30:35.845+0000 7f30a159a640 10 monclient: _finish_auth 0 2024-03-25T18:30:35.848 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:30:35.845+0000 7f30a159a640 10 monclient: _check_auth_tickets 2024-03-25T18:30:35.848 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:30:35.845+0000 7f30a159a640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:30:35.848 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:30:35.845+0000 7f30a159a640 1 -- 172.21.15.1:0/19105090 <== mon.0 v1:172.21.15.1:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1586965025 0 0) 0x7f3098001690 con 0x7f30a40b44b0 2024-03-25T18:30:35.849 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:30:35.845+0000 7f30a159a640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:30:35.849 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:30:35.845+0000 7f30a159a640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-03-25T18:30:35.849 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:30:35.845+0000 7f30a159a640 10 monclient: dump: 2024-03-25T18:30:35.849 INFO:teuthology.misc.health.smithi001.stderr:epoch 1 2024-03-25T18:30:35.849 INFO:teuthology.misc.health.smithi001.stderr:fsid c8ea72cf-9e99-4383-9b52-6c23d365fdfc 2024-03-25T18:30:35.849 INFO:teuthology.misc.health.smithi001.stderr:last_changed 2024-03-25T18:25:50.510519+0000 2024-03-25T18:30:35.849 INFO:teuthology.misc.health.smithi001.stderr:created 2024-03-25T18:25:50.510519+0000 2024-03-25T18:30:35.849 INFO:teuthology.misc.health.smithi001.stderr:min_mon_release 19 (squid) 2024-03-25T18:30:35.849 INFO:teuthology.misc.health.smithi001.stderr:election_strategy: 3 2024-03-25T18:30:35.849 INFO:teuthology.misc.health.smithi001.stderr:0: v1:172.21.15.1:6789/0 mon.a 2024-03-25T18:30:35.849 INFO:teuthology.misc.health.smithi001.stderr:1: v1:172.21.15.115:6789/0 mon.b 2024-03-25T18:30:35.849 INFO:teuthology.misc.health.smithi001.stderr:2: v1:172.21.15.1:6790/0 mon.c 2024-03-25T18:30:35.849 INFO:teuthology.misc.health.smithi001.stderr: 2024-03-25T18:30:35.849 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:30:35.845+0000 7f30a89e8640 5 monclient: authenticate success, global_id 4386 2024-03-25T18:30:35.849 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:30:35.845+0000 7f30a89e8640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T18:30:35.854 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:30:35.853+0000 7f30a159a640 1 -- 172.21.15.1:0/19105090 <== mon.0 v1:172.21.15.1:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f30980017f0 con 0x7f30a40b44b0 2024-03-25T18:30:35.855 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:30:35.853+0000 7f30a159a640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:30:35.855 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:30:35.853+0000 7f30a159a640 1 -- 172.21.15.1:0/19105090 <== mon.0 v1:172.21.15.1:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1586965025 0 0) 0x7f3098001c00 con 0x7f30a40b44b0 2024-03-25T18:30:35.855 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:30:35.853+0000 7f30a159a640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:30:35.855 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:30:35.853+0000 7f30a159a640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-25T18:30:35.855 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:30:35.853+0000 7f30a159a640 10 monclient: dump: 2024-03-25T18:30:35.855 INFO:teuthology.misc.health.smithi001.stderr:epoch 1 2024-03-25T18:30:35.855 INFO:teuthology.misc.health.smithi001.stderr:fsid c8ea72cf-9e99-4383-9b52-6c23d365fdfc 2024-03-25T18:30:35.855 INFO:teuthology.misc.health.smithi001.stderr:last_changed 2024-03-25T18:25:50.510519+0000 2024-03-25T18:30:35.855 INFO:teuthology.misc.health.smithi001.stderr:created 2024-03-25T18:25:50.510519+0000 2024-03-25T18:30:35.855 INFO:teuthology.misc.health.smithi001.stderr:min_mon_release 19 (squid) 2024-03-25T18:30:35.855 INFO:teuthology.misc.health.smithi001.stderr:election_strategy: 3 2024-03-25T18:30:35.855 INFO:teuthology.misc.health.smithi001.stderr:0: v1:172.21.15.1:6789/0 mon.a 2024-03-25T18:30:35.855 INFO:teuthology.misc.health.smithi001.stderr:1: v1:172.21.15.115:6789/0 mon.b 2024-03-25T18:30:35.855 INFO:teuthology.misc.health.smithi001.stderr:2: v1:172.21.15.1:6790/0 mon.c 2024-03-25T18:30:35.856 INFO:teuthology.misc.health.smithi001.stderr: 2024-03-25T18:30:35.856 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:30:35.853+0000 7f30a89e8640 10 monclient: get_monmap_and_config success 2024-03-25T18:30:35.856 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:30:35.853+0000 7f30a89e8640 4 set_mon_vals no callback set 2024-03-25T18:30:35.856 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:30:35.854+0000 7f30a89e8640 10 monclient: shutdown 2024-03-25T18:30:35.857 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:30:35.854+0000 7f30a89e8640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:30:35.857 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:30:35.854+0000 7f30a89e8640 1 -- 172.21.15.1:0/19105090 >> v1:172.21.15.1:6789/0 conn(0x7f30a40b44b0 legacy=0x7f30a4145860 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:30:35.857 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:30:35.854+0000 7f30a89e8640 1 -- 172.21.15.1:0/19105090 shutdown_connections 2024-03-25T18:30:35.857 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:30:35.854+0000 7f30a89e8640 1 -- 172.21.15.1:0/19105090 >> 172.21.15.1:0/19105090 conn(0x7f30a40617a0 msgr2=0x7f30a4061bb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:30:35.857 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:30:35.854+0000 7f30a89e8640 1 -- 172.21.15.1:0/19105090 shutdown_connections 2024-03-25T18:30:35.857 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:30:35.854+0000 7f30a89e8640 1 -- 172.21.15.1:0/19105090 wait complete. 2024-03-25T18:30:35.857 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:30:35.855+0000 7f30a89e8640 10 monclient: build_initial_monmap 2024-03-25T18:30:35.857 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:30:35.855+0000 7f30a89e8640 10 monclient: monmap: 2024-03-25T18:30:35.857 INFO:teuthology.misc.health.smithi001.stderr:epoch 0 2024-03-25T18:30:35.857 INFO:teuthology.misc.health.smithi001.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T18:30:35.857 INFO:teuthology.misc.health.smithi001.stderr:last_changed 0.000000 2024-03-25T18:30:35.857 INFO:teuthology.misc.health.smithi001.stderr:created 0.000000 2024-03-25T18:30:35.857 INFO:teuthology.misc.health.smithi001.stderr:min_mon_release 0 (unknown) 2024-03-25T18:30:35.857 INFO:teuthology.misc.health.smithi001.stderr:election_strategy: 1 2024-03-25T18:30:35.857 INFO:teuthology.misc.health.smithi001.stderr:0: v1:172.21.15.1:6789/0 mon.noname-a 2024-03-25T18:30:35.858 INFO:teuthology.misc.health.smithi001.stderr:1: v1:172.21.15.115:6789/0 mon.noname-b 2024-03-25T18:30:35.858 INFO:teuthology.misc.health.smithi001.stderr:2: v1:172.21.15.1:6790/0 mon.noname-c 2024-03-25T18:30:35.858 INFO:teuthology.misc.health.smithi001.stderr: 2024-03-25T18:30:35.859 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:30:35.855+0000 7f30a89e8640 1 librados: starting msgr at 2024-03-25T18:30:35.859 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:30:35.855+0000 7f30a89e8640 1 librados: starting objecter 2024-03-25T18:30:35.859 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:30:35.855+0000 7f30a89e8640 1 Processor -- start 2024-03-25T18:30:35.859 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:30:35.856+0000 7f30a89e8640 1 -- start start 2024-03-25T18:30:35.859 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:30:35.856+0000 7f30a89e8640 1 librados: setting wanted keys 2024-03-25T18:30:35.859 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:30:35.856+0000 7f30a89e8640 1 librados: calling monclient init 2024-03-25T18:30:35.859 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:30:35.856+0000 7f30a89e8640 10 monclient: init 2024-03-25T18:30:35.860 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:30:35.856+0000 7f30a89e8640 10 monclient: _reopen_session rank -1 2024-03-25T18:30:35.860 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:30:35.856+0000 7f30a89e8640 10 monclient: _add_conns ranks=[2,0,1] 2024-03-25T18:30:35.860 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:30:35.856+0000 7f30a89e8640 10 monclient(hunting): picked mon.noname-c con 0x7f30a40b1b00 addr v1:172.21.15.1:6790/0 2024-03-25T18:30:35.860 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:30:35.856+0000 7f30a89e8640 10 monclient(hunting): picked mon.noname-a con 0x7f30a40b44b0 addr v1:172.21.15.1:6789/0 2024-03-25T18:30:35.860 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:30:35.856+0000 7f30a89e8640 10 monclient(hunting): picked mon.noname-b con 0x7f30a40b4e70 addr v1:172.21.15.115:6789/0 2024-03-25T18:30:35.860 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:30:35.856+0000 7f30a89e8640 1 -- --> v1:172.21.15.1:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f30a41333c0 con 0x7f30a40b44b0 2024-03-25T18:30:35.860 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:30:35.856+0000 7f30a89e8640 1 -- --> v1:172.21.15.115:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f30a41dbb80 con 0x7f30a40b4e70 2024-03-25T18:30:35.860 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:30:35.856+0000 7f30a89e8640 1 -- --> v1:172.21.15.1:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f30a413c2b0 con 0x7f30a40b1b00 2024-03-25T18:30:35.860 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:30:35.856+0000 7f30a89e8640 10 monclient(hunting): _renew_subs 2024-03-25T18:30:35.860 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:30:35.856+0000 7f30a89e8640 10 monclient(hunting): authenticate will time out at 7716.327253s 2024-03-25T18:30:35.876 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:30:35.874+0000 7f30a2d9d640 1 --1- >> v1:172.21.15.115:6789/0 conn(0x7f30a40b4e70 0x7f30a41449c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.115:6789/0 says I am v1:172.21.15.1:60294/0 (socket says 172.21.15.1:60294) 2024-03-25T18:30:35.876 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:30:35.875+0000 7f30a2d9d640 1 -- 172.21.15.1:0/373242256 learned_addr learned my addr 172.21.15.1:0/373242256 (peer_addr_for_me v1:172.21.15.1:0/0) 2024-03-25T18:30:35.882 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:30:35.880+0000 7f3092ffd640 10 client.?.objecter ms_handle_connect 0x7f30a40b4e70 2024-03-25T18:30:35.882 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:30:35.880+0000 7f3092ffd640 10 client.?.objecter resend_mon_ops 2024-03-25T18:30:35.882 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:30:35.881+0000 7f3092ffd640 10 client.?.objecter ms_handle_connect 0x7f30a40b1b00 2024-03-25T18:30:35.883 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:30:35.881+0000 7f3092ffd640 10 client.?.objecter resend_mon_ops 2024-03-25T18:30:35.883 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:30:35.881+0000 7f3092ffd640 10 client.?.objecter ms_handle_connect 0x7f30a40b44b0 2024-03-25T18:30:35.883 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:30:35.881+0000 7f3092ffd640 10 client.?.objecter resend_mon_ops 2024-03-25T18:30:35.892 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:30:35.890+0000 7f3092ffd640 1 -- 172.21.15.1:0/373242256 <== mon.1 v1:172.21.15.115:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1422643620 0 0) 0x7f30a41dbb80 con 0x7f30a40b4e70 2024-03-25T18:30:35.892 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:30:35.890+0000 7f3092ffd640 10 monclient(hunting): _negotiate 2024-03-25T18:30:35.892 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:30:35.890+0000 7f3092ffd640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:30:35.892 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:30:35.890+0000 7f3092ffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:30:35.892 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:30:35.890+0000 7f3092ffd640 10 monclient(hunting): my global_id is 4393 2024-03-25T18:30:35.892 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:30:35.890+0000 7f3092ffd640 1 -- 172.21.15.1:0/373242256 --> v1:172.21.15.115:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f3084004eb0 con 0x7f30a40b4e70 2024-03-25T18:30:35.894 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:30:35.893+0000 7f3092ffd640 1 -- 172.21.15.1:0/373242256 <== mon.2 v1:172.21.15.1:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2415673958 0 0) 0x7f30a413c2b0 con 0x7f30a40b1b00 2024-03-25T18:30:35.894 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:30:35.893+0000 7f3092ffd640 10 monclient(hunting): _negotiate 2024-03-25T18:30:35.894 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:30:35.893+0000 7f3092ffd640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:30:35.894 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:30:35.893+0000 7f3092ffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:30:35.895 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:30:35.893+0000 7f3092ffd640 10 monclient(hunting): my global_id is 4397 2024-03-25T18:30:35.895 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:30:35.893+0000 7f3092ffd640 1 -- 172.21.15.1:0/373242256 --> v1:172.21.15.1:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f30a41dbb80 con 0x7f30a40b1b00 2024-03-25T18:30:35.900 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:30:35.898+0000 7f3092ffd640 1 -- 172.21.15.1:0/373242256 <== mon.0 v1:172.21.15.1:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3770162849 0 0) 0x7f30a41333c0 con 0x7f30a40b44b0 2024-03-25T18:30:35.900 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:30:35.898+0000 7f3092ffd640 10 monclient(hunting): _negotiate 2024-03-25T18:30:35.900 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:30:35.898+0000 7f3092ffd640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:30:35.900 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:30:35.898+0000 7f3092ffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:30:35.900 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:30:35.898+0000 7f3092ffd640 10 monclient(hunting): my global_id is 4389 2024-03-25T18:30:35.900 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:30:35.898+0000 7f3092ffd640 1 -- 172.21.15.1:0/373242256 --> v1:172.21.15.1:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f30a413c2b0 con 0x7f30a40b44b0 2024-03-25T18:30:35.905 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:30:35.903+0000 7f3092ffd640 1 -- 172.21.15.1:0/373242256 <== mon.1 v1:172.21.15.115:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 141718656 0 0) 0x7f3084004eb0 con 0x7f30a40b4e70 2024-03-25T18:30:35.906 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:30:35.903+0000 7f3092ffd640 1 -- 172.21.15.1:0/373242256 --> v1:172.21.15.115:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f30a41333c0 con 0x7f30a40b4e70 2024-03-25T18:30:35.906 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:30:35.903+0000 7f3092ffd640 1 -- 172.21.15.1:0/373242256 <== mon.1 v1:172.21.15.115:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1586965025 0 0) 0x7f309c001d10 con 0x7f30a40b4e70 2024-03-25T18:30:35.906 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:30:35.903+0000 7f3092ffd640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:30:35.906 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:30:35.903+0000 7f3092ffd640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-03-25T18:30:35.906 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:30:35.903+0000 7f3092ffd640 10 monclient(hunting): dump: 2024-03-25T18:30:35.906 INFO:teuthology.misc.health.smithi001.stderr:epoch 1 2024-03-25T18:30:35.906 INFO:teuthology.misc.health.smithi001.stderr:fsid c8ea72cf-9e99-4383-9b52-6c23d365fdfc 2024-03-25T18:30:35.906 INFO:teuthology.misc.health.smithi001.stderr:last_changed 2024-03-25T18:25:50.510519+0000 2024-03-25T18:30:35.906 INFO:teuthology.misc.health.smithi001.stderr:created 2024-03-25T18:25:50.510519+0000 2024-03-25T18:30:35.906 INFO:teuthology.misc.health.smithi001.stderr:min_mon_release 19 (squid) 2024-03-25T18:30:35.906 INFO:teuthology.misc.health.smithi001.stderr:election_strategy: 3 2024-03-25T18:30:35.906 INFO:teuthology.misc.health.smithi001.stderr:0: v1:172.21.15.1:6789/0 mon.a 2024-03-25T18:30:35.907 INFO:teuthology.misc.health.smithi001.stderr:1: v1:172.21.15.115:6789/0 mon.b 2024-03-25T18:30:35.907 INFO:teuthology.misc.health.smithi001.stderr:2: v1:172.21.15.1:6790/0 mon.c 2024-03-25T18:30:35.907 INFO:teuthology.misc.health.smithi001.stderr: 2024-03-25T18:30:35.907 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:30:35.903+0000 7f3092ffd640 10 monclient(hunting): _finish_auth 0 2024-03-25T18:30:35.907 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:30:35.904+0000 7f3092ffd640 1 -- 172.21.15.1:0/373242256 <== mon.2 v1:172.21.15.1:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 512185887 0 0) 0x7f30a41dbb80 con 0x7f30a40b1b00 2024-03-25T18:30:35.907 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:30:35.904+0000 7f3092ffd640 1 -- 172.21.15.1:0/373242256 --> v1:172.21.15.1:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f3084004eb0 con 0x7f30a40b1b00 2024-03-25T18:30:35.907 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:30:35.904+0000 7f3092ffd640 1 -- 172.21.15.1:0/373242256 <== mon.2 v1:172.21.15.1:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1586965025 0 0) 0x7f3094001800 con 0x7f30a40b1b00 2024-03-25T18:30:35.907 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:30:35.904+0000 7f3092ffd640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:30:35.907 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:30:35.904+0000 7f3092ffd640 10 monclient(hunting): got monmap 1 from mon.c (according to old e1) 2024-03-25T18:30:35.907 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:30:35.904+0000 7f3092ffd640 10 monclient(hunting): dump: 2024-03-25T18:30:35.907 INFO:teuthology.misc.health.smithi001.stderr:epoch 1 2024-03-25T18:30:35.907 INFO:teuthology.misc.health.smithi001.stderr:fsid c8ea72cf-9e99-4383-9b52-6c23d365fdfc 2024-03-25T18:30:35.907 INFO:teuthology.misc.health.smithi001.stderr:last_changed 2024-03-25T18:25:50.510519+0000 2024-03-25T18:30:35.907 INFO:teuthology.misc.health.smithi001.stderr:created 2024-03-25T18:25:50.510519+0000 2024-03-25T18:30:35.907 INFO:teuthology.misc.health.smithi001.stderr:min_mon_release 19 (squid) 2024-03-25T18:30:35.908 INFO:teuthology.misc.health.smithi001.stderr:election_strategy: 3 2024-03-25T18:30:35.908 INFO:teuthology.misc.health.smithi001.stderr:0: v1:172.21.15.1:6789/0 mon.a 2024-03-25T18:30:35.908 INFO:teuthology.misc.health.smithi001.stderr:1: v1:172.21.15.115:6789/0 mon.b 2024-03-25T18:30:35.908 INFO:teuthology.misc.health.smithi001.stderr:2: v1:172.21.15.1:6790/0 mon.c 2024-03-25T18:30:35.908 INFO:teuthology.misc.health.smithi001.stderr: 2024-03-25T18:30:35.917 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:30:35.914+0000 7f3092ffd640 1 -- 172.21.15.1:0/373242256 <== mon.2 v1:172.21.15.1:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 2108424364 0 0) 0x7f3084004eb0 con 0x7f30a40b1b00 2024-03-25T18:30:35.917 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:30:35.914+0000 7f3092ffd640 1 -- 172.21.15.1:0/373242256 >> v1:172.21.15.115:6789/0 conn(0x7f30a40b4e70 legacy=0x7f30a41449c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:30:35.917 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:30:35.914+0000 7f3092ffd640 1 -- 172.21.15.1:0/373242256 >> v1:172.21.15.1:6789/0 conn(0x7f30a40b44b0 legacy=0x7f30a41db470 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:30:35.917 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:30:35.914+0000 7f3092ffd640 10 monclient: _finish_hunting 0 2024-03-25T18:30:35.917 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:30:35.914+0000 7f3092ffd640 1 monclient: found mon.c 2024-03-25T18:30:35.917 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:30:35.914+0000 7f3092ffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:30:35.917 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:30:35.914+0000 7f3092ffd640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.1:6790/0 2024-03-25T18:30:35.917 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:30:35.914+0000 7f3092ffd640 1 -- 172.21.15.1:0/373242256 --> v1:172.21.15.1:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f30a4133850 con 0x7f30a40b1b00 2024-03-25T18:30:35.917 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:30:35.914+0000 7f3092ffd640 10 monclient: _finish_auth 0 2024-03-25T18:30:35.917 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:30:35.914+0000 7f3092ffd640 10 monclient: _check_auth_tickets 2024-03-25T18:30:35.917 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:30:35.914+0000 7f3092ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:30:35.917 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:30:35.914+0000 7f30a89e8640 5 monclient: authenticate success, global_id 4397 2024-03-25T18:30:35.917 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:30:35.914+0000 7f30a89e8640 10 monclient: _renew_subs 2024-03-25T18:30:35.917 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:30:35.914+0000 7f30a89e8640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.1:6790/0 2024-03-25T18:30:35.918 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:30:35.914+0000 7f30a89e8640 1 -- 172.21.15.1:0/373242256 --> v1:172.21.15.1:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f30a41450d0 con 0x7f30a40b1b00 2024-03-25T18:30:35.918 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:30:35.914+0000 7f30a89e8640 10 client.4397.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T18:30:35.918 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:30:35.914+0000 7f30a89e8640 10 monclient: _renew_subs 2024-03-25T18:30:35.918 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:30:35.914+0000 7f30a89e8640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.1:6790/0 2024-03-25T18:30:35.918 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:30:35.914+0000 7f30a89e8640 1 -- 172.21.15.1:0/373242256 --> v1:172.21.15.1:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f30a413a230 con 0x7f30a40b1b00 2024-03-25T18:30:35.918 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:30:35.914+0000 7f30a89e8640 1 librados: init done 2024-03-25T18:30:35.918 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:30:35.916+0000 7f30a89e8640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T18:30:35.919 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:30:35.916+0000 7f30a89e8640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:30:35.919 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:30:35.916+0000 7f30a89e8640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.1:6790/0 2024-03-25T18:30:35.919 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:30:35.916+0000 7f30a89e8640 1 -- 172.21.15.1:0/373242256 --> v1:172.21.15.1:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f30a404f730 con 0x7f30a40b1b00 2024-03-25T18:30:36.223 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:30:36.221+0000 7f3092ffd640 1 -- 172.21.15.1:0/373242256 <== mon.2 v1:172.21.15.1:6790/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f3094001cb0 con 0x7f30a40b1b00 2024-03-25T18:30:36.223 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:30:36.221+0000 7f3092ffd640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:30:36.223 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:30:36.221+0000 7f3092ffd640 1 -- 172.21.15.1:0/373242256 <== mon.2 v1:172.21.15.1:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1586965025 0 0) 0x7f3094003790 con 0x7f30a40b1b00 2024-03-25T18:30:36.223 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:30:36.221+0000 7f3092ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:30:36.223 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:30:36.221+0000 7f3092ffd640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-25T18:30:36.223 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:30:36.221+0000 7f3092ffd640 10 monclient: dump: 2024-03-25T18:30:36.223 INFO:teuthology.misc.health.smithi001.stderr:epoch 1 2024-03-25T18:30:36.223 INFO:teuthology.misc.health.smithi001.stderr:fsid c8ea72cf-9e99-4383-9b52-6c23d365fdfc 2024-03-25T18:30:36.223 INFO:teuthology.misc.health.smithi001.stderr:last_changed 2024-03-25T18:25:50.510519+0000 2024-03-25T18:30:36.223 INFO:teuthology.misc.health.smithi001.stderr:created 2024-03-25T18:25:50.510519+0000 2024-03-25T18:30:36.223 INFO:teuthology.misc.health.smithi001.stderr:min_mon_release 19 (squid) 2024-03-25T18:30:36.223 INFO:teuthology.misc.health.smithi001.stderr:election_strategy: 3 2024-03-25T18:30:36.224 INFO:teuthology.misc.health.smithi001.stderr:0: v1:172.21.15.1:6789/0 mon.a 2024-03-25T18:30:36.224 INFO:teuthology.misc.health.smithi001.stderr:1: v1:172.21.15.115:6789/0 mon.b 2024-03-25T18:30:36.224 INFO:teuthology.misc.health.smithi001.stderr:2: v1:172.21.15.1:6790/0 mon.c 2024-03-25T18:30:36.224 INFO:teuthology.misc.health.smithi001.stderr: 2024-03-25T18:30:36.224 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:30:36.221+0000 7f30a0d99640 4 set_mon_vals no callback set 2024-03-25T18:30:36.224 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:30:36.221+0000 7f3092ffd640 1 -- 172.21.15.1:0/373242256 <== mon.2 v1:172.21.15.1:6790/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 21787975 0 0) 0x7f3094003970 con 0x7f30a40b1b00 2024-03-25T18:30:36.224 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:30:36.222+0000 7f30a1d9b640 10 monclient: get_auth_request con 0x7f308406dfb0 auth_method 0 2024-03-25T18:30:36.225 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:30:36.222+0000 7f3092ffd640 10 client.4397.objecter ms_handle_connect 0x7f308406dfb0 2024-03-25T18:30:36.230 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:30:36.228+0000 7f3092ffd640 1 -- 172.21.15.1:0/373242256 <== mon.2 v1:172.21.15.1:6790/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 996254230 0 0) 0x7f3094053420 con 0x7f30a40b1b00 2024-03-25T18:30:36.230 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:30:36.228+0000 7f3092ffd640 10 client.4397.objecter ms_dispatch 0x7f30a4000b90 osd_map(10..10 src has 1..10) v4 2024-03-25T18:30:36.230 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:30:36.228+0000 7f3092ffd640 3 client.4397.objecter handle_osd_map got epochs [10,10] > 0 2024-03-25T18:30:36.230 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:30:36.228+0000 7f3092ffd640 3 client.4397.objecter handle_osd_map decoding full epoch 10 2024-03-25T18:30:36.230 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:30:36.228+0000 7f3092ffd640 20 client.4397.objecter dump_active .. 0 homeless 2024-03-25T18:30:36.230 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:30:36.228+0000 7f3092ffd640 1 -- 172.21.15.1:0/373242256 <== mon.2 v1:172.21.15.1:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f309404ef30 con 0x7f30a40b1b00 2024-03-25T18:30:36.230 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:30:36.228+0000 7f3092ffd640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:30:36.230 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:30:36.228+0000 7f3092ffd640 10 monclient: _finish_command 1 = system:0 2024-03-25T18:30:36.484 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:30:36.482+0000 7f30a89e8640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:30:36.484 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:30:36.482+0000 7f30a89e8640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:30:36.484 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:30:36.482+0000 7f30a89e8640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.1:6790/0 2024-03-25T18:30:36.484 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:30:36.482+0000 7f30a89e8640 1 -- 172.21.15.1:0/373242256 --> v1:172.21.15.1:6790/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f30a4145260 con 0x7f30a40b1b00 2024-03-25T18:30:36.659 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:30:36.657+0000 7f3092ffd640 1 -- 172.21.15.1:0/373242256 <== mon.2 v1:172.21.15.1:6790/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 4092213348) 0x7f30a4145260 con 0x7f30a40b1b00 2024-03-25T18:30:36.659 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:30:36.657+0000 7f3092ffd640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:30:36.660 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:30:36.657+0000 7f3092ffd640 10 monclient: _finish_command 2 = system:0 2024-03-25T18:30:36.660 INFO:teuthology.misc.health.smithi001.stdout: 2024-03-25T18:30:36.660 INFO:teuthology.misc.health.smithi001.stdout:{"epoch":10,"fsid":"c8ea72cf-9e99-4383-9b52-6c23d365fdfc","created":"2024-03-25T18:26:38.856638+0000","modified":"2024-03-25T18:26:44.224208+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T18:26:44.224208+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"b9f34922-3a84-49be-8426-807c1eceaefc","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"42bf306f-55e3-4634-9874-8b4b73ac6644","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"292cb23a-18a9-4207-a7cc-4d5727feaec0","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"45e0e9ab-898d-4673-9590-b4117bc5e84a","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"831a5a09-73ab-4a61-9872-088e76bdbc66","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"88cd903d-3b55-4840-9396-6e3cfe005282","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"89db7355-bc9d-46f4-8ab0-e8d9e744e026","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"cac47bee-0914-4f08-896f-348838882866","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T18:30:36.662 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:30:36.660+0000 7f30a89e8640 10 librados: watch_flush enter 2024-03-25T18:30:36.663 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:30:36.660+0000 7f30a89e8640 10 librados: watch_flush exit 2024-03-25T18:30:36.663 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:30:36.660+0000 7f30a89e8640 20 client.4397.objecter shutdown clearing up homeless session... 2024-03-25T18:30:36.663 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:30:36.660+0000 7f30a89e8640 10 client.4397.objecter successfully canceled tick 2024-03-25T18:30:36.663 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:30:36.660+0000 7f30a89e8640 1 -- 172.21.15.1:0/373242256 >> v1:172.21.15.115:6800/4237379176 conn(0x7f308406dfb0 legacy=0x7f3084070470 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:30:36.663 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:30:36.660+0000 7f30a89e8640 10 monclient: shutdown 2024-03-25T18:30:36.663 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:30:36.660+0000 7f30a89e8640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:30:36.663 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:30:36.660+0000 7f30a89e8640 1 -- 172.21.15.1:0/373242256 >> v1:172.21.15.1:6790/0 conn(0x7f30a40b1b00 legacy=0x7f30a41dad60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:30:36.663 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:30:36.660+0000 7f30a89e8640 1 -- 172.21.15.1:0/373242256 shutdown_connections 2024-03-25T18:30:36.663 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:30:36.660+0000 7f30a89e8640 1 -- 172.21.15.1:0/373242256 >> 172.21.15.1:0/373242256 conn(0x7f30a40617a0 msgr2=0x7f30a40b2710 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:30:36.663 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:30:36.660+0000 7f30a89e8640 1 -- 172.21.15.1:0/373242256 shutdown_connections 2024-03-25T18:30:36.663 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:30:36.660+0000 7f30a89e8640 1 -- 172.21.15.1:0/373242256 wait complete. 2024-03-25T18:30:36.663 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:30:36.660+0000 7f30a89e8640 1 librados: shutdown 2024-03-25T18:30:36.675 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T18:30:43.481 DEBUG:teuthology.orchestra.run.smithi001:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T18:30:43.596 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:30:43.595+0000 7efd8cfed640 10 monclient: get_monmap_and_config 2024-03-25T18:30:43.596 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:30:43.595+0000 7efd8cfed640 10 monclient: build_initial_monmap 2024-03-25T18:30:43.596 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:30:43.595+0000 7efd8cfed640 10 monclient: monmap: 2024-03-25T18:30:43.596 INFO:teuthology.misc.health.smithi001.stderr:epoch 0 2024-03-25T18:30:43.596 INFO:teuthology.misc.health.smithi001.stderr:fsid c8ea72cf-9e99-4383-9b52-6c23d365fdfc 2024-03-25T18:30:43.596 INFO:teuthology.misc.health.smithi001.stderr:last_changed 2024-03-25T18:30:43.595942+0000 2024-03-25T18:30:43.596 INFO:teuthology.misc.health.smithi001.stderr:created 2024-03-25T18:30:43.595942+0000 2024-03-25T18:30:43.596 INFO:teuthology.misc.health.smithi001.stderr:min_mon_release 0 (unknown) 2024-03-25T18:30:43.596 INFO:teuthology.misc.health.smithi001.stderr:election_strategy: 3 2024-03-25T18:30:43.596 INFO:teuthology.misc.health.smithi001.stderr:0: v1:172.21.15.1:6789/0 mon.noname-a 2024-03-25T18:30:43.596 INFO:teuthology.misc.health.smithi001.stderr:1: v1:172.21.15.115:6789/0 mon.noname-c 2024-03-25T18:30:43.596 INFO:teuthology.misc.health.smithi001.stderr:2: v1:172.21.15.1:6790/0 mon.noname-b 2024-03-25T18:30:43.596 INFO:teuthology.misc.health.smithi001.stderr: 2024-03-25T18:30:43.597 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:30:43.596+0000 7efd8cfed640 1 Processor -- start 2024-03-25T18:30:43.597 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:30:43.596+0000 7efd8cfed640 1 -- start start 2024-03-25T18:30:43.597 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:30:43.596+0000 7efd8cfed640 10 monclient: init 2024-03-25T18:30:43.598 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:30:43.597+0000 7efd8cfed640 10 monclient: _reopen_session rank -1 2024-03-25T18:30:43.598 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:30:43.597+0000 7efd8cfed640 10 monclient: _add_conns ranks=[0,2,1] 2024-03-25T18:30:43.598 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:30:43.597+0000 7efd8cfed640 10 monclient(hunting): picked mon.noname-a con 0x7efd880b4ff0 addr v1:172.21.15.1:6789/0 2024-03-25T18:30:43.598 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:30:43.597+0000 7efd8cfed640 10 monclient(hunting): picked mon.noname-b con 0x7efd880b4670 addr v1:172.21.15.1:6790/0 2024-03-25T18:30:43.598 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:30:43.597+0000 7efd8cfed640 10 monclient(hunting): picked mon.noname-c con 0x7efd8813c480 addr v1:172.21.15.115:6789/0 2024-03-25T18:30:43.599 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:30:43.597+0000 7efd8cfed640 1 -- --> v1:172.21.15.1:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7efd880ab0e0 con 0x7efd880b4ff0 2024-03-25T18:30:43.599 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:30:43.597+0000 7efd8cfed640 1 -- --> v1:172.21.15.115:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7efd880ab3c0 con 0x7efd8813c480 2024-03-25T18:30:43.599 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:30:43.597+0000 7efd8cfed640 1 -- --> v1:172.21.15.1:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7efd880a9f30 con 0x7efd880b4670 2024-03-25T18:30:43.599 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:30:43.597+0000 7efd8cfed640 10 monclient(hunting): _renew_subs 2024-03-25T18:30:43.599 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:30:43.597+0000 7efd8cfed640 10 monclient(hunting): authenticate will time out at 7724.067163s 2024-03-25T18:30:43.605 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:30:43.604+0000 7efd85d9b640 1 --1- >> v1:172.21.15.1:6790/0 conn(0x7efd880b4670 0x7efd881425c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.1:6790/0 says I am v1:172.21.15.1:40738/0 (socket says 172.21.15.1:40738) 2024-03-25T18:30:43.605 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:30:43.604+0000 7efd85d9b640 1 -- 172.21.15.1:0/4293699197 learned_addr learned my addr 172.21.15.1:0/4293699197 (peer_addr_for_me v1:172.21.15.1:0/0) 2024-03-25T18:30:43.617 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:30:43.617+0000 7efd8559a640 1 -- 172.21.15.1:0/4293699197 <== mon.2 v1:172.21.15.1:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 858300619 0 0) 0x7efd880a9f30 con 0x7efd880b4670 2024-03-25T18:30:43.618 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:30:43.617+0000 7efd8559a640 10 monclient(hunting): _negotiate 2024-03-25T18:30:43.618 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:30:43.617+0000 7efd8559a640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:30:43.618 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:30:43.617+0000 7efd8559a640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:30:43.618 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:30:43.617+0000 7efd8559a640 10 monclient(hunting): my global_id is 4400 2024-03-25T18:30:43.618 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:30:43.617+0000 7efd8559a640 1 -- 172.21.15.1:0/4293699197 --> v1:172.21.15.1:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7efd70003520 con 0x7efd880b4670 2024-03-25T18:30:43.621 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:30:43.620+0000 7efd8559a640 1 -- 172.21.15.1:0/4293699197 <== mon.0 v1:172.21.15.1:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3500803432 0 0) 0x7efd880ab0e0 con 0x7efd880b4ff0 2024-03-25T18:30:43.621 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:30:43.620+0000 7efd8559a640 10 monclient(hunting): _negotiate 2024-03-25T18:30:43.621 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:30:43.620+0000 7efd8559a640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:30:43.621 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:30:43.620+0000 7efd8559a640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:30:43.621 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:30:43.620+0000 7efd8559a640 10 monclient(hunting): my global_id is 4392 2024-03-25T18:30:43.621 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:30:43.620+0000 7efd8559a640 1 -- 172.21.15.1:0/4293699197 --> v1:172.21.15.1:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7efd880a9f30 con 0x7efd880b4ff0 2024-03-25T18:30:43.624 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:30:43.623+0000 7efd8559a640 1 -- 172.21.15.1:0/4293699197 <== mon.1 v1:172.21.15.115:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1339907168 0 0) 0x7efd880ab3c0 con 0x7efd8813c480 2024-03-25T18:30:43.624 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:30:43.623+0000 7efd8559a640 10 monclient(hunting): _negotiate 2024-03-25T18:30:43.624 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:30:43.623+0000 7efd8559a640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:30:43.624 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:30:43.623+0000 7efd8559a640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:30:43.624 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:30:43.623+0000 7efd8559a640 10 monclient(hunting): my global_id is 4396 2024-03-25T18:30:43.625 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:30:43.623+0000 7efd8559a640 1 -- 172.21.15.1:0/4293699197 --> v1:172.21.15.115:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7efd880ab0e0 con 0x7efd8813c480 2024-03-25T18:30:43.628 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:30:43.627+0000 7efd8559a640 1 -- 172.21.15.1:0/4293699197 <== mon.2 v1:172.21.15.1:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1321445135 0 0) 0x7efd70003520 con 0x7efd880b4670 2024-03-25T18:30:43.628 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:30:43.627+0000 7efd8559a640 1 -- 172.21.15.1:0/4293699197 >> v1:172.21.15.115:6789/0 conn(0x7efd8813c480 legacy=0x7efd8814dc00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:30:43.628 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:30:43.627+0000 7efd8559a640 1 -- 172.21.15.1:0/4293699197 >> v1:172.21.15.1:6789/0 conn(0x7efd880b4ff0 legacy=0x7efd8813bd70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:30:43.628 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:30:43.627+0000 7efd8559a640 10 monclient: _finish_hunting 0 2024-03-25T18:30:43.628 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:30:43.627+0000 7efd8559a640 1 monclient: found mon.noname-b 2024-03-25T18:30:43.628 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:30:43.627+0000 7efd8559a640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:30:43.628 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:30:43.627+0000 7efd8559a640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.1:6790/0 2024-03-25T18:30:43.628 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:30:43.627+0000 7efd8559a640 1 -- 172.21.15.1:0/4293699197 --> v1:172.21.15.1:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7efd880a9b70 con 0x7efd880b4670 2024-03-25T18:30:43.628 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:30:43.627+0000 7efd8559a640 10 monclient: _finish_auth 0 2024-03-25T18:30:43.628 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:30:43.627+0000 7efd8559a640 10 monclient: _check_auth_tickets 2024-03-25T18:30:43.628 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:30:43.627+0000 7efd8559a640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:30:43.629 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:30:43.627+0000 7efd8559a640 1 -- 172.21.15.1:0/4293699197 <== mon.2 v1:172.21.15.1:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1586965025 0 0) 0x7efd7c001690 con 0x7efd880b4670 2024-03-25T18:30:43.629 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:30:43.627+0000 7efd8559a640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:30:43.629 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:30:43.627+0000 7efd8559a640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-03-25T18:30:43.629 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:30:43.627+0000 7efd8559a640 10 monclient: dump: 2024-03-25T18:30:43.629 INFO:teuthology.misc.health.smithi001.stderr:epoch 1 2024-03-25T18:30:43.629 INFO:teuthology.misc.health.smithi001.stderr:fsid c8ea72cf-9e99-4383-9b52-6c23d365fdfc 2024-03-25T18:30:43.629 INFO:teuthology.misc.health.smithi001.stderr:last_changed 2024-03-25T18:25:50.510519+0000 2024-03-25T18:30:43.629 INFO:teuthology.misc.health.smithi001.stderr:created 2024-03-25T18:25:50.510519+0000 2024-03-25T18:30:43.629 INFO:teuthology.misc.health.smithi001.stderr:min_mon_release 19 (squid) 2024-03-25T18:30:43.629 INFO:teuthology.misc.health.smithi001.stderr:election_strategy: 3 2024-03-25T18:30:43.629 INFO:teuthology.misc.health.smithi001.stderr:0: v1:172.21.15.1:6789/0 mon.a 2024-03-25T18:30:43.629 INFO:teuthology.misc.health.smithi001.stderr:1: v1:172.21.15.115:6789/0 mon.b 2024-03-25T18:30:43.629 INFO:teuthology.misc.health.smithi001.stderr:2: v1:172.21.15.1:6790/0 mon.c 2024-03-25T18:30:43.629 INFO:teuthology.misc.health.smithi001.stderr: 2024-03-25T18:30:43.629 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:30:43.627+0000 7efd8cfed640 5 monclient: authenticate success, global_id 4400 2024-03-25T18:30:43.629 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:30:43.627+0000 7efd8cfed640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T18:30:43.636 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:30:43.635+0000 7efd8559a640 1 -- 172.21.15.1:0/4293699197 <== mon.2 v1:172.21.15.1:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7efd7c0017f0 con 0x7efd880b4670 2024-03-25T18:30:43.636 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:30:43.635+0000 7efd8559a640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:30:43.637 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:30:43.636+0000 7efd8559a640 1 -- 172.21.15.1:0/4293699197 <== mon.2 v1:172.21.15.1:6790/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1586965025 0 0) 0x7efd7c001c00 con 0x7efd880b4670 2024-03-25T18:30:43.637 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:30:43.636+0000 7efd8cfed640 10 monclient: get_monmap_and_config success 2024-03-25T18:30:43.637 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:30:43.636+0000 7efd8cfed640 4 set_mon_vals no callback set 2024-03-25T18:30:43.637 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:30:43.636+0000 7efd8559a640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:30:43.637 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:30:43.636+0000 7efd8559a640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-25T18:30:43.637 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:30:43.636+0000 7efd8559a640 10 monclient: dump: 2024-03-25T18:30:43.637 INFO:teuthology.misc.health.smithi001.stderr:epoch 1 2024-03-25T18:30:43.637 INFO:teuthology.misc.health.smithi001.stderr:fsid c8ea72cf-9e99-4383-9b52-6c23d365fdfc 2024-03-25T18:30:43.637 INFO:teuthology.misc.health.smithi001.stderr:last_changed 2024-03-25T18:25:50.510519+0000 2024-03-25T18:30:43.637 INFO:teuthology.misc.health.smithi001.stderr:created 2024-03-25T18:25:50.510519+0000 2024-03-25T18:30:43.637 INFO:teuthology.misc.health.smithi001.stderr:min_mon_release 19 (squid) 2024-03-25T18:30:43.637 INFO:teuthology.misc.health.smithi001.stderr:election_strategy: 3 2024-03-25T18:30:43.637 INFO:teuthology.misc.health.smithi001.stderr:0: v1:172.21.15.1:6789/0 mon.a 2024-03-25T18:30:43.637 INFO:teuthology.misc.health.smithi001.stderr:1: v1:172.21.15.115:6789/0 mon.b 2024-03-25T18:30:43.637 INFO:teuthology.misc.health.smithi001.stderr:2: v1:172.21.15.1:6790/0 mon.c 2024-03-25T18:30:43.637 INFO:teuthology.misc.health.smithi001.stderr: 2024-03-25T18:30:43.638 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:30:43.636+0000 7efd8cfed640 10 monclient: shutdown 2024-03-25T18:30:43.639 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:30:43.636+0000 7efd8cfed640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:30:43.639 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:30:43.636+0000 7efd8cfed640 1 -- 172.21.15.1:0/4293699197 >> v1:172.21.15.1:6790/0 conn(0x7efd880b4670 legacy=0x7efd881425c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:30:43.639 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:30:43.636+0000 7efd8cfed640 1 -- 172.21.15.1:0/4293699197 shutdown_connections 2024-03-25T18:30:43.639 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:30:43.636+0000 7efd8cfed640 1 -- 172.21.15.1:0/4293699197 >> 172.21.15.1:0/4293699197 conn(0x7efd880617a0 msgr2=0x7efd88061bb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:30:43.639 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:30:43.636+0000 7efd8cfed640 1 -- 172.21.15.1:0/4293699197 shutdown_connections 2024-03-25T18:30:43.639 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:30:43.636+0000 7efd8cfed640 1 -- 172.21.15.1:0/4293699197 wait complete. 2024-03-25T18:30:43.639 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:30:43.637+0000 7efd8cfed640 10 monclient: build_initial_monmap 2024-03-25T18:30:43.639 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:30:43.637+0000 7efd8cfed640 10 monclient: monmap: 2024-03-25T18:30:43.639 INFO:teuthology.misc.health.smithi001.stderr:epoch 0 2024-03-25T18:30:43.639 INFO:teuthology.misc.health.smithi001.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T18:30:43.639 INFO:teuthology.misc.health.smithi001.stderr:last_changed 0.000000 2024-03-25T18:30:43.639 INFO:teuthology.misc.health.smithi001.stderr:created 0.000000 2024-03-25T18:30:43.639 INFO:teuthology.misc.health.smithi001.stderr:min_mon_release 0 (unknown) 2024-03-25T18:30:43.639 INFO:teuthology.misc.health.smithi001.stderr:election_strategy: 1 2024-03-25T18:30:43.639 INFO:teuthology.misc.health.smithi001.stderr:0: v1:172.21.15.1:6789/0 mon.noname-a 2024-03-25T18:30:43.639 INFO:teuthology.misc.health.smithi001.stderr:1: v1:172.21.15.115:6789/0 mon.noname-b 2024-03-25T18:30:43.640 INFO:teuthology.misc.health.smithi001.stderr:2: v1:172.21.15.1:6790/0 mon.noname-c 2024-03-25T18:30:43.640 INFO:teuthology.misc.health.smithi001.stderr: 2024-03-25T18:30:43.640 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:30:43.638+0000 7efd8cfed640 1 librados: starting msgr at 2024-03-25T18:30:43.640 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:30:43.638+0000 7efd8cfed640 1 librados: starting objecter 2024-03-25T18:30:43.640 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:30:43.638+0000 7efd8cfed640 1 Processor -- start 2024-03-25T18:30:43.641 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:30:43.638+0000 7efd8cfed640 1 -- start start 2024-03-25T18:30:43.641 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:30:43.638+0000 7efd8cfed640 1 librados: setting wanted keys 2024-03-25T18:30:43.641 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:30:43.638+0000 7efd8cfed640 1 librados: calling monclient init 2024-03-25T18:30:43.641 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:30:43.638+0000 7efd8cfed640 10 monclient: init 2024-03-25T18:30:43.641 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:30:43.638+0000 7efd8cfed640 10 monclient: _reopen_session rank -1 2024-03-25T18:30:43.641 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:30:43.638+0000 7efd8cfed640 10 monclient: _add_conns ranks=[1,2,0] 2024-03-25T18:30:43.641 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:30:43.638+0000 7efd8cfed640 10 monclient(hunting): picked mon.noname-b con 0x7efd880b4670 addr v1:172.21.15.115:6789/0 2024-03-25T18:30:43.641 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:30:43.638+0000 7efd8cfed640 10 monclient(hunting): picked mon.noname-c con 0x7efd880b4ff0 addr v1:172.21.15.1:6790/0 2024-03-25T18:30:43.641 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:30:43.639+0000 7efd8cfed640 10 monclient(hunting): picked mon.noname-a con 0x7efd8813c480 addr v1:172.21.15.1:6789/0 2024-03-25T18:30:43.641 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:30:43.639+0000 7efd8cfed640 1 -- --> v1:172.21.15.1:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7efd880ac170 con 0x7efd8813c480 2024-03-25T18:30:43.641 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:30:43.639+0000 7efd8cfed640 1 -- --> v1:172.21.15.115:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7efd880ac450 con 0x7efd880b4670 2024-03-25T18:30:43.641 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:30:43.639+0000 7efd8cfed640 1 -- --> v1:172.21.15.1:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7efd880708a0 con 0x7efd880b4ff0 2024-03-25T18:30:43.641 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:30:43.639+0000 7efd8cfed640 10 monclient(hunting): _renew_subs 2024-03-25T18:30:43.641 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:30:43.639+0000 7efd8cfed640 10 monclient(hunting): authenticate will time out at 7724.108838s 2024-03-25T18:30:43.646 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:30:43.645+0000 7efd8659c640 1 --1- >> v1:172.21.15.115:6789/0 conn(0x7efd880b4670 0x7efd880acb50 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.115:6789/0 says I am v1:172.21.15.1:46980/0 (socket says 172.21.15.1:46980) 2024-03-25T18:30:43.646 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:30:43.645+0000 7efd8659c640 1 -- 172.21.15.1:0/194296975 learned_addr learned my addr 172.21.15.1:0/194296975 (peer_addr_for_me v1:172.21.15.1:0/0) 2024-03-25T18:30:43.646 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:30:43.646+0000 7efd76ffd640 10 client.?.objecter ms_handle_connect 0x7efd880b4670 2024-03-25T18:30:43.647 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:30:43.646+0000 7efd76ffd640 10 client.?.objecter resend_mon_ops 2024-03-25T18:30:43.651 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:30:43.651+0000 7efd76ffd640 1 -- 172.21.15.1:0/194296975 <== mon.1 v1:172.21.15.115:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2821133226 0 0) 0x7efd880ac450 con 0x7efd880b4670 2024-03-25T18:30:43.652 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:30:43.651+0000 7efd76ffd640 10 monclient(hunting): _negotiate 2024-03-25T18:30:43.652 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:30:43.651+0000 7efd76ffd640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:30:43.652 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:30:43.651+0000 7efd76ffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:30:43.652 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:30:43.651+0000 7efd76ffd640 10 monclient(hunting): my global_id is 4399 2024-03-25T18:30:43.652 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:30:43.651+0000 7efd76ffd640 1 -- 172.21.15.1:0/194296975 --> v1:172.21.15.115:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7efd68004eb0 con 0x7efd880b4670 2024-03-25T18:30:43.655 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:30:43.654+0000 7efd76ffd640 10 client.?.objecter ms_handle_connect 0x7efd880b4ff0 2024-03-25T18:30:43.655 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:30:43.654+0000 7efd76ffd640 10 client.?.objecter resend_mon_ops 2024-03-25T18:30:43.662 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:30:43.661+0000 7efd76ffd640 1 -- 172.21.15.1:0/194296975 <== mon.1 v1:172.21.15.115:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1164987914 0 0) 0x7efd68004eb0 con 0x7efd880b4670 2024-03-25T18:30:43.662 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:30:43.661+0000 7efd76ffd640 1 -- 172.21.15.1:0/194296975 --> v1:172.21.15.115:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7efd880ac450 con 0x7efd880b4670 2024-03-25T18:30:43.662 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:30:43.661+0000 7efd76ffd640 1 -- 172.21.15.1:0/194296975 <== mon.1 v1:172.21.15.115:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1586965025 0 0) 0x7efd78001a30 con 0x7efd880b4670 2024-03-25T18:30:43.662 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:30:43.661+0000 7efd76ffd640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:30:43.662 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:30:43.661+0000 7efd76ffd640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-03-25T18:30:43.662 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:30:43.661+0000 7efd76ffd640 10 monclient(hunting): dump: 2024-03-25T18:30:43.662 INFO:teuthology.misc.health.smithi001.stderr:epoch 1 2024-03-25T18:30:43.662 INFO:teuthology.misc.health.smithi001.stderr:fsid c8ea72cf-9e99-4383-9b52-6c23d365fdfc 2024-03-25T18:30:43.662 INFO:teuthology.misc.health.smithi001.stderr:last_changed 2024-03-25T18:25:50.510519+0000 2024-03-25T18:30:43.662 INFO:teuthology.misc.health.smithi001.stderr:created 2024-03-25T18:25:50.510519+0000 2024-03-25T18:30:43.662 INFO:teuthology.misc.health.smithi001.stderr:min_mon_release 19 (squid) 2024-03-25T18:30:43.662 INFO:teuthology.misc.health.smithi001.stderr:election_strategy: 3 2024-03-25T18:30:43.662 INFO:teuthology.misc.health.smithi001.stderr:0: v1:172.21.15.1:6789/0 mon.a 2024-03-25T18:30:43.662 INFO:teuthology.misc.health.smithi001.stderr:1: v1:172.21.15.115:6789/0 mon.b 2024-03-25T18:30:43.662 INFO:teuthology.misc.health.smithi001.stderr:2: v1:172.21.15.1:6790/0 mon.c 2024-03-25T18:30:43.663 INFO:teuthology.misc.health.smithi001.stderr: 2024-03-25T18:30:43.663 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:30:43.661+0000 7efd76ffd640 10 monclient(hunting): _finish_auth 0 2024-03-25T18:30:43.664 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:30:43.663+0000 7efd76ffd640 1 -- 172.21.15.1:0/194296975 <== mon.2 v1:172.21.15.1:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 657700562 0 0) 0x7efd880708a0 con 0x7efd880b4ff0 2024-03-25T18:30:43.664 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:30:43.664+0000 7efd76ffd640 10 monclient(hunting): _negotiate 2024-03-25T18:30:43.664 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:30:43.664+0000 7efd76ffd640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:30:43.664 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:30:43.664+0000 7efd76ffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:30:43.665 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:30:43.664+0000 7efd76ffd640 10 monclient(hunting): my global_id is 4403 2024-03-25T18:30:43.665 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:30:43.664+0000 7efd76ffd640 1 -- 172.21.15.1:0/194296975 --> v1:172.21.15.1:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7efd68004eb0 con 0x7efd880b4ff0 2024-03-25T18:30:43.665 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:30:43.665+0000 7efd76ffd640 10 client.?.objecter ms_handle_connect 0x7efd8813c480 2024-03-25T18:30:43.665 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:30:43.665+0000 7efd76ffd640 10 client.?.objecter resend_mon_ops 2024-03-25T18:30:43.677 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:30:43.676+0000 7efd76ffd640 1 -- 172.21.15.1:0/194296975 <== mon.0 v1:172.21.15.1:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3714491026 0 0) 0x7efd880ac170 con 0x7efd8813c480 2024-03-25T18:30:43.677 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:30:43.676+0000 7efd76ffd640 10 monclient(hunting): _negotiate 2024-03-25T18:30:43.677 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:30:43.676+0000 7efd76ffd640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:30:43.677 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:30:43.676+0000 7efd76ffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:30:43.677 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:30:43.676+0000 7efd76ffd640 10 monclient(hunting): my global_id is 4395 2024-03-25T18:30:43.677 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:30:43.676+0000 7efd76ffd640 1 -- 172.21.15.1:0/194296975 --> v1:172.21.15.1:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7efd880708a0 con 0x7efd8813c480 2024-03-25T18:30:43.680 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:30:43.679+0000 7efd76ffd640 1 -- 172.21.15.1:0/194296975 <== mon.2 v1:172.21.15.1:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 748667563 0 0) 0x7efd68004eb0 con 0x7efd880b4ff0 2024-03-25T18:30:43.680 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:30:43.679+0000 7efd76ffd640 1 -- 172.21.15.1:0/194296975 --> v1:172.21.15.1:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7efd880ac170 con 0x7efd880b4ff0 2024-03-25T18:30:43.680 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:30:43.679+0000 7efd76ffd640 1 -- 172.21.15.1:0/194296975 <== mon.2 v1:172.21.15.1:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1586965025 0 0) 0x7efd7c001b00 con 0x7efd880b4ff0 2024-03-25T18:30:43.680 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:30:43.679+0000 7efd76ffd640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:30:43.680 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:30:43.679+0000 7efd76ffd640 10 monclient(hunting): got monmap 1 from mon.c (according to old e1) 2024-03-25T18:30:43.680 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:30:43.679+0000 7efd76ffd640 10 monclient(hunting): dump: 2024-03-25T18:30:43.680 INFO:teuthology.misc.health.smithi001.stderr:epoch 1 2024-03-25T18:30:43.681 INFO:teuthology.misc.health.smithi001.stderr:fsid c8ea72cf-9e99-4383-9b52-6c23d365fdfc 2024-03-25T18:30:43.681 INFO:teuthology.misc.health.smithi001.stderr:last_changed 2024-03-25T18:25:50.510519+0000 2024-03-25T18:30:43.681 INFO:teuthology.misc.health.smithi001.stderr:created 2024-03-25T18:25:50.510519+0000 2024-03-25T18:30:43.681 INFO:teuthology.misc.health.smithi001.stderr:min_mon_release 19 (squid) 2024-03-25T18:30:43.681 INFO:teuthology.misc.health.smithi001.stderr:election_strategy: 3 2024-03-25T18:30:43.681 INFO:teuthology.misc.health.smithi001.stderr:0: v1:172.21.15.1:6789/0 mon.a 2024-03-25T18:30:43.681 INFO:teuthology.misc.health.smithi001.stderr:1: v1:172.21.15.115:6789/0 mon.b 2024-03-25T18:30:43.681 INFO:teuthology.misc.health.smithi001.stderr:2: v1:172.21.15.1:6790/0 mon.c 2024-03-25T18:30:43.681 INFO:teuthology.misc.health.smithi001.stderr: 2024-03-25T18:30:43.682 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:30:43.680+0000 7efd76ffd640 1 -- 172.21.15.1:0/194296975 <== mon.1 v1:172.21.15.115:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 599131570 0 0) 0x7efd880ac450 con 0x7efd880b4670 2024-03-25T18:30:43.682 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:30:43.681+0000 7efd76ffd640 1 -- 172.21.15.1:0/194296975 >> v1:172.21.15.1:6790/0 conn(0x7efd880b4ff0 legacy=0x7efd880715a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:30:43.683 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:30:43.681+0000 7efd76ffd640 1 -- 172.21.15.1:0/194296975 >> v1:172.21.15.1:6789/0 conn(0x7efd8813c480 legacy=0x7efd8806eed0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:30:43.683 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:30:43.681+0000 7efd76ffd640 10 monclient: _finish_hunting 0 2024-03-25T18:30:43.683 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:30:43.681+0000 7efd76ffd640 1 monclient: found mon.b 2024-03-25T18:30:43.683 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:30:43.681+0000 7efd76ffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:30:43.683 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:30:43.681+0000 7efd76ffd640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.115:6789/0 2024-03-25T18:30:43.683 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:30:43.681+0000 7efd76ffd640 1 -- 172.21.15.1:0/194296975 --> v1:172.21.15.115:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7efd88071cb0 con 0x7efd880b4670 2024-03-25T18:30:43.683 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:30:43.681+0000 7efd76ffd640 10 monclient: _finish_auth 0 2024-03-25T18:30:43.683 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:30:43.681+0000 7efd76ffd640 10 monclient: _check_auth_tickets 2024-03-25T18:30:43.683 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:30:43.681+0000 7efd76ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:30:43.683 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:30:43.681+0000 7efd8cfed640 5 monclient: authenticate success, global_id 4399 2024-03-25T18:30:43.683 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:30:43.681+0000 7efd8cfed640 10 monclient: _renew_subs 2024-03-25T18:30:43.683 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:30:43.681+0000 7efd8cfed640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.115:6789/0 2024-03-25T18:30:43.683 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:30:43.681+0000 7efd8cfed640 1 -- 172.21.15.1:0/194296975 --> v1:172.21.15.115:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7efd88071f60 con 0x7efd880b4670 2024-03-25T18:30:43.683 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:30:43.681+0000 7efd8cfed640 10 client.4399.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T18:30:43.683 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:30:43.681+0000 7efd8cfed640 10 monclient: _renew_subs 2024-03-25T18:30:43.683 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:30:43.681+0000 7efd8cfed640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.115:6789/0 2024-03-25T18:30:43.684 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:30:43.681+0000 7efd8cfed640 1 -- 172.21.15.1:0/194296975 --> v1:172.21.15.115:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7efd881e7830 con 0x7efd880b4670 2024-03-25T18:30:43.684 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:30:43.681+0000 7efd8cfed640 1 librados: init done 2024-03-25T18:30:43.684 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:30:43.682+0000 7efd8cfed640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T18:30:43.684 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:30:43.682+0000 7efd8cfed640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:30:43.684 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:30:43.682+0000 7efd8cfed640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.115:6789/0 2024-03-25T18:30:43.684 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:30:43.682+0000 7efd8cfed640 1 -- 172.21.15.1:0/194296975 --> v1:172.21.15.115:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7efd8806dbd0 con 0x7efd880b4670 2024-03-25T18:30:43.896 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:30:43.895+0000 7efd76ffd640 1 -- 172.21.15.1:0/194296975 <== mon.1 v1:172.21.15.115:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7efd78001d10 con 0x7efd880b4670 2024-03-25T18:30:43.896 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:30:43.895+0000 7efd76ffd640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:30:43.896 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:30:43.895+0000 7efd84d99640 4 set_mon_vals no callback set 2024-03-25T18:30:43.952 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:30:43.951+0000 7efd76ffd640 1 -- 172.21.15.1:0/194296975 <== mon.1 v1:172.21.15.115:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1586965025 0 0) 0x7efd78003a60 con 0x7efd880b4670 2024-03-25T18:30:43.952 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:30:43.951+0000 7efd76ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:30:43.952 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:30:43.951+0000 7efd76ffd640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-25T18:30:43.952 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:30:43.951+0000 7efd76ffd640 10 monclient: dump: 2024-03-25T18:30:43.952 INFO:teuthology.misc.health.smithi001.stderr:epoch 1 2024-03-25T18:30:43.952 INFO:teuthology.misc.health.smithi001.stderr:fsid c8ea72cf-9e99-4383-9b52-6c23d365fdfc 2024-03-25T18:30:43.952 INFO:teuthology.misc.health.smithi001.stderr:last_changed 2024-03-25T18:25:50.510519+0000 2024-03-25T18:30:43.952 INFO:teuthology.misc.health.smithi001.stderr:created 2024-03-25T18:25:50.510519+0000 2024-03-25T18:30:43.952 INFO:teuthology.misc.health.smithi001.stderr:min_mon_release 19 (squid) 2024-03-25T18:30:43.952 INFO:teuthology.misc.health.smithi001.stderr:election_strategy: 3 2024-03-25T18:30:43.953 INFO:teuthology.misc.health.smithi001.stderr:0: v1:172.21.15.1:6789/0 mon.a 2024-03-25T18:30:43.953 INFO:teuthology.misc.health.smithi001.stderr:1: v1:172.21.15.115:6789/0 mon.b 2024-03-25T18:30:43.953 INFO:teuthology.misc.health.smithi001.stderr:2: v1:172.21.15.1:6790/0 mon.c 2024-03-25T18:30:43.953 INFO:teuthology.misc.health.smithi001.stderr: 2024-03-25T18:30:43.953 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:30:43.952+0000 7efd76ffd640 1 -- 172.21.15.1:0/194296975 <== mon.1 v1:172.21.15.115:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 21787975 0 0) 0x7efd78003c40 con 0x7efd880b4670 2024-03-25T18:30:43.953 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:30:43.952+0000 7efd85d9b640 10 monclient: get_auth_request con 0x7efd6806e080 auth_method 0 2024-03-25T18:30:43.953 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:30:43.953+0000 7efd76ffd640 10 client.4399.objecter ms_handle_connect 0x7efd6806e080 2024-03-25T18:30:43.956 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:30:43.955+0000 7efd76ffd640 1 -- 172.21.15.1:0/194296975 <== mon.1 v1:172.21.15.115:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 996254230 0 0) 0x7efd78054930 con 0x7efd880b4670 2024-03-25T18:30:43.956 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:30:43.955+0000 7efd76ffd640 10 client.4399.objecter ms_dispatch 0x7efd88000b90 osd_map(10..10 src has 1..10) v4 2024-03-25T18:30:43.956 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:30:43.955+0000 7efd76ffd640 3 client.4399.objecter handle_osd_map got epochs [10,10] > 0 2024-03-25T18:30:43.956 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:30:43.955+0000 7efd76ffd640 3 client.4399.objecter handle_osd_map decoding full epoch 10 2024-03-25T18:30:43.956 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:30:43.955+0000 7efd76ffd640 20 client.4399.objecter dump_active .. 0 homeless 2024-03-25T18:30:43.958 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:30:43.958+0000 7efd76ffd640 1 -- 172.21.15.1:0/194296975 <== mon.1 v1:172.21.15.115:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7efd78050440 con 0x7efd880b4670 2024-03-25T18:30:43.959 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:30:43.958+0000 7efd76ffd640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:30:43.959 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:30:43.958+0000 7efd76ffd640 10 monclient: _finish_command 1 = system:0 2024-03-25T18:30:44.214 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:30:44.213+0000 7efd8cfed640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:30:44.214 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:30:44.213+0000 7efd8cfed640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:30:44.214 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:30:44.213+0000 7efd8cfed640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.115:6789/0 2024-03-25T18:30:44.214 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:30:44.213+0000 7efd8cfed640 1 -- 172.21.15.1:0/194296975 --> v1:172.21.15.115:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7efd880688c0 con 0x7efd880b4670 2024-03-25T18:30:44.234 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:30:44.233+0000 7efd76ffd640 1 -- 172.21.15.1:0/194296975 <== mon.1 v1:172.21.15.115:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 4092213348) 0x7efd78052370 con 0x7efd880b4670 2024-03-25T18:30:44.234 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:30:44.233+0000 7efd76ffd640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:30:44.234 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:30:44.233+0000 7efd76ffd640 10 monclient: _finish_command 2 = system:0 2024-03-25T18:30:44.234 INFO:teuthology.misc.health.smithi001.stdout: 2024-03-25T18:30:44.234 INFO:teuthology.misc.health.smithi001.stdout:{"epoch":10,"fsid":"c8ea72cf-9e99-4383-9b52-6c23d365fdfc","created":"2024-03-25T18:26:38.856638+0000","modified":"2024-03-25T18:26:44.224208+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T18:26:44.224208+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"b9f34922-3a84-49be-8426-807c1eceaefc","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"42bf306f-55e3-4634-9874-8b4b73ac6644","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"292cb23a-18a9-4207-a7cc-4d5727feaec0","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"45e0e9ab-898d-4673-9590-b4117bc5e84a","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"831a5a09-73ab-4a61-9872-088e76bdbc66","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"88cd903d-3b55-4840-9396-6e3cfe005282","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"89db7355-bc9d-46f4-8ab0-e8d9e744e026","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"cac47bee-0914-4f08-896f-348838882866","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T18:30:44.237 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:30:44.235+0000 7efd8cfed640 10 librados: watch_flush enter 2024-03-25T18:30:44.237 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:30:44.235+0000 7efd8cfed640 10 librados: watch_flush exit 2024-03-25T18:30:44.237 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:30:44.235+0000 7efd8cfed640 20 client.4399.objecter shutdown clearing up homeless session... 2024-03-25T18:30:44.237 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:30:44.235+0000 7efd8cfed640 10 client.4399.objecter successfully canceled tick 2024-03-25T18:30:44.237 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:30:44.235+0000 7efd8cfed640 1 -- 172.21.15.1:0/194296975 >> v1:172.21.15.115:6800/4237379176 conn(0x7efd6806e080 legacy=0x7efd68070540 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:30:44.237 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:30:44.236+0000 7efd8cfed640 10 monclient: shutdown 2024-03-25T18:30:44.237 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:30:44.236+0000 7efd8cfed640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:30:44.237 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:30:44.236+0000 7efd8cfed640 1 -- 172.21.15.1:0/194296975 >> v1:172.21.15.115:6789/0 conn(0x7efd880b4670 legacy=0x7efd880acb50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:30:44.237 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:30:44.236+0000 7efd8cfed640 1 -- 172.21.15.1:0/194296975 shutdown_connections 2024-03-25T18:30:44.237 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:30:44.236+0000 7efd8cfed640 1 -- 172.21.15.1:0/194296975 >> 172.21.15.1:0/194296975 conn(0x7efd880617a0 msgr2=0x7efd881455d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:30:44.237 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:30:44.236+0000 7efd8cfed640 1 -- 172.21.15.1:0/194296975 shutdown_connections 2024-03-25T18:30:44.237 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:30:44.236+0000 7efd8cfed640 1 -- 172.21.15.1:0/194296975 wait complete. 2024-03-25T18:30:44.237 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:30:44.236+0000 7efd8cfed640 1 librados: shutdown 2024-03-25T18:30:44.250 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T18:30:51.057 DEBUG:teuthology.orchestra.run.smithi001:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T18:30:51.176 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:30:51.175+0000 7f7a2713b640 10 monclient: get_monmap_and_config 2024-03-25T18:30:51.176 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:30:51.175+0000 7f7a2713b640 10 monclient: build_initial_monmap 2024-03-25T18:30:51.176 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:30:51.175+0000 7f7a2713b640 10 monclient: monmap: 2024-03-25T18:30:51.176 INFO:teuthology.misc.health.smithi001.stderr:epoch 0 2024-03-25T18:30:51.176 INFO:teuthology.misc.health.smithi001.stderr:fsid c8ea72cf-9e99-4383-9b52-6c23d365fdfc 2024-03-25T18:30:51.176 INFO:teuthology.misc.health.smithi001.stderr:last_changed 2024-03-25T18:30:51.176295+0000 2024-03-25T18:30:51.176 INFO:teuthology.misc.health.smithi001.stderr:created 2024-03-25T18:30:51.176295+0000 2024-03-25T18:30:51.176 INFO:teuthology.misc.health.smithi001.stderr:min_mon_release 0 (unknown) 2024-03-25T18:30:51.177 INFO:teuthology.misc.health.smithi001.stderr:election_strategy: 3 2024-03-25T18:30:51.177 INFO:teuthology.misc.health.smithi001.stderr:0: v1:172.21.15.1:6789/0 mon.noname-a 2024-03-25T18:30:51.177 INFO:teuthology.misc.health.smithi001.stderr:1: v1:172.21.15.115:6789/0 mon.noname-c 2024-03-25T18:30:51.177 INFO:teuthology.misc.health.smithi001.stderr:2: v1:172.21.15.1:6790/0 mon.noname-b 2024-03-25T18:30:51.177 INFO:teuthology.misc.health.smithi001.stderr: 2024-03-25T18:30:51.178 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:30:51.176+0000 7f7a2713b640 1 Processor -- start 2024-03-25T18:30:51.178 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:30:51.176+0000 7f7a2713b640 1 -- start start 2024-03-25T18:30:51.178 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:30:51.176+0000 7f7a2713b640 10 monclient: init 2024-03-25T18:30:51.178 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:30:51.177+0000 7f7a2713b640 10 monclient: _reopen_session rank -1 2024-03-25T18:30:51.178 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:30:51.177+0000 7f7a2713b640 10 monclient: _add_conns ranks=[0,2,1] 2024-03-25T18:30:51.178 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:30:51.177+0000 7f7a2713b640 10 monclient(hunting): picked mon.noname-a con 0x7f7a20142e80 addr v1:172.21.15.1:6789/0 2024-03-25T18:30:51.178 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:30:51.177+0000 7f7a2713b640 10 monclient(hunting): picked mon.noname-b con 0x7f7a200af5d0 addr v1:172.21.15.1:6790/0 2024-03-25T18:30:51.178 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:30:51.177+0000 7f7a2713b640 10 monclient(hunting): picked mon.noname-c con 0x7f7a200b0220 addr v1:172.21.15.115:6789/0 2024-03-25T18:30:51.179 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:30:51.177+0000 7f7a2713b640 1 -- --> v1:172.21.15.1:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7a200afb40 con 0x7f7a20142e80 2024-03-25T18:30:51.179 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:30:51.177+0000 7f7a2713b640 1 -- --> v1:172.21.15.115:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7a200b1210 con 0x7f7a200b0220 2024-03-25T18:30:51.179 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:30:51.177+0000 7f7a2713b640 1 -- --> v1:172.21.15.1:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7a20146220 con 0x7f7a200af5d0 2024-03-25T18:30:51.179 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:30:51.177+0000 7f7a2713b640 10 monclient(hunting): _renew_subs 2024-03-25T18:30:51.179 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:30:51.177+0000 7f7a2713b640 10 monclient(hunting): authenticate will time out at 7731.647230s 2024-03-25T18:30:51.185 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:30:51.185+0000 7f7a1ffff640 1 --1- >> v1:172.21.15.1:6790/0 conn(0x7f7a200af5d0 0x7f7a20145b10 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.1:6790/0 says I am v1:172.21.15.1:45660/0 (socket says 172.21.15.1:45660) 2024-03-25T18:30:51.186 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:30:51.185+0000 7f7a1ffff640 1 -- 172.21.15.1:0/1715011704 learned_addr learned my addr 172.21.15.1:0/1715011704 (peer_addr_for_me v1:172.21.15.1:0/0) 2024-03-25T18:30:51.193 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:30:51.193+0000 7f7a1f7fe640 1 -- 172.21.15.1:0/1715011704 <== mon.2 v1:172.21.15.1:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3008604157 0 0) 0x7f7a20146220 con 0x7f7a200af5d0 2024-03-25T18:30:51.193 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:30:51.193+0000 7f7a1f7fe640 10 monclient(hunting): _negotiate 2024-03-25T18:30:51.194 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:30:51.193+0000 7f7a1f7fe640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:30:51.194 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:30:51.193+0000 7f7a1f7fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:30:51.194 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:30:51.193+0000 7f7a1f7fe640 10 monclient(hunting): my global_id is 4406 2024-03-25T18:30:51.194 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:30:51.193+0000 7f7a1f7fe640 1 -- 172.21.15.1:0/1715011704 --> v1:172.21.15.1:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f7a10003520 con 0x7f7a200af5d0 2024-03-25T18:30:51.202 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:30:51.200+0000 7f7a1f7fe640 1 -- 172.21.15.1:0/1715011704 <== mon.2 v1:172.21.15.1:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1220856012 0 0) 0x7f7a10003520 con 0x7f7a200af5d0 2024-03-25T18:30:51.202 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:30:51.201+0000 7f7a1f7fe640 1 -- 172.21.15.1:0/1715011704 >> v1:172.21.15.115:6789/0 conn(0x7f7a200b0220 legacy=0x7f7a2014dbd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:30:51.202 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:30:51.201+0000 7f7a1f7fe640 1 -- 172.21.15.1:0/1715011704 >> v1:172.21.15.1:6789/0 conn(0x7f7a20142e80 legacy=0x7f7a2006aba0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-25T18:30:51.202 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:30:51.201+0000 7f7a1f7fe640 10 monclient: _finish_hunting 0 2024-03-25T18:30:51.202 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:30:51.201+0000 7f7a1f7fe640 1 monclient: found mon.noname-b 2024-03-25T18:30:51.202 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:30:51.201+0000 7f7a1f7fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:30:51.202 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:30:51.201+0000 7f7a1f7fe640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.1:6790/0 2024-03-25T18:30:51.202 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:30:51.201+0000 7f7a1f7fe640 1 -- 172.21.15.1:0/1715011704 --> v1:172.21.15.1:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7a201463b0 con 0x7f7a200af5d0 2024-03-25T18:30:51.202 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:30:51.201+0000 7f7a1f7fe640 10 monclient: _finish_auth 0 2024-03-25T18:30:51.202 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:30:51.201+0000 7f7a1f7fe640 10 monclient: _check_auth_tickets 2024-03-25T18:30:51.202 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:30:51.201+0000 7f7a1f7fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:30:51.202 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:30:51.201+0000 7f7a1f7fe640 1 -- 172.21.15.1:0/1715011704 <== mon.2 v1:172.21.15.1:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1586965025 0 0) 0x7f7a0c001690 con 0x7f7a200af5d0 2024-03-25T18:30:51.202 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:30:51.201+0000 7f7a1f7fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:30:51.202 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:30:51.201+0000 7f7a1f7fe640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-03-25T18:30:51.203 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:30:51.201+0000 7f7a1f7fe640 10 monclient: dump: 2024-03-25T18:30:51.203 INFO:teuthology.misc.health.smithi001.stderr:epoch 1 2024-03-25T18:30:51.203 INFO:teuthology.misc.health.smithi001.stderr:fsid c8ea72cf-9e99-4383-9b52-6c23d365fdfc 2024-03-25T18:30:51.203 INFO:teuthology.misc.health.smithi001.stderr:last_changed 2024-03-25T18:25:50.510519+0000 2024-03-25T18:30:51.203 INFO:teuthology.misc.health.smithi001.stderr:created 2024-03-25T18:25:50.510519+0000 2024-03-25T18:30:51.203 INFO:teuthology.misc.health.smithi001.stderr:min_mon_release 19 (squid) 2024-03-25T18:30:51.203 INFO:teuthology.misc.health.smithi001.stderr:election_strategy: 3 2024-03-25T18:30:51.203 INFO:teuthology.misc.health.smithi001.stderr:0: v1:172.21.15.1:6789/0 mon.a 2024-03-25T18:30:51.203 INFO:teuthology.misc.health.smithi001.stderr:1: v1:172.21.15.115:6789/0 mon.b 2024-03-25T18:30:51.203 INFO:teuthology.misc.health.smithi001.stderr:2: v1:172.21.15.1:6790/0 mon.c 2024-03-25T18:30:51.203 INFO:teuthology.misc.health.smithi001.stderr: 2024-03-25T18:30:51.203 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:30:51.201+0000 7f7a2713b640 5 monclient: authenticate success, global_id 4406 2024-03-25T18:30:51.203 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:30:51.201+0000 7f7a2713b640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T18:30:51.205 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:30:51.204+0000 7f7a1f7fe640 1 -- 172.21.15.1:0/1715011704 <== mon.2 v1:172.21.15.1:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f7a0c001860 con 0x7f7a200af5d0 2024-03-25T18:30:51.205 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:30:51.204+0000 7f7a1f7fe640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:30:51.205 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:30:51.204+0000 7f7a2713b640 10 monclient: get_monmap_and_config success 2024-03-25T18:30:51.205 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:30:51.204+0000 7f7a2713b640 4 set_mon_vals no callback set 2024-03-25T18:30:51.206 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:30:51.205+0000 7f7a2713b640 10 monclient: shutdown 2024-03-25T18:30:51.206 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:30:51.205+0000 7f7a2713b640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:30:51.206 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:30:51.205+0000 7f7a2713b640 1 -- 172.21.15.1:0/1715011704 >> v1:172.21.15.1:6790/0 conn(0x7f7a200af5d0 legacy=0x7f7a20145b10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:30:51.206 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:30:51.205+0000 7f7a2713b640 1 -- 172.21.15.1:0/1715011704 shutdown_connections 2024-03-25T18:30:51.206 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:30:51.205+0000 7f7a2713b640 1 -- 172.21.15.1:0/1715011704 >> 172.21.15.1:0/1715011704 conn(0x7f7a200b4bd0 msgr2=0x7f7a200b4fe0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:30:51.207 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:30:51.205+0000 7f7a2713b640 1 -- 172.21.15.1:0/1715011704 shutdown_connections 2024-03-25T18:30:51.207 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:30:51.205+0000 7f7a2713b640 1 -- 172.21.15.1:0/1715011704 wait complete. 2024-03-25T18:30:51.208 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:30:51.206+0000 7f7a2713b640 10 monclient: build_initial_monmap 2024-03-25T18:30:51.208 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:30:51.206+0000 7f7a2713b640 10 monclient: monmap: 2024-03-25T18:30:51.208 INFO:teuthology.misc.health.smithi001.stderr:epoch 0 2024-03-25T18:30:51.208 INFO:teuthology.misc.health.smithi001.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T18:30:51.208 INFO:teuthology.misc.health.smithi001.stderr:last_changed 0.000000 2024-03-25T18:30:51.208 INFO:teuthology.misc.health.smithi001.stderr:created 0.000000 2024-03-25T18:30:51.208 INFO:teuthology.misc.health.smithi001.stderr:min_mon_release 0 (unknown) 2024-03-25T18:30:51.208 INFO:teuthology.misc.health.smithi001.stderr:election_strategy: 1 2024-03-25T18:30:51.208 INFO:teuthology.misc.health.smithi001.stderr:0: v1:172.21.15.1:6789/0 mon.noname-a 2024-03-25T18:30:51.208 INFO:teuthology.misc.health.smithi001.stderr:1: v1:172.21.15.115:6789/0 mon.noname-b 2024-03-25T18:30:51.208 INFO:teuthology.misc.health.smithi001.stderr:2: v1:172.21.15.1:6790/0 mon.noname-c 2024-03-25T18:30:51.209 INFO:teuthology.misc.health.smithi001.stderr: 2024-03-25T18:30:51.209 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:30:51.207+0000 7f7a2713b640 1 librados: starting msgr at 2024-03-25T18:30:51.209 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:30:51.207+0000 7f7a2713b640 1 librados: starting objecter 2024-03-25T18:30:51.209 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:30:51.207+0000 7f7a2713b640 1 Processor -- start 2024-03-25T18:30:51.209 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:30:51.207+0000 7f7a2713b640 1 -- start start 2024-03-25T18:30:51.209 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:30:51.207+0000 7f7a2713b640 1 librados: setting wanted keys 2024-03-25T18:30:51.209 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:30:51.207+0000 7f7a2713b640 1 librados: calling monclient init 2024-03-25T18:30:51.209 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:30:51.207+0000 7f7a2713b640 10 monclient: init 2024-03-25T18:30:51.210 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:30:51.207+0000 7f7a2713b640 10 monclient: _reopen_session rank -1 2024-03-25T18:30:51.210 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:30:51.207+0000 7f7a2713b640 10 monclient: _add_conns ranks=[0,1,2] 2024-03-25T18:30:51.210 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:30:51.207+0000 7f7a2713b640 10 monclient(hunting): picked mon.noname-a con 0x7f7a200af5d0 addr v1:172.21.15.1:6789/0 2024-03-25T18:30:51.210 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:30:51.207+0000 7f7a2713b640 10 monclient(hunting): picked mon.noname-b con 0x7f7a200b0220 addr v1:172.21.15.115:6789/0 2024-03-25T18:30:51.210 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:30:51.208+0000 7f7a2713b640 10 monclient(hunting): picked mon.noname-c con 0x7f7a20142e80 addr v1:172.21.15.1:6790/0 2024-03-25T18:30:51.211 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:30:51.208+0000 7f7a2713b640 1 -- --> v1:172.21.15.1:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7a20140df0 con 0x7f7a200af5d0 2024-03-25T18:30:51.211 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:30:51.208+0000 7f7a2713b640 1 -- --> v1:172.21.15.115:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7a20141280 con 0x7f7a200b0220 2024-03-25T18:30:51.211 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:30:51.208+0000 7f7a2713b640 1 -- --> v1:172.21.15.1:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7a200708d0 con 0x7f7a20142e80 2024-03-25T18:30:51.211 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:30:51.208+0000 7f7a2713b640 10 monclient(hunting): _renew_subs 2024-03-25T18:30:51.211 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:30:51.208+0000 7f7a2713b640 10 monclient(hunting): authenticate will time out at 7731.677735s 2024-03-25T18:30:51.213 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:30:51.213+0000 7f7a24ed7640 1 --1- >> v1:172.21.15.1:6789/0 conn(0x7f7a200af5d0 0x7f7a2006f120 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.1:6789/0 says I am v1:172.21.15.1:53636/0 (socket says 172.21.15.1:53636) 2024-03-25T18:30:51.213 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:30:51.213+0000 7f7a24ed7640 1 -- 172.21.15.1:0/1937796878 learned_addr learned my addr 172.21.15.1:0/1937796878 (peer_addr_for_me v1:172.21.15.1:0/0) 2024-03-25T18:30:51.217 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:30:51.217+0000 7f7a1d7fa640 10 client.?.objecter ms_handle_connect 0x7f7a200af5d0 2024-03-25T18:30:51.217 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:30:51.217+0000 7f7a1d7fa640 10 client.?.objecter resend_mon_ops 2024-03-25T18:30:51.227 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:30:51.226+0000 7f7a1d7fa640 10 client.?.objecter ms_handle_connect 0x7f7a200b0220 2024-03-25T18:30:51.227 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:30:51.226+0000 7f7a1d7fa640 10 client.?.objecter resend_mon_ops 2024-03-25T18:30:51.233 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:30:51.232+0000 7f7a1d7fa640 1 -- 172.21.15.1:0/1937796878 <== mon.0 v1:172.21.15.1:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 255195012 0 0) 0x7f7a20140df0 con 0x7f7a200af5d0 2024-03-25T18:30:51.233 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:30:51.232+0000 7f7a1d7fa640 10 monclient(hunting): _negotiate 2024-03-25T18:30:51.233 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:30:51.232+0000 7f7a1d7fa640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:30:51.233 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:30:51.232+0000 7f7a1d7fa640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:30:51.233 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:30:51.232+0000 7f7a1d7fa640 10 monclient(hunting): my global_id is 4398 2024-03-25T18:30:51.233 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:30:51.232+0000 7f7a1d7fa640 1 -- 172.21.15.1:0/1937796878 --> v1:172.21.15.1:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f7a04004eb0 con 0x7f7a200af5d0 2024-03-25T18:30:51.237 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:30:51.236+0000 7f7a1d7fa640 10 client.?.objecter ms_handle_connect 0x7f7a20142e80 2024-03-25T18:30:51.237 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:30:51.236+0000 7f7a1d7fa640 10 client.?.objecter resend_mon_ops 2024-03-25T18:30:51.238 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:30:51.237+0000 7f7a1d7fa640 1 -- 172.21.15.1:0/1937796878 <== mon.1 v1:172.21.15.115:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1762879320 0 0) 0x7f7a20141280 con 0x7f7a200b0220 2024-03-25T18:30:51.238 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:30:51.237+0000 7f7a1d7fa640 10 monclient(hunting): _negotiate 2024-03-25T18:30:51.238 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:30:51.237+0000 7f7a1d7fa640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:30:51.239 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:30:51.237+0000 7f7a1d7fa640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:30:51.239 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:30:51.237+0000 7f7a1d7fa640 10 monclient(hunting): my global_id is 4405 2024-03-25T18:30:51.239 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:30:51.237+0000 7f7a1d7fa640 1 -- 172.21.15.1:0/1937796878 --> v1:172.21.15.115:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f7a20140df0 con 0x7f7a200b0220 2024-03-25T18:30:51.249 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:30:51.248+0000 7f7a1d7fa640 1 -- 172.21.15.1:0/1937796878 <== mon.1 v1:172.21.15.115:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2932914015 0 0) 0x7f7a20140df0 con 0x7f7a200b0220 2024-03-25T18:30:51.249 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:30:51.248+0000 7f7a1d7fa640 1 -- 172.21.15.1:0/1937796878 --> v1:172.21.15.115:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f7a20141280 con 0x7f7a200b0220 2024-03-25T18:30:51.249 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:30:51.248+0000 7f7a1d7fa640 1 -- 172.21.15.1:0/1937796878 <== mon.1 v1:172.21.15.115:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1586965025 0 0) 0x7f7a0c001550 con 0x7f7a200b0220 2024-03-25T18:30:51.249 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:30:51.248+0000 7f7a1d7fa640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:30:51.249 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:30:51.248+0000 7f7a1d7fa640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-03-25T18:30:51.249 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:30:51.248+0000 7f7a1d7fa640 10 monclient(hunting): dump: 2024-03-25T18:30:51.249 INFO:teuthology.misc.health.smithi001.stderr:epoch 1 2024-03-25T18:30:51.249 INFO:teuthology.misc.health.smithi001.stderr:fsid c8ea72cf-9e99-4383-9b52-6c23d365fdfc 2024-03-25T18:30:51.249 INFO:teuthology.misc.health.smithi001.stderr:last_changed 2024-03-25T18:25:50.510519+0000 2024-03-25T18:30:51.249 INFO:teuthology.misc.health.smithi001.stderr:created 2024-03-25T18:25:50.510519+0000 2024-03-25T18:30:51.249 INFO:teuthology.misc.health.smithi001.stderr:min_mon_release 19 (squid) 2024-03-25T18:30:51.249 INFO:teuthology.misc.health.smithi001.stderr:election_strategy: 3 2024-03-25T18:30:51.250 INFO:teuthology.misc.health.smithi001.stderr:0: v1:172.21.15.1:6789/0 mon.a 2024-03-25T18:30:51.250 INFO:teuthology.misc.health.smithi001.stderr:1: v1:172.21.15.115:6789/0 mon.b 2024-03-25T18:30:51.250 INFO:teuthology.misc.health.smithi001.stderr:2: v1:172.21.15.1:6790/0 mon.c 2024-03-25T18:30:51.250 INFO:teuthology.misc.health.smithi001.stderr: 2024-03-25T18:30:51.250 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:30:51.248+0000 7f7a1d7fa640 10 monclient(hunting): _finish_auth 0 2024-03-25T18:30:51.252 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:30:51.252+0000 7f7a1d7fa640 1 -- 172.21.15.1:0/1937796878 <== mon.0 v1:172.21.15.1:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 4127154273 0 0) 0x7f7a04004eb0 con 0x7f7a200af5d0 2024-03-25T18:30:51.253 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:30:51.252+0000 7f7a1d7fa640 1 -- 172.21.15.1:0/1937796878 --> v1:172.21.15.1:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f7a20140df0 con 0x7f7a200af5d0 2024-03-25T18:30:51.253 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:30:51.252+0000 7f7a1d7fa640 1 -- 172.21.15.1:0/1937796878 <== mon.0 v1:172.21.15.1:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1586965025 0 0) 0x7f7a140017d0 con 0x7f7a200af5d0 2024-03-25T18:30:51.253 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:30:51.252+0000 7f7a1d7fa640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:30:51.253 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:30:51.252+0000 7f7a1d7fa640 10 monclient(hunting): got monmap 1 from mon.a (according to old e1) 2024-03-25T18:30:51.253 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:30:51.252+0000 7f7a1d7fa640 10 monclient(hunting): dump: 2024-03-25T18:30:51.253 INFO:teuthology.misc.health.smithi001.stderr:epoch 1 2024-03-25T18:30:51.253 INFO:teuthology.misc.health.smithi001.stderr:fsid c8ea72cf-9e99-4383-9b52-6c23d365fdfc 2024-03-25T18:30:51.253 INFO:teuthology.misc.health.smithi001.stderr:last_changed 2024-03-25T18:25:50.510519+0000 2024-03-25T18:30:51.253 INFO:teuthology.misc.health.smithi001.stderr:created 2024-03-25T18:25:50.510519+0000 2024-03-25T18:30:51.253 INFO:teuthology.misc.health.smithi001.stderr:min_mon_release 19 (squid) 2024-03-25T18:30:51.253 INFO:teuthology.misc.health.smithi001.stderr:election_strategy: 3 2024-03-25T18:30:51.253 INFO:teuthology.misc.health.smithi001.stderr:0: v1:172.21.15.1:6789/0 mon.a 2024-03-25T18:30:51.253 INFO:teuthology.misc.health.smithi001.stderr:1: v1:172.21.15.115:6789/0 mon.b 2024-03-25T18:30:51.253 INFO:teuthology.misc.health.smithi001.stderr:2: v1:172.21.15.1:6790/0 mon.c 2024-03-25T18:30:51.253 INFO:teuthology.misc.health.smithi001.stderr: 2024-03-25T18:30:51.256 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:30:51.256+0000 7f7a1d7fa640 1 -- 172.21.15.1:0/1937796878 <== mon.2 v1:172.21.15.1:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2994209903 0 0) 0x7f7a200708d0 con 0x7f7a20142e80 2024-03-25T18:30:51.256 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:30:51.256+0000 7f7a1d7fa640 10 monclient(hunting): _negotiate 2024-03-25T18:30:51.257 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:30:51.256+0000 7f7a1d7fa640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:30:51.257 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:30:51.256+0000 7f7a1d7fa640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:30:51.257 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:30:51.256+0000 7f7a1d7fa640 10 monclient(hunting): my global_id is 4409 2024-03-25T18:30:51.257 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:30:51.256+0000 7f7a1d7fa640 1 -- 172.21.15.1:0/1937796878 --> v1:172.21.15.1:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f7a04004eb0 con 0x7f7a20142e80 2024-03-25T18:30:51.260 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:30:51.259+0000 7f7a1d7fa640 1 -- 172.21.15.1:0/1937796878 <== mon.1 v1:172.21.15.115:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 656716293 0 0) 0x7f7a20141280 con 0x7f7a200b0220 2024-03-25T18:30:51.260 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:30:51.259+0000 7f7a1d7fa640 1 -- 172.21.15.1:0/1937796878 >> v1:172.21.15.1:6790/0 conn(0x7f7a20142e80 legacy=0x7f7a20143630 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:30:51.260 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:30:51.259+0000 7f7a1d7fa640 1 -- 172.21.15.1:0/1937796878 >> v1:172.21.15.1:6789/0 conn(0x7f7a200af5d0 legacy=0x7f7a2006f120 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:30:51.261 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:30:51.259+0000 7f7a1d7fa640 10 monclient: _finish_hunting 0 2024-03-25T18:30:51.261 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:30:51.259+0000 7f7a1d7fa640 1 monclient: found mon.b 2024-03-25T18:30:51.261 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:30:51.259+0000 7f7a1d7fa640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:30:51.261 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:30:51.259+0000 7f7a1d7fa640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.115:6789/0 2024-03-25T18:30:51.261 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:30:51.259+0000 7f7a1d7fa640 1 -- 172.21.15.1:0/1937796878 --> v1:172.21.15.115:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7a2006f830 con 0x7f7a200b0220 2024-03-25T18:30:51.261 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:30:51.259+0000 7f7a1d7fa640 10 monclient: _finish_auth 0 2024-03-25T18:30:51.261 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:30:51.259+0000 7f7a1d7fa640 10 monclient: _check_auth_tickets 2024-03-25T18:30:51.261 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:30:51.259+0000 7f7a1d7fa640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:30:51.261 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:30:51.259+0000 7f7a2713b640 5 monclient: authenticate success, global_id 4405 2024-03-25T18:30:51.261 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:30:51.259+0000 7f7a2713b640 10 monclient: _renew_subs 2024-03-25T18:30:51.261 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:30:51.259+0000 7f7a2713b640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.115:6789/0 2024-03-25T18:30:51.261 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:30:51.259+0000 7f7a2713b640 1 -- 172.21.15.1:0/1937796878 --> v1:172.21.15.115:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7a20133500 con 0x7f7a200b0220 2024-03-25T18:30:51.261 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:30:51.259+0000 7f7a2713b640 10 client.4405.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T18:30:51.261 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:30:51.259+0000 7f7a2713b640 10 monclient: _renew_subs 2024-03-25T18:30:51.261 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:30:51.259+0000 7f7a2713b640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.115:6789/0 2024-03-25T18:30:51.262 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:30:51.259+0000 7f7a2713b640 1 -- 172.21.15.1:0/1937796878 --> v1:172.21.15.115:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f7a20145280 con 0x7f7a200b0220 2024-03-25T18:30:51.262 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:30:51.259+0000 7f7a2713b640 1 librados: init done 2024-03-25T18:30:51.262 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:30:51.260+0000 7f7a2713b640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T18:30:51.262 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:30:51.260+0000 7f7a2713b640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:30:51.262 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:30:51.260+0000 7f7a2713b640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.115:6789/0 2024-03-25T18:30:51.262 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:30:51.260+0000 7f7a2713b640 1 -- 172.21.15.1:0/1937796878 --> v1:172.21.15.115:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7a20140bc0 con 0x7f7a200b0220 2024-03-25T18:30:51.281 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:30:51.280+0000 7f7a1d7fa640 1 -- 172.21.15.1:0/1937796878 <== mon.1 v1:172.21.15.115:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f7a0c004a90 con 0x7f7a200b0220 2024-03-25T18:30:51.281 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:30:51.280+0000 7f7a1d7fa640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:30:51.281 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:30:51.280+0000 7f7a1d7fa640 1 -- 172.21.15.1:0/1937796878 <== mon.1 v1:172.21.15.115:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1586965025 0 0) 0x7f7a0c001250 con 0x7f7a200b0220 2024-03-25T18:30:51.281 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:30:51.280+0000 7f7a1d7fa640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:30:51.281 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:30:51.280+0000 7f7a1d7fa640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-25T18:30:51.281 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:30:51.280+0000 7f7a1d7fa640 10 monclient: dump: 2024-03-25T18:30:51.281 INFO:teuthology.misc.health.smithi001.stderr:epoch 1 2024-03-25T18:30:51.281 INFO:teuthology.misc.health.smithi001.stderr:fsid c8ea72cf-9e99-4383-9b52-6c23d365fdfc 2024-03-25T18:30:51.281 INFO:teuthology.misc.health.smithi001.stderr:last_changed 2024-03-25T18:25:50.510519+0000 2024-03-25T18:30:51.281 INFO:teuthology.misc.health.smithi001.stderr:created 2024-03-25T18:25:50.510519+0000 2024-03-25T18:30:51.281 INFO:teuthology.misc.health.smithi001.stderr:min_mon_release 19 (squid) 2024-03-25T18:30:51.281 INFO:teuthology.misc.health.smithi001.stderr:election_strategy: 3 2024-03-25T18:30:51.282 INFO:teuthology.misc.health.smithi001.stderr:0: v1:172.21.15.1:6789/0 mon.a 2024-03-25T18:30:51.282 INFO:teuthology.misc.health.smithi001.stderr:1: v1:172.21.15.115:6789/0 mon.b 2024-03-25T18:30:51.282 INFO:teuthology.misc.health.smithi001.stderr:2: v1:172.21.15.1:6790/0 mon.c 2024-03-25T18:30:51.282 INFO:teuthology.misc.health.smithi001.stderr: 2024-03-25T18:30:51.282 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:30:51.280+0000 7f7a1effd640 4 set_mon_vals no callback set 2024-03-25T18:30:51.282 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:30:51.281+0000 7f7a1d7fa640 1 -- 172.21.15.1:0/1937796878 <== mon.1 v1:172.21.15.115:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 21787975 0 0) 0x7f7a0c01aa70 con 0x7f7a200b0220 2024-03-25T18:30:51.282 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:30:51.282+0000 7f7a24ed7640 10 monclient: get_auth_request con 0x7f7a0406df00 auth_method 0 2024-03-25T18:30:51.283 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:30:51.282+0000 7f7a1d7fa640 10 client.4405.objecter ms_handle_connect 0x7f7a0406df00 2024-03-25T18:30:51.283 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:30:51.283+0000 7f7a1d7fa640 1 -- 172.21.15.1:0/1937796878 <== mon.1 v1:172.21.15.115:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 996254230 0 0) 0x7f7a0c054690 con 0x7f7a200b0220 2024-03-25T18:30:51.283 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:30:51.283+0000 7f7a1d7fa640 10 client.4405.objecter ms_dispatch 0x7f7a20000b90 osd_map(10..10 src has 1..10) v4 2024-03-25T18:30:51.283 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:30:51.283+0000 7f7a1d7fa640 3 client.4405.objecter handle_osd_map got epochs [10,10] > 0 2024-03-25T18:30:51.284 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:30:51.283+0000 7f7a1d7fa640 3 client.4405.objecter handle_osd_map decoding full epoch 10 2024-03-25T18:30:51.284 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:30:51.283+0000 7f7a1d7fa640 20 client.4405.objecter dump_active .. 0 homeless 2024-03-25T18:30:51.527 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:30:51.526+0000 7f7a1d7fa640 1 -- 172.21.15.1:0/1937796878 <== mon.1 v1:172.21.15.115:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f7a0c0501a0 con 0x7f7a200b0220 2024-03-25T18:30:51.527 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:30:51.526+0000 7f7a1d7fa640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:30:51.527 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:30:51.526+0000 7f7a1d7fa640 10 monclient: _finish_command 1 = system:0 2024-03-25T18:30:51.794 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:30:51.793+0000 7f7a2713b640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:30:51.794 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:30:51.793+0000 7f7a2713b640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:30:51.794 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:30:51.793+0000 7f7a2713b640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.115:6789/0 2024-03-25T18:30:51.794 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:30:51.793+0000 7f7a2713b640 1 -- 172.21.15.1:0/1937796878 --> v1:172.21.15.115:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f7a20140fa0 con 0x7f7a200b0220 2024-03-25T18:30:51.898 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:30:51.897+0000 7f7a1d7fa640 1 -- 172.21.15.1:0/1937796878 <== mon.1 v1:172.21.15.115:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 4092213348) 0x7f7a20140fa0 con 0x7f7a200b0220 2024-03-25T18:30:51.898 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:30:51.897+0000 7f7a1d7fa640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:30:51.898 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:30:51.897+0000 7f7a1d7fa640 10 monclient: _finish_command 2 = system:0 2024-03-25T18:30:51.898 INFO:teuthology.misc.health.smithi001.stdout: 2024-03-25T18:30:51.899 INFO:teuthology.misc.health.smithi001.stdout:{"epoch":10,"fsid":"c8ea72cf-9e99-4383-9b52-6c23d365fdfc","created":"2024-03-25T18:26:38.856638+0000","modified":"2024-03-25T18:26:44.224208+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T18:26:44.224208+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"b9f34922-3a84-49be-8426-807c1eceaefc","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"42bf306f-55e3-4634-9874-8b4b73ac6644","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"292cb23a-18a9-4207-a7cc-4d5727feaec0","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"45e0e9ab-898d-4673-9590-b4117bc5e84a","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"831a5a09-73ab-4a61-9872-088e76bdbc66","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"88cd903d-3b55-4840-9396-6e3cfe005282","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"89db7355-bc9d-46f4-8ab0-e8d9e744e026","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"cac47bee-0914-4f08-896f-348838882866","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T18:30:51.901 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:30:51.899+0000 7f7a2713b640 10 librados: watch_flush enter 2024-03-25T18:30:51.901 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:30:51.899+0000 7f7a2713b640 10 librados: watch_flush exit 2024-03-25T18:30:51.901 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:30:51.899+0000 7f7a2713b640 20 client.4405.objecter shutdown clearing up homeless session... 2024-03-25T18:30:51.901 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:30:51.899+0000 7f7a2713b640 10 client.4405.objecter successfully canceled tick 2024-03-25T18:30:51.901 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:30:51.900+0000 7f7a2713b640 1 -- 172.21.15.1:0/1937796878 >> v1:172.21.15.115:6800/4237379176 conn(0x7f7a0406df00 legacy=0x7f7a040703c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:30:51.901 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:30:51.900+0000 7f7a2713b640 10 monclient: shutdown 2024-03-25T18:30:51.901 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:30:51.900+0000 7f7a2713b640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:30:51.901 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:30:51.900+0000 7f7a2713b640 1 -- 172.21.15.1:0/1937796878 >> v1:172.21.15.115:6789/0 conn(0x7f7a200b0220 legacy=0x7f7a20132df0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:30:51.902 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:30:51.900+0000 7f7a2713b640 1 -- 172.21.15.1:0/1937796878 shutdown_connections 2024-03-25T18:30:51.902 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:30:51.900+0000 7f7a2713b640 1 -- 172.21.15.1:0/1937796878 >> 172.21.15.1:0/1937796878 conn(0x7f7a200b4bd0 msgr2=0x7f7a20145b10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:30:51.902 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:30:51.900+0000 7f7a2713b640 1 -- 172.21.15.1:0/1937796878 shutdown_connections 2024-03-25T18:30:51.902 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:30:51.900+0000 7f7a2713b640 1 -- 172.21.15.1:0/1937796878 wait complete. 2024-03-25T18:30:51.902 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:30:51.900+0000 7f7a2713b640 1 librados: shutdown 2024-03-25T18:30:51.913 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T18:30:56.640 INFO:tasks.ceph.mon.a.smithi001.stderr:==00:00:04:35.198 34876== Warning: unimplemented fcntl command: 1036 2024-03-25T18:30:56.650 INFO:tasks.ceph.mon.a.smithi001.stderr:==00:00:04:35.209 34876== Warning: unimplemented fcntl command: 1036 2024-03-25T18:30:56.687 INFO:tasks.ceph.mon.a.smithi001.stderr:==00:00:04:35.245 34876== Warning: unimplemented fcntl command: 1036 2024-03-25T18:30:56.704 INFO:tasks.ceph.mon.b.smithi115.stderr:==00:00:04:35.188 34836== Warning: unimplemented fcntl command: 1036 2024-03-25T18:30:56.710 INFO:tasks.ceph.mon.b.smithi115.stderr:==00:00:04:35.194 34836== Warning: unimplemented fcntl command: 1036 2024-03-25T18:30:56.714 INFO:tasks.ceph.mon.c.smithi001.stderr:==00:00:04:35.272 34877== Warning: unimplemented fcntl command: 1036 2024-03-25T18:30:56.720 INFO:tasks.ceph.mon.c.smithi001.stderr:==00:00:04:35.278 34877== Warning: unimplemented fcntl command: 1036 2024-03-25T18:30:56.731 INFO:tasks.ceph.mon.b.smithi115.stderr:==00:00:04:35.214 34836== Warning: unimplemented fcntl command: 1036 2024-03-25T18:30:56.745 INFO:tasks.ceph.mon.c.smithi001.stderr:==00:00:04:35.303 34877== Warning: unimplemented fcntl command: 1036 2024-03-25T18:30:58.720 DEBUG:teuthology.orchestra.run.smithi001:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T18:30:58.837 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:30:58.836+0000 7f00ba758640 10 monclient: get_monmap_and_config 2024-03-25T18:30:58.838 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:30:58.836+0000 7f00ba758640 10 monclient: build_initial_monmap 2024-03-25T18:30:58.838 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:30:58.836+0000 7f00ba758640 10 monclient: monmap: 2024-03-25T18:30:58.838 INFO:teuthology.misc.health.smithi001.stderr:epoch 0 2024-03-25T18:30:58.838 INFO:teuthology.misc.health.smithi001.stderr:fsid c8ea72cf-9e99-4383-9b52-6c23d365fdfc 2024-03-25T18:30:58.838 INFO:teuthology.misc.health.smithi001.stderr:last_changed 2024-03-25T18:30:58.837343+0000 2024-03-25T18:30:58.838 INFO:teuthology.misc.health.smithi001.stderr:created 2024-03-25T18:30:58.837343+0000 2024-03-25T18:30:58.838 INFO:teuthology.misc.health.smithi001.stderr:min_mon_release 0 (unknown) 2024-03-25T18:30:58.838 INFO:teuthology.misc.health.smithi001.stderr:election_strategy: 3 2024-03-25T18:30:58.838 INFO:teuthology.misc.health.smithi001.stderr:0: v1:172.21.15.1:6789/0 mon.noname-a 2024-03-25T18:30:58.838 INFO:teuthology.misc.health.smithi001.stderr:1: v1:172.21.15.115:6789/0 mon.noname-c 2024-03-25T18:30:58.839 INFO:teuthology.misc.health.smithi001.stderr:2: v1:172.21.15.1:6790/0 mon.noname-b 2024-03-25T18:30:58.839 INFO:teuthology.misc.health.smithi001.stderr: 2024-03-25T18:30:58.840 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:30:58.837+0000 7f00ba758640 1 Processor -- start 2024-03-25T18:30:58.840 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:30:58.837+0000 7f00ba758640 1 -- start start 2024-03-25T18:30:58.840 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:30:58.837+0000 7f00ba758640 10 monclient: init 2024-03-25T18:30:58.840 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:30:58.838+0000 7f00ba758640 10 monclient: _reopen_session rank -1 2024-03-25T18:30:58.840 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:30:58.838+0000 7f00ba758640 10 monclient: _add_conns ranks=[1,0,2] 2024-03-25T18:30:58.840 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:30:58.838+0000 7f00ba758640 10 monclient(hunting): picked mon.noname-c con 0x7f00b40b4ff0 addr v1:172.21.15.115:6789/0 2024-03-25T18:30:58.840 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:30:58.838+0000 7f00ba758640 10 monclient(hunting): picked mon.noname-a con 0x7f00b40b4670 addr v1:172.21.15.1:6789/0 2024-03-25T18:30:58.840 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:30:58.838+0000 7f00ba758640 10 monclient(hunting): picked mon.noname-b con 0x7f00b413c480 addr v1:172.21.15.1:6790/0 2024-03-25T18:30:58.840 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:30:58.838+0000 7f00ba758640 1 -- --> v1:172.21.15.1:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f00b40ab0e0 con 0x7f00b40b4670 2024-03-25T18:30:58.840 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:30:58.838+0000 7f00ba758640 1 -- --> v1:172.21.15.115:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f00b40ab3c0 con 0x7f00b40b4ff0 2024-03-25T18:30:58.840 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:30:58.838+0000 7f00ba758640 1 -- --> v1:172.21.15.1:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f00b40a9f30 con 0x7f00b413c480 2024-03-25T18:30:58.841 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:30:58.838+0000 7f00ba758640 10 monclient(hunting): _renew_subs 2024-03-25T18:30:58.841 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:30:58.838+0000 7f00ba758640 10 monclient(hunting): authenticate will time out at 7739.308065s 2024-03-25T18:30:58.842 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:30:58.842+0000 7f00b8cf5640 1 --1- >> v1:172.21.15.1:6790/0 conn(0x7f00b413c480 0x7f00b414dc00 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.1:6790/0 says I am v1:172.21.15.1:34700/0 (socket says 172.21.15.1:34700) 2024-03-25T18:30:58.842 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:30:58.842+0000 7f00b8cf5640 1 -- 172.21.15.1:0/4140534694 learned_addr learned my addr 172.21.15.1:0/4140534694 (peer_addr_for_me v1:172.21.15.1:0/0) 2024-03-25T18:30:58.926 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:30:58.925+0000 7f00b2ffd640 1 -- 172.21.15.1:0/4140534694 <== mon.0 v1:172.21.15.1:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3852499864 0 0) 0x7f00b40ab0e0 con 0x7f00b40b4670 2024-03-25T18:30:58.926 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:30:58.925+0000 7f00b2ffd640 10 monclient(hunting): _negotiate 2024-03-25T18:30:58.926 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:30:58.925+0000 7f00b2ffd640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:30:58.926 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:30:58.925+0000 7f00b2ffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:30:58.926 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:30:58.925+0000 7f00b2ffd640 10 monclient(hunting): my global_id is 4401 2024-03-25T18:30:58.926 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:30:58.925+0000 7f00b2ffd640 1 -- 172.21.15.1:0/4140534694 --> v1:172.21.15.1:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f009c003520 con 0x7f00b40b4670 2024-03-25T18:30:58.956 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:30:58.956+0000 7f00b2ffd640 1 -- 172.21.15.1:0/4140534694 <== mon.1 v1:172.21.15.115:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2161182884 0 0) 0x7f00b40ab3c0 con 0x7f00b40b4ff0 2024-03-25T18:30:58.957 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:30:58.956+0000 7f00b2ffd640 10 monclient(hunting): _negotiate 2024-03-25T18:30:58.957 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:30:58.956+0000 7f00b2ffd640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:30:58.957 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:30:58.956+0000 7f00b2ffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:30:58.957 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:30:58.956+0000 7f00b2ffd640 10 monclient(hunting): my global_id is 4408 2024-03-25T18:30:58.957 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:30:58.956+0000 7f00b2ffd640 1 -- 172.21.15.1:0/4140534694 --> v1:172.21.15.115:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f00b40ab0e0 con 0x7f00b40b4ff0 2024-03-25T18:30:58.960 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:30:58.959+0000 7f00b2ffd640 1 -- 172.21.15.1:0/4140534694 <== mon.0 v1:172.21.15.1:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3927435554 0 0) 0x7f009c003520 con 0x7f00b40b4670 2024-03-25T18:30:58.960 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:30:58.959+0000 7f00b2ffd640 1 -- 172.21.15.1:0/4140534694 >> v1:172.21.15.1:6790/0 conn(0x7f00b413c480 legacy=0x7f00b414dc00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:30:58.961 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:30:58.959+0000 7f00b2ffd640 1 -- 172.21.15.1:0/4140534694 >> v1:172.21.15.115:6789/0 conn(0x7f00b40b4ff0 legacy=0x7f00b413bd70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:30:58.961 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:30:58.959+0000 7f00b2ffd640 10 monclient: _finish_hunting 0 2024-03-25T18:30:58.961 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:30:58.959+0000 7f00b2ffd640 1 monclient: found mon.noname-a 2024-03-25T18:30:58.961 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:30:58.959+0000 7f00b2ffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:30:58.961 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:30:58.959+0000 7f00b2ffd640 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.1:6789/0 2024-03-25T18:30:58.961 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:30:58.959+0000 7f00b2ffd640 1 -- 172.21.15.1:0/4140534694 --> v1:172.21.15.1:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f00b40a9b70 con 0x7f00b40b4670 2024-03-25T18:30:58.961 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:30:58.959+0000 7f00b2ffd640 10 monclient: _finish_auth 0 2024-03-25T18:30:58.961 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:30:58.959+0000 7f00b2ffd640 10 monclient: _check_auth_tickets 2024-03-25T18:30:58.961 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:30:58.959+0000 7f00b2ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:30:58.961 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:30:58.959+0000 7f00b2ffd640 1 -- 172.21.15.1:0/4140534694 <== mon.0 v1:172.21.15.1:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1586965025 0 0) 0x7f00a8001690 con 0x7f00b40b4670 2024-03-25T18:30:58.961 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:30:58.959+0000 7f00ba758640 5 monclient: authenticate success, global_id 4401 2024-03-25T18:30:58.961 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:30:58.959+0000 7f00b2ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:30:58.961 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:30:58.959+0000 7f00b2ffd640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-03-25T18:30:58.962 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:30:58.959+0000 7f00b2ffd640 10 monclient: dump: 2024-03-25T18:30:58.962 INFO:teuthology.misc.health.smithi001.stderr:epoch 1 2024-03-25T18:30:58.962 INFO:teuthology.misc.health.smithi001.stderr:fsid c8ea72cf-9e99-4383-9b52-6c23d365fdfc 2024-03-25T18:30:58.962 INFO:teuthology.misc.health.smithi001.stderr:last_changed 2024-03-25T18:25:50.510519+0000 2024-03-25T18:30:58.962 INFO:teuthology.misc.health.smithi001.stderr:created 2024-03-25T18:25:50.510519+0000 2024-03-25T18:30:58.962 INFO:teuthology.misc.health.smithi001.stderr:min_mon_release 19 (squid) 2024-03-25T18:30:58.962 INFO:teuthology.misc.health.smithi001.stderr:election_strategy: 3 2024-03-25T18:30:58.962 INFO:teuthology.misc.health.smithi001.stderr:0: v1:172.21.15.1:6789/0 mon.a 2024-03-25T18:30:58.962 INFO:teuthology.misc.health.smithi001.stderr:1: v1:172.21.15.115:6789/0 mon.b 2024-03-25T18:30:58.962 INFO:teuthology.misc.health.smithi001.stderr:2: v1:172.21.15.1:6790/0 mon.c 2024-03-25T18:30:58.962 INFO:teuthology.misc.health.smithi001.stderr: 2024-03-25T18:30:58.963 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:30:58.959+0000 7f00ba758640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T18:30:58.971 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:30:58.970+0000 7f00b2ffd640 1 -- 172.21.15.1:0/4140534694 <== mon.0 v1:172.21.15.1:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f00a80017f0 con 0x7f00b40b4670 2024-03-25T18:30:58.971 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:30:58.970+0000 7f00b2ffd640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:30:58.971 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:30:58.970+0000 7f00b2ffd640 1 -- 172.21.15.1:0/4140534694 <== mon.0 v1:172.21.15.1:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1586965025 0 0) 0x7f00a8001c00 con 0x7f00b40b4670 2024-03-25T18:30:58.971 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:30:58.970+0000 7f00b2ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:30:58.971 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:30:58.970+0000 7f00b2ffd640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-25T18:30:58.971 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:30:58.970+0000 7f00b2ffd640 10 monclient: dump: 2024-03-25T18:30:58.972 INFO:teuthology.misc.health.smithi001.stderr:epoch 1 2024-03-25T18:30:58.972 INFO:teuthology.misc.health.smithi001.stderr:fsid c8ea72cf-9e99-4383-9b52-6c23d365fdfc 2024-03-25T18:30:58.972 INFO:teuthology.misc.health.smithi001.stderr:last_changed 2024-03-25T18:25:50.510519+0000 2024-03-25T18:30:58.972 INFO:teuthology.misc.health.smithi001.stderr:created 2024-03-25T18:25:50.510519+0000 2024-03-25T18:30:58.972 INFO:teuthology.misc.health.smithi001.stderr:min_mon_release 19 (squid) 2024-03-25T18:30:58.972 INFO:teuthology.misc.health.smithi001.stderr:election_strategy: 3 2024-03-25T18:30:58.972 INFO:teuthology.misc.health.smithi001.stderr:0: v1:172.21.15.1:6789/0 mon.a 2024-03-25T18:30:58.972 INFO:teuthology.misc.health.smithi001.stderr:1: v1:172.21.15.115:6789/0 mon.b 2024-03-25T18:30:58.972 INFO:teuthology.misc.health.smithi001.stderr:2: v1:172.21.15.1:6790/0 mon.c 2024-03-25T18:30:58.972 INFO:teuthology.misc.health.smithi001.stderr: 2024-03-25T18:30:58.972 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:30:58.970+0000 7f00ba758640 10 monclient: get_monmap_and_config success 2024-03-25T18:30:58.972 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:30:58.970+0000 7f00ba758640 4 set_mon_vals no callback set 2024-03-25T18:30:58.975 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:30:58.971+0000 7f00ba758640 10 monclient: shutdown 2024-03-25T18:30:58.975 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:30:58.971+0000 7f00ba758640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:30:58.975 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:30:58.971+0000 7f00ba758640 1 -- 172.21.15.1:0/4140534694 >> v1:172.21.15.1:6789/0 conn(0x7f00b40b4670 legacy=0x7f00b41425c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:30:58.975 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:30:58.971+0000 7f00ba758640 1 -- 172.21.15.1:0/4140534694 shutdown_connections 2024-03-25T18:30:58.975 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:30:58.971+0000 7f00ba758640 1 -- 172.21.15.1:0/4140534694 >> 172.21.15.1:0/4140534694 conn(0x7f00b40617a0 msgr2=0x7f00b4061bb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:30:58.975 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:30:58.971+0000 7f00ba758640 1 -- 172.21.15.1:0/4140534694 shutdown_connections 2024-03-25T18:30:58.975 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:30:58.971+0000 7f00ba758640 1 -- 172.21.15.1:0/4140534694 wait complete. 2024-03-25T18:30:58.975 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:30:58.972+0000 7f00ba758640 10 monclient: build_initial_monmap 2024-03-25T18:30:58.975 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:30:58.972+0000 7f00ba758640 10 monclient: monmap: 2024-03-25T18:30:58.976 INFO:teuthology.misc.health.smithi001.stderr:epoch 0 2024-03-25T18:30:58.976 INFO:teuthology.misc.health.smithi001.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T18:30:58.976 INFO:teuthology.misc.health.smithi001.stderr:last_changed 0.000000 2024-03-25T18:30:58.976 INFO:teuthology.misc.health.smithi001.stderr:created 0.000000 2024-03-25T18:30:58.976 INFO:teuthology.misc.health.smithi001.stderr:min_mon_release 0 (unknown) 2024-03-25T18:30:58.976 INFO:teuthology.misc.health.smithi001.stderr:election_strategy: 1 2024-03-25T18:30:58.976 INFO:teuthology.misc.health.smithi001.stderr:0: v1:172.21.15.1:6789/0 mon.noname-a 2024-03-25T18:30:58.976 INFO:teuthology.misc.health.smithi001.stderr:1: v1:172.21.15.115:6789/0 mon.noname-b 2024-03-25T18:30:58.976 INFO:teuthology.misc.health.smithi001.stderr:2: v1:172.21.15.1:6790/0 mon.noname-c 2024-03-25T18:30:58.976 INFO:teuthology.misc.health.smithi001.stderr: 2024-03-25T18:30:58.976 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:30:58.972+0000 7f00ba758640 1 librados: starting msgr at 2024-03-25T18:30:58.976 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:30:58.972+0000 7f00ba758640 1 librados: starting objecter 2024-03-25T18:30:58.977 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:30:58.972+0000 7f00ba758640 1 Processor -- start 2024-03-25T18:30:58.977 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:30:58.972+0000 7f00ba758640 1 -- start start 2024-03-25T18:30:58.977 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:30:58.972+0000 7f00ba758640 1 librados: setting wanted keys 2024-03-25T18:30:58.977 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:30:58.972+0000 7f00ba758640 1 librados: calling monclient init 2024-03-25T18:30:58.977 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:30:58.972+0000 7f00ba758640 10 monclient: init 2024-03-25T18:30:58.977 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:30:58.973+0000 7f00ba758640 10 monclient: _reopen_session rank -1 2024-03-25T18:30:58.977 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:30:58.973+0000 7f00ba758640 10 monclient: _add_conns ranks=[0,1,2] 2024-03-25T18:30:58.977 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:30:58.973+0000 7f00ba758640 10 monclient(hunting): picked mon.noname-a con 0x7f00b40b4670 addr v1:172.21.15.1:6789/0 2024-03-25T18:30:58.977 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:30:58.973+0000 7f00ba758640 10 monclient(hunting): picked mon.noname-b con 0x7f00b40b4ff0 addr v1:172.21.15.115:6789/0 2024-03-25T18:30:58.977 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:30:58.973+0000 7f00ba758640 10 monclient(hunting): picked mon.noname-c con 0x7f00b413c480 addr v1:172.21.15.1:6790/0 2024-03-25T18:30:58.977 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:30:58.973+0000 7f00ba758640 1 -- --> v1:172.21.15.1:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f00b40aedf0 con 0x7f00b40b4670 2024-03-25T18:30:58.977 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:30:58.973+0000 7f00ba758640 1 -- --> v1:172.21.15.115:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f00b40b0220 con 0x7f00b40b4ff0 2024-03-25T18:30:58.977 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:30:58.973+0000 7f00ba758640 1 -- --> v1:172.21.15.1:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f00b40b0500 con 0x7f00b413c480 2024-03-25T18:30:58.978 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:30:58.973+0000 7f00ba758640 10 monclient(hunting): _renew_subs 2024-03-25T18:30:58.978 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:30:58.973+0000 7f00ba758640 10 monclient(hunting): authenticate will time out at 7739.443338s 2024-03-25T18:30:58.978 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:30:58.975+0000 7f00b37fe640 1 --1- >> v1:172.21.15.115:6789/0 conn(0x7f00b40b4ff0 0x7f00b40b0c00 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.115:6789/0 says I am v1:172.21.15.1:43228/0 (socket says 172.21.15.1:43228) 2024-03-25T18:30:58.978 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:30:58.975+0000 7f00b37fe640 1 -- 172.21.15.1:0/2642148039 learned_addr learned my addr 172.21.15.1:0/2642148039 (peer_addr_for_me v1:172.21.15.1:0/0) 2024-03-25T18:30:58.978 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:30:58.976+0000 7f00b0ff9640 10 client.?.objecter ms_handle_connect 0x7f00b40b4ff0 2024-03-25T18:30:58.979 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:30:58.976+0000 7f00b0ff9640 10 client.?.objecter resend_mon_ops 2024-03-25T18:30:58.992 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:30:58.991+0000 7f00b0ff9640 10 client.?.objecter ms_handle_connect 0x7f00b413c480 2024-03-25T18:30:58.992 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:30:58.991+0000 7f00b0ff9640 10 client.?.objecter resend_mon_ops 2024-03-25T18:30:59.059 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:30:59.059+0000 7f00b0ff9640 10 client.?.objecter ms_handle_connect 0x7f00b40b4670 2024-03-25T18:30:59.060 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:30:59.059+0000 7f00b0ff9640 10 client.?.objecter resend_mon_ops 2024-03-25T18:30:59.072 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:30:59.071+0000 7f00b0ff9640 1 -- 172.21.15.1:0/2642148039 <== mon.1 v1:172.21.15.115:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 290080199 0 0) 0x7f00b40b0220 con 0x7f00b40b4ff0 2024-03-25T18:30:59.072 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:30:59.071+0000 7f00b0ff9640 10 monclient(hunting): _negotiate 2024-03-25T18:30:59.072 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:30:59.071+0000 7f00b0ff9640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:30:59.072 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:30:59.071+0000 7f00b0ff9640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:30:59.072 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:30:59.071+0000 7f00b0ff9640 10 monclient(hunting): my global_id is 4411 2024-03-25T18:30:59.072 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:30:59.071+0000 7f00b0ff9640 1 -- 172.21.15.1:0/2642148039 --> v1:172.21.15.115:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f00a0004ee0 con 0x7f00b40b4ff0 2024-03-25T18:30:59.074 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:30:59.074+0000 7f00b0ff9640 1 -- 172.21.15.1:0/2642148039 <== mon.2 v1:172.21.15.1:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3007901506 0 0) 0x7f00b40b0500 con 0x7f00b413c480 2024-03-25T18:30:59.075 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:30:59.074+0000 7f00b0ff9640 10 monclient(hunting): _negotiate 2024-03-25T18:30:59.075 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:30:59.074+0000 7f00b0ff9640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:30:59.075 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:30:59.074+0000 7f00b0ff9640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:30:59.075 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:30:59.074+0000 7f00b0ff9640 10 monclient(hunting): my global_id is 4415 2024-03-25T18:30:59.075 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:30:59.074+0000 7f00b0ff9640 1 -- 172.21.15.1:0/2642148039 --> v1:172.21.15.1:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f00b40b0220 con 0x7f00b413c480 2024-03-25T18:30:59.084 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:30:59.084+0000 7f00b0ff9640 1 -- 172.21.15.1:0/2642148039 <== mon.0 v1:172.21.15.1:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2408099577 0 0) 0x7f00b40aedf0 con 0x7f00b40b4670 2024-03-25T18:30:59.085 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:30:59.084+0000 7f00b0ff9640 10 monclient(hunting): _negotiate 2024-03-25T18:30:59.085 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:30:59.084+0000 7f00b0ff9640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:30:59.085 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:30:59.084+0000 7f00b0ff9640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:30:59.085 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:30:59.084+0000 7f00b0ff9640 10 monclient(hunting): my global_id is 4404 2024-03-25T18:30:59.085 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:30:59.084+0000 7f00b0ff9640 1 -- 172.21.15.1:0/2642148039 --> v1:172.21.15.1:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f00b40b0500 con 0x7f00b40b4670 2024-03-25T18:30:59.089 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:30:59.088+0000 7f00b0ff9640 1 -- 172.21.15.1:0/2642148039 <== mon.1 v1:172.21.15.115:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3732502287 0 0) 0x7f00a0004ee0 con 0x7f00b40b4ff0 2024-03-25T18:30:59.089 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:30:59.088+0000 7f00b0ff9640 1 -- 172.21.15.1:0/2642148039 --> v1:172.21.15.115:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f00b40aedf0 con 0x7f00b40b4ff0 2024-03-25T18:30:59.089 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:30:59.088+0000 7f00b0ff9640 1 -- 172.21.15.1:0/2642148039 <== mon.1 v1:172.21.15.115:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1586965025 0 0) 0x7f00a8001690 con 0x7f00b40b4ff0 2024-03-25T18:30:59.089 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:30:59.088+0000 7f00b0ff9640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:30:59.089 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:30:59.088+0000 7f00b0ff9640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-03-25T18:30:59.089 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:30:59.088+0000 7f00b0ff9640 10 monclient(hunting): dump: 2024-03-25T18:30:59.090 INFO:teuthology.misc.health.smithi001.stderr:epoch 1 2024-03-25T18:30:59.090 INFO:teuthology.misc.health.smithi001.stderr:fsid c8ea72cf-9e99-4383-9b52-6c23d365fdfc 2024-03-25T18:30:59.090 INFO:teuthology.misc.health.smithi001.stderr:last_changed 2024-03-25T18:25:50.510519+0000 2024-03-25T18:30:59.090 INFO:teuthology.misc.health.smithi001.stderr:created 2024-03-25T18:25:50.510519+0000 2024-03-25T18:30:59.090 INFO:teuthology.misc.health.smithi001.stderr:min_mon_release 19 (squid) 2024-03-25T18:30:59.090 INFO:teuthology.misc.health.smithi001.stderr:election_strategy: 3 2024-03-25T18:30:59.090 INFO:teuthology.misc.health.smithi001.stderr:0: v1:172.21.15.1:6789/0 mon.a 2024-03-25T18:30:59.090 INFO:teuthology.misc.health.smithi001.stderr:1: v1:172.21.15.115:6789/0 mon.b 2024-03-25T18:30:59.090 INFO:teuthology.misc.health.smithi001.stderr:2: v1:172.21.15.1:6790/0 mon.c 2024-03-25T18:30:59.090 INFO:teuthology.misc.health.smithi001.stderr: 2024-03-25T18:30:59.090 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:30:59.088+0000 7f00b0ff9640 10 monclient(hunting): _finish_auth 0 2024-03-25T18:30:59.103 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:30:59.103+0000 7f00b0ff9640 1 -- 172.21.15.1:0/2642148039 <== mon.0 v1:172.21.15.1:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 221292467 0 0) 0x7f00b40b0500 con 0x7f00b40b4670 2024-03-25T18:30:59.104 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:30:59.103+0000 7f00b0ff9640 1 -- 172.21.15.1:0/2642148039 --> v1:172.21.15.1:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f00a0004ee0 con 0x7f00b40b4670 2024-03-25T18:30:59.104 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:30:59.103+0000 7f00b0ff9640 1 -- 172.21.15.1:0/2642148039 <== mon.0 v1:172.21.15.1:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1586965025 0 0) 0x7f00a4001930 con 0x7f00b40b4670 2024-03-25T18:30:59.104 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:30:59.103+0000 7f00b0ff9640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:30:59.104 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:30:59.103+0000 7f00b0ff9640 10 monclient(hunting): got monmap 1 from mon.a (according to old e1) 2024-03-25T18:30:59.104 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:30:59.103+0000 7f00b0ff9640 10 monclient(hunting): dump: 2024-03-25T18:30:59.104 INFO:teuthology.misc.health.smithi001.stderr:epoch 1 2024-03-25T18:30:59.104 INFO:teuthology.misc.health.smithi001.stderr:fsid c8ea72cf-9e99-4383-9b52-6c23d365fdfc 2024-03-25T18:30:59.104 INFO:teuthology.misc.health.smithi001.stderr:last_changed 2024-03-25T18:25:50.510519+0000 2024-03-25T18:30:59.104 INFO:teuthology.misc.health.smithi001.stderr:created 2024-03-25T18:25:50.510519+0000 2024-03-25T18:30:59.104 INFO:teuthology.misc.health.smithi001.stderr:min_mon_release 19 (squid) 2024-03-25T18:30:59.104 INFO:teuthology.misc.health.smithi001.stderr:election_strategy: 3 2024-03-25T18:30:59.104 INFO:teuthology.misc.health.smithi001.stderr:0: v1:172.21.15.1:6789/0 mon.a 2024-03-25T18:30:59.105 INFO:teuthology.misc.health.smithi001.stderr:1: v1:172.21.15.115:6789/0 mon.b 2024-03-25T18:30:59.105 INFO:teuthology.misc.health.smithi001.stderr:2: v1:172.21.15.1:6790/0 mon.c 2024-03-25T18:30:59.105 INFO:teuthology.misc.health.smithi001.stderr: 2024-03-25T18:30:59.106 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:30:59.105+0000 7f00b0ff9640 1 -- 172.21.15.1:0/2642148039 <== mon.1 v1:172.21.15.115:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 1393742181 0 0) 0x7f00b40aedf0 con 0x7f00b40b4ff0 2024-03-25T18:30:59.106 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:30:59.105+0000 7f00b0ff9640 1 -- 172.21.15.1:0/2642148039 >> v1:172.21.15.1:6790/0 conn(0x7f00b413c480 legacy=0x7f00b41e2170 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:30:59.106 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:30:59.105+0000 7f00b0ff9640 1 -- 172.21.15.1:0/2642148039 >> v1:172.21.15.1:6789/0 conn(0x7f00b40b4670 legacy=0x7f00b40ae380 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:30:59.106 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:30:59.105+0000 7f00b0ff9640 10 monclient: _finish_hunting 0 2024-03-25T18:30:59.106 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:30:59.105+0000 7f00b0ff9640 1 monclient: found mon.b 2024-03-25T18:30:59.106 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:30:59.105+0000 7f00b0ff9640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:30:59.107 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:30:59.105+0000 7f00b0ff9640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.115:6789/0 2024-03-25T18:30:59.107 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:30:59.105+0000 7f00b0ff9640 1 -- 172.21.15.1:0/2642148039 --> v1:172.21.15.115:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f00b40b07e0 con 0x7f00b40b4ff0 2024-03-25T18:30:59.107 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:30:59.105+0000 7f00b0ff9640 10 monclient: _finish_auth 0 2024-03-25T18:30:59.107 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:30:59.105+0000 7f00b0ff9640 10 monclient: _check_auth_tickets 2024-03-25T18:30:59.107 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:30:59.105+0000 7f00b0ff9640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:30:59.107 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:30:59.105+0000 7f00ba758640 5 monclient: authenticate success, global_id 4411 2024-03-25T18:30:59.107 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:30:59.105+0000 7f00ba758640 10 monclient: _renew_subs 2024-03-25T18:30:59.107 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:30:59.105+0000 7f00ba758640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.115:6789/0 2024-03-25T18:30:59.107 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:30:59.105+0000 7f00ba758640 1 -- 172.21.15.1:0/2642148039 --> v1:172.21.15.115:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f00b40b1310 con 0x7f00b40b4ff0 2024-03-25T18:30:59.107 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:30:59.105+0000 7f00ba758640 10 client.4411.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T18:30:59.107 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:30:59.105+0000 7f00ba758640 10 monclient: _renew_subs 2024-03-25T18:30:59.107 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:30:59.105+0000 7f00ba758640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.115:6789/0 2024-03-25T18:30:59.108 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:30:59.105+0000 7f00ba758640 1 -- 172.21.15.1:0/2642148039 --> v1:172.21.15.115:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f00b40af460 con 0x7f00b40b4ff0 2024-03-25T18:30:59.108 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:30:59.105+0000 7f00ba758640 1 librados: init done 2024-03-25T18:30:59.108 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:30:59.106+0000 7f00ba758640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T18:30:59.108 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:30:59.106+0000 7f00ba758640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:30:59.108 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:30:59.106+0000 7f00ba758640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.115:6789/0 2024-03-25T18:30:59.108 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:30:59.106+0000 7f00ba758640 1 -- 172.21.15.1:0/2642148039 --> v1:172.21.15.115:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f00b406dbd0 con 0x7f00b40b4ff0 2024-03-25T18:30:59.385 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:30:59.384+0000 7f00b0ff9640 1 -- 172.21.15.1:0/2642148039 <== mon.1 v1:172.21.15.115:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f00a80038a0 con 0x7f00b40b4ff0 2024-03-25T18:30:59.385 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:30:59.384+0000 7f00b0ff9640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:30:59.385 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:30:59.384+0000 7f00b0ff9640 1 -- 172.21.15.1:0/2642148039 <== mon.1 v1:172.21.15.115:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1586965025 0 0) 0x7f00a8003c70 con 0x7f00b40b4ff0 2024-03-25T18:30:59.385 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:30:59.384+0000 7f00b0ff9640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:30:59.385 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:30:59.384+0000 7f00b0ff9640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-25T18:30:59.385 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:30:59.384+0000 7f00b0ff9640 10 monclient: dump: 2024-03-25T18:30:59.385 INFO:teuthology.misc.health.smithi001.stderr:epoch 1 2024-03-25T18:30:59.386 INFO:teuthology.misc.health.smithi001.stderr:fsid c8ea72cf-9e99-4383-9b52-6c23d365fdfc 2024-03-25T18:30:59.386 INFO:teuthology.misc.health.smithi001.stderr:last_changed 2024-03-25T18:25:50.510519+0000 2024-03-25T18:30:59.386 INFO:teuthology.misc.health.smithi001.stderr:created 2024-03-25T18:25:50.510519+0000 2024-03-25T18:30:59.386 INFO:teuthology.misc.health.smithi001.stderr:min_mon_release 19 (squid) 2024-03-25T18:30:59.386 INFO:teuthology.misc.health.smithi001.stderr:election_strategy: 3 2024-03-25T18:30:59.386 INFO:teuthology.misc.health.smithi001.stderr:0: v1:172.21.15.1:6789/0 mon.a 2024-03-25T18:30:59.386 INFO:teuthology.misc.health.smithi001.stderr:1: v1:172.21.15.115:6789/0 mon.b 2024-03-25T18:30:59.386 INFO:teuthology.misc.health.smithi001.stderr:2: v1:172.21.15.1:6790/0 mon.c 2024-03-25T18:30:59.386 INFO:teuthology.misc.health.smithi001.stderr: 2024-03-25T18:30:59.386 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:30:59.384+0000 7f00b27fc640 4 set_mon_vals no callback set 2024-03-25T18:30:59.387 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:30:59.385+0000 7f00b0ff9640 1 -- 172.21.15.1:0/2642148039 <== mon.1 v1:172.21.15.115:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 21787975 0 0) 0x7f00a8003e90 con 0x7f00b40b4ff0 2024-03-25T18:30:59.387 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:30:59.385+0000 7f00b3fff640 10 monclient: get_auth_request con 0x7f00a006dfa0 auth_method 0 2024-03-25T18:30:59.387 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:30:59.386+0000 7f00b0ff9640 10 client.4411.objecter ms_handle_connect 0x7f00a006dfa0 2024-03-25T18:30:59.392 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:30:59.391+0000 7f00b0ff9640 1 -- 172.21.15.1:0/2642148039 <== mon.1 v1:172.21.15.115:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 996254230 0 0) 0x7f00a8054920 con 0x7f00b40b4ff0 2024-03-25T18:30:59.392 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:30:59.391+0000 7f00b0ff9640 10 client.4411.objecter ms_dispatch 0x7f00b4000b90 osd_map(10..10 src has 1..10) v4 2024-03-25T18:30:59.392 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:30:59.391+0000 7f00b0ff9640 3 client.4411.objecter handle_osd_map got epochs [10,10] > 0 2024-03-25T18:30:59.392 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:30:59.391+0000 7f00b0ff9640 3 client.4411.objecter handle_osd_map decoding full epoch 10 2024-03-25T18:30:59.392 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:30:59.391+0000 7f00b0ff9640 20 client.4411.objecter dump_active .. 0 homeless 2024-03-25T18:30:59.393 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:30:59.392+0000 7f00b0ff9640 1 -- 172.21.15.1:0/2642148039 <== mon.1 v1:172.21.15.115:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f00a8050430 con 0x7f00b40b4ff0 2024-03-25T18:30:59.393 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:30:59.392+0000 7f00b0ff9640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:30:59.393 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:30:59.392+0000 7f00b0ff9640 10 monclient: _finish_command 1 = system:0 2024-03-25T18:30:59.651 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:30:59.651+0000 7f00ba758640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:30:59.652 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:30:59.651+0000 7f00ba758640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:30:59.652 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:30:59.651+0000 7f00ba758640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.115:6789/0 2024-03-25T18:30:59.652 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:30:59.651+0000 7f00ba758640 1 -- 172.21.15.1:0/2642148039 --> v1:172.21.15.115:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f00b40688c0 con 0x7f00b40b4ff0 2024-03-25T18:30:59.681 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:30:59.680+0000 7f00b0ff9640 1 -- 172.21.15.1:0/2642148039 <== mon.1 v1:172.21.15.115:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 4092213348) 0x7f00a8052360 con 0x7f00b40b4ff0 2024-03-25T18:30:59.681 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:30:59.680+0000 7f00b0ff9640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:30:59.682 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:30:59.680+0000 7f00b0ff9640 10 monclient: _finish_command 2 = system:0 2024-03-25T18:30:59.682 INFO:teuthology.misc.health.smithi001.stdout: 2024-03-25T18:30:59.682 INFO:teuthology.misc.health.smithi001.stdout:{"epoch":10,"fsid":"c8ea72cf-9e99-4383-9b52-6c23d365fdfc","created":"2024-03-25T18:26:38.856638+0000","modified":"2024-03-25T18:26:44.224208+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T18:26:44.224208+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"b9f34922-3a84-49be-8426-807c1eceaefc","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"42bf306f-55e3-4634-9874-8b4b73ac6644","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"292cb23a-18a9-4207-a7cc-4d5727feaec0","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"45e0e9ab-898d-4673-9590-b4117bc5e84a","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"831a5a09-73ab-4a61-9872-088e76bdbc66","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"88cd903d-3b55-4840-9396-6e3cfe005282","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"89db7355-bc9d-46f4-8ab0-e8d9e744e026","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"cac47bee-0914-4f08-896f-348838882866","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T18:30:59.684 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:30:59.683+0000 7f00ba758640 10 librados: watch_flush enter 2024-03-25T18:30:59.685 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:30:59.683+0000 7f00ba758640 10 librados: watch_flush exit 2024-03-25T18:30:59.685 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:30:59.683+0000 7f00ba758640 20 client.4411.objecter shutdown clearing up homeless session... 2024-03-25T18:30:59.685 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:30:59.683+0000 7f00ba758640 10 client.4411.objecter successfully canceled tick 2024-03-25T18:30:59.685 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:30:59.683+0000 7f00ba758640 1 -- 172.21.15.1:0/2642148039 >> v1:172.21.15.115:6800/4237379176 conn(0x7f00a006dfa0 legacy=0x7f00a0070460 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:30:59.685 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:30:59.683+0000 7f00ba758640 10 monclient: shutdown 2024-03-25T18:30:59.685 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:30:59.683+0000 7f00ba758640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:30:59.685 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:30:59.683+0000 7f00ba758640 1 -- 172.21.15.1:0/2642148039 >> v1:172.21.15.115:6789/0 conn(0x7f00b40b4ff0 legacy=0x7f00b40b0c00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:30:59.685 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:30:59.683+0000 7f00ba758640 1 -- 172.21.15.1:0/2642148039 shutdown_connections 2024-03-25T18:30:59.685 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:30:59.683+0000 7f00ba758640 1 -- 172.21.15.1:0/2642148039 >> 172.21.15.1:0/2642148039 conn(0x7f00b40617a0 msgr2=0x7f00b40b2a00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:30:59.685 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:30:59.683+0000 7f00ba758640 1 -- 172.21.15.1:0/2642148039 shutdown_connections 2024-03-25T18:30:59.685 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:30:59.683+0000 7f00ba758640 1 -- 172.21.15.1:0/2642148039 wait complete. 2024-03-25T18:30:59.686 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:30:59.683+0000 7f00ba758640 1 librados: shutdown 2024-03-25T18:30:59.696 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T18:31:06.502 DEBUG:teuthology.orchestra.run.smithi001:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T18:31:06.616 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:31:06.615+0000 7f755184a640 10 monclient: get_monmap_and_config 2024-03-25T18:31:06.616 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:31:06.615+0000 7f755184a640 10 monclient: build_initial_monmap 2024-03-25T18:31:06.616 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:31:06.615+0000 7f755184a640 10 monclient: monmap: 2024-03-25T18:31:06.616 INFO:teuthology.misc.health.smithi001.stderr:epoch 0 2024-03-25T18:31:06.616 INFO:teuthology.misc.health.smithi001.stderr:fsid c8ea72cf-9e99-4383-9b52-6c23d365fdfc 2024-03-25T18:31:06.616 INFO:teuthology.misc.health.smithi001.stderr:last_changed 2024-03-25T18:31:06.616247+0000 2024-03-25T18:31:06.616 INFO:teuthology.misc.health.smithi001.stderr:created 2024-03-25T18:31:06.616247+0000 2024-03-25T18:31:06.616 INFO:teuthology.misc.health.smithi001.stderr:min_mon_release 0 (unknown) 2024-03-25T18:31:06.616 INFO:teuthology.misc.health.smithi001.stderr:election_strategy: 3 2024-03-25T18:31:06.617 INFO:teuthology.misc.health.smithi001.stderr:0: v1:172.21.15.1:6789/0 mon.noname-a 2024-03-25T18:31:06.617 INFO:teuthology.misc.health.smithi001.stderr:1: v1:172.21.15.115:6789/0 mon.noname-c 2024-03-25T18:31:06.617 INFO:teuthology.misc.health.smithi001.stderr:2: v1:172.21.15.1:6790/0 mon.noname-b 2024-03-25T18:31:06.617 INFO:teuthology.misc.health.smithi001.stderr: 2024-03-25T18:31:06.618 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:31:06.616+0000 7f755184a640 1 Processor -- start 2024-03-25T18:31:06.618 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:31:06.616+0000 7f755184a640 1 -- start start 2024-03-25T18:31:06.618 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:31:06.616+0000 7f755184a640 10 monclient: init 2024-03-25T18:31:06.618 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:31:06.616+0000 7f755184a640 10 monclient: _reopen_session rank -1 2024-03-25T18:31:06.618 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:31:06.616+0000 7f755184a640 10 monclient: _add_conns ranks=[1,0,2] 2024-03-25T18:31:06.618 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:31:06.616+0000 7f755184a640 10 monclient(hunting): picked mon.noname-c con 0x7f754c0b4ff0 addr v1:172.21.15.115:6789/0 2024-03-25T18:31:06.619 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:31:06.616+0000 7f755184a640 10 monclient(hunting): picked mon.noname-a con 0x7f754c0b4670 addr v1:172.21.15.1:6789/0 2024-03-25T18:31:06.619 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:31:06.616+0000 7f755184a640 10 monclient(hunting): picked mon.noname-b con 0x7f754c13c480 addr v1:172.21.15.1:6790/0 2024-03-25T18:31:06.619 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:31:06.616+0000 7f755184a640 1 -- --> v1:172.21.15.1:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f754c0ab0e0 con 0x7f754c0b4670 2024-03-25T18:31:06.619 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:31:06.617+0000 7f755184a640 1 -- --> v1:172.21.15.115:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f754c0ab3c0 con 0x7f754c0b4ff0 2024-03-25T18:31:06.619 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:31:06.617+0000 7f755184a640 1 -- --> v1:172.21.15.1:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f754c0a9f30 con 0x7f754c13c480 2024-03-25T18:31:06.619 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:31:06.617+0000 7f755184a640 10 monclient(hunting): _renew_subs 2024-03-25T18:31:06.619 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:31:06.617+0000 7f755184a640 10 monclient(hunting): authenticate will time out at 7747.086857s 2024-03-25T18:31:06.632 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:31:06.631+0000 7f754affd640 1 --1- >> v1:172.21.15.115:6789/0 conn(0x7f754c0b4ff0 0x7f754c13bd70 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.115:6789/0 says I am v1:172.21.15.1:42898/0 (socket says 172.21.15.1:42898) 2024-03-25T18:31:06.632 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:31:06.631+0000 7f754affd640 1 -- 172.21.15.1:0/3763415235 learned_addr learned my addr 172.21.15.1:0/3763415235 (peer_addr_for_me v1:172.21.15.1:0/0) 2024-03-25T18:31:06.638 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:31:06.637+0000 7f7549ffb640 1 -- 172.21.15.1:0/3763415235 <== mon.1 v1:172.21.15.115:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 570742670 0 0) 0x7f754c0ab3c0 con 0x7f754c0b4ff0 2024-03-25T18:31:06.638 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:31:06.637+0000 7f7549ffb640 10 monclient(hunting): _negotiate 2024-03-25T18:31:06.638 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:31:06.637+0000 7f7549ffb640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:31:06.638 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:31:06.637+0000 7f7549ffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:31:06.638 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:31:06.637+0000 7f7549ffb640 10 monclient(hunting): my global_id is 4414 2024-03-25T18:31:06.638 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:31:06.637+0000 7f7549ffb640 1 -- 172.21.15.1:0/3763415235 --> v1:172.21.15.115:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f7534003520 con 0x7f754c0b4ff0 2024-03-25T18:31:06.648 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:31:06.647+0000 7f7549ffb640 1 -- 172.21.15.1:0/3763415235 <== mon.0 v1:172.21.15.1:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3032542994 0 0) 0x7f754c0ab0e0 con 0x7f754c0b4670 2024-03-25T18:31:06.648 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:31:06.647+0000 7f7549ffb640 10 monclient(hunting): _negotiate 2024-03-25T18:31:06.648 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:31:06.647+0000 7f7549ffb640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:31:06.648 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:31:06.647+0000 7f7549ffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:31:06.648 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:31:06.647+0000 7f7549ffb640 10 monclient(hunting): my global_id is 4407 2024-03-25T18:31:06.649 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:31:06.647+0000 7f7549ffb640 1 -- 172.21.15.1:0/3763415235 --> v1:172.21.15.1:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f754c0ab3c0 con 0x7f754c0b4670 2024-03-25T18:31:06.660 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:31:06.658+0000 7f7549ffb640 1 -- 172.21.15.1:0/3763415235 <== mon.1 v1:172.21.15.115:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1395196668 0 0) 0x7f7534003520 con 0x7f754c0b4ff0 2024-03-25T18:31:06.660 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:31:06.659+0000 7f7549ffb640 1 -- 172.21.15.1:0/3763415235 >> v1:172.21.15.1:6790/0 conn(0x7f754c13c480 legacy=0x7f754c14dc00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:31:06.660 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:31:06.659+0000 7f7549ffb640 1 -- 172.21.15.1:0/3763415235 >> v1:172.21.15.1:6789/0 conn(0x7f754c0b4670 legacy=0x7f754c1425c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:31:06.660 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:31:06.659+0000 7f7549ffb640 10 monclient: _finish_hunting 0 2024-03-25T18:31:06.660 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:31:06.659+0000 7f7549ffb640 1 monclient: found mon.noname-c 2024-03-25T18:31:06.660 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:31:06.659+0000 7f7549ffb640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:31:06.660 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:31:06.659+0000 7f7549ffb640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.115:6789/0 2024-03-25T18:31:06.660 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:31:06.659+0000 7f7549ffb640 1 -- 172.21.15.1:0/3763415235 --> v1:172.21.15.115:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f754c0a9b70 con 0x7f754c0b4ff0 2024-03-25T18:31:06.660 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:31:06.659+0000 7f7549ffb640 10 monclient: _finish_auth 0 2024-03-25T18:31:06.660 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:31:06.659+0000 7f7549ffb640 10 monclient: _check_auth_tickets 2024-03-25T18:31:06.660 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:31:06.659+0000 7f7549ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:31:06.660 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:31:06.659+0000 7f7549ffb640 1 -- 172.21.15.1:0/3763415235 <== mon.1 v1:172.21.15.115:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1586965025 0 0) 0x7f753c001690 con 0x7f754c0b4ff0 2024-03-25T18:31:06.660 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:31:06.659+0000 7f7549ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:31:06.661 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:31:06.659+0000 7f7549ffb640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-03-25T18:31:06.661 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:31:06.659+0000 7f7549ffb640 10 monclient: dump: 2024-03-25T18:31:06.661 INFO:teuthology.misc.health.smithi001.stderr:epoch 1 2024-03-25T18:31:06.661 INFO:teuthology.misc.health.smithi001.stderr:fsid c8ea72cf-9e99-4383-9b52-6c23d365fdfc 2024-03-25T18:31:06.661 INFO:teuthology.misc.health.smithi001.stderr:last_changed 2024-03-25T18:25:50.510519+0000 2024-03-25T18:31:06.661 INFO:teuthology.misc.health.smithi001.stderr:created 2024-03-25T18:25:50.510519+0000 2024-03-25T18:31:06.661 INFO:teuthology.misc.health.smithi001.stderr:min_mon_release 19 (squid) 2024-03-25T18:31:06.661 INFO:teuthology.misc.health.smithi001.stderr:election_strategy: 3 2024-03-25T18:31:06.661 INFO:teuthology.misc.health.smithi001.stderr:0: v1:172.21.15.1:6789/0 mon.a 2024-03-25T18:31:06.661 INFO:teuthology.misc.health.smithi001.stderr:1: v1:172.21.15.115:6789/0 mon.b 2024-03-25T18:31:06.661 INFO:teuthology.misc.health.smithi001.stderr:2: v1:172.21.15.1:6790/0 mon.c 2024-03-25T18:31:06.661 INFO:teuthology.misc.health.smithi001.stderr: 2024-03-25T18:31:06.661 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:31:06.659+0000 7f755184a640 5 monclient: authenticate success, global_id 4414 2024-03-25T18:31:06.661 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:31:06.659+0000 7f755184a640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T18:31:06.686 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:31:06.685+0000 7f7549ffb640 1 -- 172.21.15.1:0/3763415235 <== mon.1 v1:172.21.15.115:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f753c0017f0 con 0x7f754c0b4ff0 2024-03-25T18:31:06.686 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:31:06.685+0000 7f7549ffb640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:31:06.686 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:31:06.685+0000 7f7549ffb640 1 -- 172.21.15.1:0/3763415235 <== mon.1 v1:172.21.15.115:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1586965025 0 0) 0x7f753c001c00 con 0x7f754c0b4ff0 2024-03-25T18:31:06.686 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:31:06.685+0000 7f7549ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:31:06.686 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:31:06.685+0000 7f7549ffb640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-25T18:31:06.686 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:31:06.685+0000 7f7549ffb640 10 monclient: dump: 2024-03-25T18:31:06.686 INFO:teuthology.misc.health.smithi001.stderr:epoch 1 2024-03-25T18:31:06.686 INFO:teuthology.misc.health.smithi001.stderr:fsid c8ea72cf-9e99-4383-9b52-6c23d365fdfc 2024-03-25T18:31:06.686 INFO:teuthology.misc.health.smithi001.stderr:last_changed 2024-03-25T18:25:50.510519+0000 2024-03-25T18:31:06.686 INFO:teuthology.misc.health.smithi001.stderr:created 2024-03-25T18:25:50.510519+0000 2024-03-25T18:31:06.687 INFO:teuthology.misc.health.smithi001.stderr:min_mon_release 19 (squid) 2024-03-25T18:31:06.687 INFO:teuthology.misc.health.smithi001.stderr:election_strategy: 3 2024-03-25T18:31:06.687 INFO:teuthology.misc.health.smithi001.stderr:0: v1:172.21.15.1:6789/0 mon.a 2024-03-25T18:31:06.687 INFO:teuthology.misc.health.smithi001.stderr:1: v1:172.21.15.115:6789/0 mon.b 2024-03-25T18:31:06.687 INFO:teuthology.misc.health.smithi001.stderr:2: v1:172.21.15.1:6790/0 mon.c 2024-03-25T18:31:06.687 INFO:teuthology.misc.health.smithi001.stderr: 2024-03-25T18:31:06.687 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:31:06.685+0000 7f755184a640 10 monclient: get_monmap_and_config success 2024-03-25T18:31:06.687 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:31:06.685+0000 7f755184a640 4 set_mon_vals no callback set 2024-03-25T18:31:06.689 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:31:06.686+0000 7f755184a640 10 monclient: shutdown 2024-03-25T18:31:06.689 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:31:06.686+0000 7f755184a640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:31:06.690 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:31:06.686+0000 7f755184a640 1 -- 172.21.15.1:0/3763415235 >> v1:172.21.15.115:6789/0 conn(0x7f754c0b4ff0 legacy=0x7f754c13bd70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:31:06.690 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:31:06.686+0000 7f755184a640 1 -- 172.21.15.1:0/3763415235 shutdown_connections 2024-03-25T18:31:06.690 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:31:06.686+0000 7f755184a640 1 -- 172.21.15.1:0/3763415235 >> 172.21.15.1:0/3763415235 conn(0x7f754c0617a0 msgr2=0x7f754c061bb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:31:06.690 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:31:06.686+0000 7f755184a640 1 -- 172.21.15.1:0/3763415235 shutdown_connections 2024-03-25T18:31:06.690 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:31:06.686+0000 7f755184a640 1 -- 172.21.15.1:0/3763415235 wait complete. 2024-03-25T18:31:06.690 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:31:06.687+0000 7f755184a640 10 monclient: build_initial_monmap 2024-03-25T18:31:06.690 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:31:06.687+0000 7f755184a640 10 monclient: monmap: 2024-03-25T18:31:06.690 INFO:teuthology.misc.health.smithi001.stderr:epoch 0 2024-03-25T18:31:06.690 INFO:teuthology.misc.health.smithi001.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T18:31:06.690 INFO:teuthology.misc.health.smithi001.stderr:last_changed 0.000000 2024-03-25T18:31:06.690 INFO:teuthology.misc.health.smithi001.stderr:created 0.000000 2024-03-25T18:31:06.690 INFO:teuthology.misc.health.smithi001.stderr:min_mon_release 0 (unknown) 2024-03-25T18:31:06.691 INFO:teuthology.misc.health.smithi001.stderr:election_strategy: 1 2024-03-25T18:31:06.691 INFO:teuthology.misc.health.smithi001.stderr:0: v1:172.21.15.1:6789/0 mon.noname-a 2024-03-25T18:31:06.691 INFO:teuthology.misc.health.smithi001.stderr:1: v1:172.21.15.115:6789/0 mon.noname-b 2024-03-25T18:31:06.691 INFO:teuthology.misc.health.smithi001.stderr:2: v1:172.21.15.1:6790/0 mon.noname-c 2024-03-25T18:31:06.691 INFO:teuthology.misc.health.smithi001.stderr: 2024-03-25T18:31:06.691 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:31:06.687+0000 7f755184a640 1 librados: starting msgr at 2024-03-25T18:31:06.691 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:31:06.687+0000 7f755184a640 1 librados: starting objecter 2024-03-25T18:31:06.691 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:31:06.687+0000 7f755184a640 1 Processor -- start 2024-03-25T18:31:06.691 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:31:06.687+0000 7f755184a640 1 -- start start 2024-03-25T18:31:06.691 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:31:06.687+0000 7f755184a640 1 librados: setting wanted keys 2024-03-25T18:31:06.691 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:31:06.687+0000 7f755184a640 1 librados: calling monclient init 2024-03-25T18:31:06.691 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:31:06.687+0000 7f755184a640 10 monclient: init 2024-03-25T18:31:06.691 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:31:06.688+0000 7f755184a640 10 monclient: _reopen_session rank -1 2024-03-25T18:31:06.691 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:31:06.688+0000 7f755184a640 10 monclient: _add_conns ranks=[1,2,0] 2024-03-25T18:31:06.691 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:31:06.688+0000 7f755184a640 10 monclient(hunting): picked mon.noname-b con 0x7f754c0b4670 addr v1:172.21.15.115:6789/0 2024-03-25T18:31:06.692 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:31:06.688+0000 7f755184a640 10 monclient(hunting): picked mon.noname-c con 0x7f754c0b4ff0 addr v1:172.21.15.1:6790/0 2024-03-25T18:31:06.692 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:31:06.688+0000 7f755184a640 10 monclient(hunting): picked mon.noname-a con 0x7f754c13c480 addr v1:172.21.15.1:6789/0 2024-03-25T18:31:06.692 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:31:06.688+0000 7f755184a640 1 -- --> v1:172.21.15.1:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f754c13c990 con 0x7f754c13c480 2024-03-25T18:31:06.692 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:31:06.688+0000 7f755184a640 1 -- --> v1:172.21.15.115:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f754c143cc0 con 0x7f754c0b4670 2024-03-25T18:31:06.692 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:31:06.688+0000 7f755184a640 1 -- --> v1:172.21.15.1:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f754c13f800 con 0x7f754c0b4ff0 2024-03-25T18:31:06.692 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:31:06.688+0000 7f755184a640 10 monclient(hunting): _renew_subs 2024-03-25T18:31:06.692 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:31:06.688+0000 7f755184a640 10 monclient(hunting): authenticate will time out at 7747.158273s 2024-03-25T18:31:06.704 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:31:06.703+0000 7f754affd640 1 --1- >> v1:172.21.15.115:6789/0 conn(0x7f754c0b4670 0x7f754c13e320 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.115:6789/0 says I am v1:172.21.15.1:42968/0 (socket says 172.21.15.1:42968) 2024-03-25T18:31:06.704 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:31:06.703+0000 7f754affd640 1 -- 172.21.15.1:0/1946982626 learned_addr learned my addr 172.21.15.1:0/1946982626 (peer_addr_for_me v1:172.21.15.1:0/0) 2024-03-25T18:31:06.707 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:31:06.707+0000 7f7533fff640 10 client.?.objecter ms_handle_connect 0x7f754c0b4670 2024-03-25T18:31:06.707 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:31:06.707+0000 7f7533fff640 10 client.?.objecter resend_mon_ops 2024-03-25T18:31:06.720 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:31:06.719+0000 7f7533fff640 1 -- 172.21.15.1:0/1946982626 <== mon.1 v1:172.21.15.115:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1763867579 0 0) 0x7f754c143cc0 con 0x7f754c0b4670 2024-03-25T18:31:06.720 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:31:06.720+0000 7f7533fff640 10 monclient(hunting): _negotiate 2024-03-25T18:31:06.720 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:31:06.720+0000 7f7533fff640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:31:06.721 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:31:06.720+0000 7f7533fff640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:31:06.721 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:31:06.720+0000 7f7533fff640 10 monclient(hunting): my global_id is 4417 2024-03-25T18:31:06.721 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:31:06.720+0000 7f7533fff640 1 -- 172.21.15.1:0/1946982626 --> v1:172.21.15.115:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f7538004eb0 con 0x7f754c0b4670 2024-03-25T18:31:06.723 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:31:06.722+0000 7f7533fff640 10 client.?.objecter ms_handle_connect 0x7f754c0b4ff0 2024-03-25T18:31:06.723 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:31:06.722+0000 7f7533fff640 10 client.?.objecter resend_mon_ops 2024-03-25T18:31:06.727 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:31:06.727+0000 7f7533fff640 10 client.?.objecter ms_handle_connect 0x7f754c13c480 2024-03-25T18:31:06.728 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:31:06.727+0000 7f7533fff640 10 client.?.objecter resend_mon_ops 2024-03-25T18:31:06.738 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:31:06.737+0000 7f7533fff640 1 -- 172.21.15.1:0/1946982626 <== mon.2 v1:172.21.15.1:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2457901278 0 0) 0x7f754c13f800 con 0x7f754c0b4ff0 2024-03-25T18:31:06.738 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:31:06.737+0000 7f7533fff640 10 monclient(hunting): _negotiate 2024-03-25T18:31:06.738 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:31:06.737+0000 7f7533fff640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:31:06.738 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:31:06.737+0000 7f7533fff640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:31:06.738 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:31:06.737+0000 7f7533fff640 10 monclient(hunting): my global_id is 4421 2024-03-25T18:31:06.738 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:31:06.737+0000 7f7533fff640 1 -- 172.21.15.1:0/1946982626 --> v1:172.21.15.1:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f754c143cc0 con 0x7f754c0b4ff0 2024-03-25T18:31:06.799 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:31:06.798+0000 7f7533fff640 1 -- 172.21.15.1:0/1946982626 <== mon.0 v1:172.21.15.1:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1359654423 0 0) 0x7f754c13c990 con 0x7f754c13c480 2024-03-25T18:31:06.799 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:31:06.798+0000 7f7533fff640 10 monclient(hunting): _negotiate 2024-03-25T18:31:06.799 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:31:06.798+0000 7f7533fff640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:31:06.799 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:31:06.798+0000 7f7533fff640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:31:06.799 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:31:06.798+0000 7f7533fff640 10 monclient(hunting): my global_id is 4410 2024-03-25T18:31:06.799 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:31:06.798+0000 7f7533fff640 1 -- 172.21.15.1:0/1946982626 --> v1:172.21.15.1:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f754c13f800 con 0x7f754c13c480 2024-03-25T18:31:06.838 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:31:06.837+0000 7f7533fff640 1 -- 172.21.15.1:0/1946982626 <== mon.0 v1:172.21.15.1:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 234297678 0 0) 0x7f754c13f800 con 0x7f754c13c480 2024-03-25T18:31:06.838 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:31:06.837+0000 7f7533fff640 1 -- 172.21.15.1:0/1946982626 --> v1:172.21.15.1:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f754c13c990 con 0x7f754c13c480 2024-03-25T18:31:06.838 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:31:06.837+0000 7f7533fff640 1 -- 172.21.15.1:0/1946982626 <== mon.0 v1:172.21.15.1:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1586965025 0 0) 0x7f7544001a60 con 0x7f754c13c480 2024-03-25T18:31:06.838 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:31:06.837+0000 7f7533fff640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:31:06.838 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:31:06.837+0000 7f7533fff640 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2024-03-25T18:31:06.838 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:31:06.837+0000 7f7533fff640 10 monclient(hunting): dump: 2024-03-25T18:31:06.839 INFO:teuthology.misc.health.smithi001.stderr:epoch 1 2024-03-25T18:31:06.839 INFO:teuthology.misc.health.smithi001.stderr:fsid c8ea72cf-9e99-4383-9b52-6c23d365fdfc 2024-03-25T18:31:06.839 INFO:teuthology.misc.health.smithi001.stderr:last_changed 2024-03-25T18:25:50.510519+0000 2024-03-25T18:31:06.839 INFO:teuthology.misc.health.smithi001.stderr:created 2024-03-25T18:25:50.510519+0000 2024-03-25T18:31:06.839 INFO:teuthology.misc.health.smithi001.stderr:min_mon_release 19 (squid) 2024-03-25T18:31:06.839 INFO:teuthology.misc.health.smithi001.stderr:election_strategy: 3 2024-03-25T18:31:06.839 INFO:teuthology.misc.health.smithi001.stderr:0: v1:172.21.15.1:6789/0 mon.a 2024-03-25T18:31:06.839 INFO:teuthology.misc.health.smithi001.stderr:1: v1:172.21.15.115:6789/0 mon.b 2024-03-25T18:31:06.839 INFO:teuthology.misc.health.smithi001.stderr:2: v1:172.21.15.1:6790/0 mon.c 2024-03-25T18:31:06.839 INFO:teuthology.misc.health.smithi001.stderr: 2024-03-25T18:31:06.839 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:31:06.837+0000 7f7533fff640 10 monclient(hunting): _finish_auth 0 2024-03-25T18:31:06.863 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:31:06.862+0000 7f7533fff640 1 -- 172.21.15.1:0/1946982626 <== mon.0 v1:172.21.15.1:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 3465021789 0 0) 0x7f754c13c990 con 0x7f754c13c480 2024-03-25T18:31:06.864 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:31:06.862+0000 7f7533fff640 1 -- 172.21.15.1:0/1946982626 >> v1:172.21.15.1:6790/0 conn(0x7f754c0b4ff0 legacy=0x7f754c13ce70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:31:06.864 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:31:06.862+0000 7f7533fff640 1 -- 172.21.15.1:0/1946982626 >> v1:172.21.15.115:6789/0 conn(0x7f754c0b4670 legacy=0x7f754c13e320 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:31:06.864 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:31:06.862+0000 7f7533fff640 10 monclient: _finish_hunting 0 2024-03-25T18:31:06.864 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:31:06.862+0000 7f7533fff640 1 monclient: found mon.a 2024-03-25T18:31:06.864 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:31:06.862+0000 7f7533fff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:31:06.864 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:31:06.862+0000 7f7533fff640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.1:6789/0 2024-03-25T18:31:06.864 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:31:06.862+0000 7f7533fff640 1 -- 172.21.15.1:0/1946982626 --> v1:172.21.15.1:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f754c13f990 con 0x7f754c13c480 2024-03-25T18:31:06.864 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:31:06.862+0000 7f7533fff640 10 monclient: _finish_auth 0 2024-03-25T18:31:06.864 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:31:06.862+0000 7f7533fff640 10 monclient: _check_auth_tickets 2024-03-25T18:31:06.864 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:31:06.862+0000 7f7533fff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:31:06.864 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:31:06.862+0000 7f755184a640 5 monclient: authenticate success, global_id 4410 2024-03-25T18:31:06.865 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:31:06.862+0000 7f755184a640 10 monclient: _renew_subs 2024-03-25T18:31:06.865 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:31:06.862+0000 7f755184a640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.1:6789/0 2024-03-25T18:31:06.865 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:31:06.862+0000 7f755184a640 1 -- 172.21.15.1:0/1946982626 --> v1:172.21.15.1:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f754c1436e0 con 0x7f754c13c480 2024-03-25T18:31:06.865 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:31:06.862+0000 7f755184a640 10 client.4410.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T18:31:06.865 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:31:06.862+0000 7f755184a640 10 monclient: _renew_subs 2024-03-25T18:31:06.865 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:31:06.862+0000 7f755184a640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.1:6789/0 2024-03-25T18:31:06.865 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:31:06.862+0000 7f755184a640 1 -- 172.21.15.1:0/1946982626 --> v1:172.21.15.1:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f754c1e3570 con 0x7f754c13c480 2024-03-25T18:31:06.865 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:31:06.862+0000 7f755184a640 1 librados: init done 2024-03-25T18:31:06.866 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:31:06.863+0000 7f755184a640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T18:31:06.866 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:31:06.863+0000 7f755184a640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:31:06.866 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:31:06.863+0000 7f755184a640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.1:6789/0 2024-03-25T18:31:06.866 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:31:06.863+0000 7f755184a640 1 -- 172.21.15.1:0/1946982626 --> v1:172.21.15.1:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f754c13cb90 con 0x7f754c13c480 2024-03-25T18:31:07.201 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:31:07.200+0000 7f7533fff640 1 -- 172.21.15.1:0/1946982626 <== mon.0 v1:172.21.15.1:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f7544001f90 con 0x7f754c13c480 2024-03-25T18:31:07.201 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:31:07.200+0000 7f7533fff640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:31:07.201 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:31:07.200+0000 7f7533fff640 1 -- 172.21.15.1:0/1946982626 <== mon.0 v1:172.21.15.1:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1586965025 0 0) 0x7f7544003970 con 0x7f754c13c480 2024-03-25T18:31:07.201 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:31:07.200+0000 7f7533fff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:31:07.201 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:31:07.200+0000 7f7533fff640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-25T18:31:07.202 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:31:07.200+0000 7f7533fff640 10 monclient: dump: 2024-03-25T18:31:07.202 INFO:teuthology.misc.health.smithi001.stderr:epoch 1 2024-03-25T18:31:07.202 INFO:teuthology.misc.health.smithi001.stderr:fsid c8ea72cf-9e99-4383-9b52-6c23d365fdfc 2024-03-25T18:31:07.202 INFO:teuthology.misc.health.smithi001.stderr:last_changed 2024-03-25T18:25:50.510519+0000 2024-03-25T18:31:07.202 INFO:teuthology.misc.health.smithi001.stderr:created 2024-03-25T18:25:50.510519+0000 2024-03-25T18:31:07.202 INFO:teuthology.misc.health.smithi001.stderr:min_mon_release 19 (squid) 2024-03-25T18:31:07.202 INFO:teuthology.misc.health.smithi001.stderr:election_strategy: 3 2024-03-25T18:31:07.202 INFO:teuthology.misc.health.smithi001.stderr:0: v1:172.21.15.1:6789/0 mon.a 2024-03-25T18:31:07.202 INFO:teuthology.misc.health.smithi001.stderr:1: v1:172.21.15.115:6789/0 mon.b 2024-03-25T18:31:07.202 INFO:teuthology.misc.health.smithi001.stderr:2: v1:172.21.15.1:6790/0 mon.c 2024-03-25T18:31:07.203 INFO:teuthology.misc.health.smithi001.stderr: 2024-03-25T18:31:07.203 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:31:07.200+0000 7f75497fa640 4 set_mon_vals no callback set 2024-03-25T18:31:07.203 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:31:07.201+0000 7f7533fff640 1 -- 172.21.15.1:0/1946982626 <== mon.0 v1:172.21.15.1:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 21787975 0 0) 0x7f7544019940 con 0x7f754c13c480 2024-03-25T18:31:07.203 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:31:07.202+0000 7f754affd640 10 monclient: get_auth_request con 0x7f753806ddd0 auth_method 0 2024-03-25T18:31:07.204 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:31:07.202+0000 7f7533fff640 10 client.4410.objecter ms_handle_connect 0x7f753806ddd0 2024-03-25T18:31:07.207 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:31:07.206+0000 7f7533fff640 1 -- 172.21.15.1:0/1946982626 <== mon.0 v1:172.21.15.1:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 996254230 0 0) 0x7f75440535d0 con 0x7f754c13c480 2024-03-25T18:31:07.208 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:31:07.206+0000 7f7533fff640 10 client.4410.objecter ms_dispatch 0x7f754c000b90 osd_map(10..10 src has 1..10) v4 2024-03-25T18:31:07.208 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:31:07.206+0000 7f7533fff640 3 client.4410.objecter handle_osd_map got epochs [10,10] > 0 2024-03-25T18:31:07.208 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:31:07.206+0000 7f7533fff640 3 client.4410.objecter handle_osd_map decoding full epoch 10 2024-03-25T18:31:07.208 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:31:07.206+0000 7f7533fff640 20 client.4410.objecter dump_active .. 0 homeless 2024-03-25T18:31:07.208 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:31:07.207+0000 7f7533fff640 1 -- 172.21.15.1:0/1946982626 <== mon.0 v1:172.21.15.1:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f754404f0e0 con 0x7f754c13c480 2024-03-25T18:31:07.208 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:31:07.207+0000 7f7533fff640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:31:07.208 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:31:07.207+0000 7f7533fff640 10 monclient: _finish_command 1 = system:0 2024-03-25T18:31:07.461 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:31:07.460+0000 7f755184a640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:31:07.461 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:31:07.460+0000 7f755184a640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:31:07.461 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:31:07.460+0000 7f755184a640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.1:6789/0 2024-03-25T18:31:07.462 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:31:07.460+0000 7f755184a640 1 -- 172.21.15.1:0/1946982626 --> v1:172.21.15.1:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f754c143870 con 0x7f754c13c480 2024-03-25T18:31:07.518 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:31:07.517+0000 7f7533fff640 1 -- 172.21.15.1:0/1946982626 <== mon.0 v1:172.21.15.1:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 4092213348) 0x7f7544051010 con 0x7f754c13c480 2024-03-25T18:31:07.519 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:31:07.518+0000 7f7533fff640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:31:07.519 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:31:07.518+0000 7f7533fff640 10 monclient: _finish_command 2 = system:0 2024-03-25T18:31:07.519 INFO:teuthology.misc.health.smithi001.stdout: 2024-03-25T18:31:07.519 INFO:teuthology.misc.health.smithi001.stdout:{"epoch":10,"fsid":"c8ea72cf-9e99-4383-9b52-6c23d365fdfc","created":"2024-03-25T18:26:38.856638+0000","modified":"2024-03-25T18:26:44.224208+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T18:26:44.224208+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"b9f34922-3a84-49be-8426-807c1eceaefc","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"42bf306f-55e3-4634-9874-8b4b73ac6644","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"292cb23a-18a9-4207-a7cc-4d5727feaec0","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"45e0e9ab-898d-4673-9590-b4117bc5e84a","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"831a5a09-73ab-4a61-9872-088e76bdbc66","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"88cd903d-3b55-4840-9396-6e3cfe005282","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"89db7355-bc9d-46f4-8ab0-e8d9e744e026","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"cac47bee-0914-4f08-896f-348838882866","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T18:31:07.522 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:31:07.520+0000 7f755184a640 10 librados: watch_flush enter 2024-03-25T18:31:07.522 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:31:07.520+0000 7f755184a640 10 librados: watch_flush exit 2024-03-25T18:31:07.522 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:31:07.520+0000 7f755184a640 20 client.4410.objecter shutdown clearing up homeless session... 2024-03-25T18:31:07.522 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:31:07.520+0000 7f755184a640 10 client.4410.objecter successfully canceled tick 2024-03-25T18:31:07.522 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:31:07.520+0000 7f755184a640 1 -- 172.21.15.1:0/1946982626 >> v1:172.21.15.115:6800/4237379176 conn(0x7f753806ddd0 legacy=0x7f7538070290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:31:07.522 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:31:07.520+0000 7f755184a640 10 monclient: shutdown 2024-03-25T18:31:07.523 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:31:07.520+0000 7f755184a640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:31:07.523 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:31:07.520+0000 7f755184a640 1 -- 172.21.15.1:0/1946982626 >> v1:172.21.15.1:6789/0 conn(0x7f754c13c480 legacy=0x7f754c13d580 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:31:07.523 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:31:07.520+0000 7f755184a640 1 -- 172.21.15.1:0/1946982626 shutdown_connections 2024-03-25T18:31:07.523 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:31:07.520+0000 7f755184a640 1 -- 172.21.15.1:0/1946982626 >> 172.21.15.1:0/1946982626 conn(0x7f754c0617a0 msgr2=0x7f754c1407f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:31:07.523 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:31:07.521+0000 7f755184a640 1 -- 172.21.15.1:0/1946982626 shutdown_connections 2024-03-25T18:31:07.523 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:31:07.521+0000 7f755184a640 1 -- 172.21.15.1:0/1946982626 wait complete. 2024-03-25T18:31:07.523 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:31:07.521+0000 7f755184a640 1 librados: shutdown 2024-03-25T18:31:07.535 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T18:31:14.343 DEBUG:teuthology.orchestra.run.smithi001:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T18:31:14.456 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:31:14.455+0000 7f1c41274640 10 monclient: get_monmap_and_config 2024-03-25T18:31:14.456 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:31:14.455+0000 7f1c41274640 10 monclient: build_initial_monmap 2024-03-25T18:31:14.456 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:31:14.455+0000 7f1c41274640 10 monclient: monmap: 2024-03-25T18:31:14.456 INFO:teuthology.misc.health.smithi001.stderr:epoch 0 2024-03-25T18:31:14.456 INFO:teuthology.misc.health.smithi001.stderr:fsid c8ea72cf-9e99-4383-9b52-6c23d365fdfc 2024-03-25T18:31:14.456 INFO:teuthology.misc.health.smithi001.stderr:last_changed 2024-03-25T18:31:14.455943+0000 2024-03-25T18:31:14.456 INFO:teuthology.misc.health.smithi001.stderr:created 2024-03-25T18:31:14.455943+0000 2024-03-25T18:31:14.456 INFO:teuthology.misc.health.smithi001.stderr:min_mon_release 0 (unknown) 2024-03-25T18:31:14.457 INFO:teuthology.misc.health.smithi001.stderr:election_strategy: 3 2024-03-25T18:31:14.457 INFO:teuthology.misc.health.smithi001.stderr:0: v1:172.21.15.1:6789/0 mon.noname-a 2024-03-25T18:31:14.457 INFO:teuthology.misc.health.smithi001.stderr:1: v1:172.21.15.115:6789/0 mon.noname-c 2024-03-25T18:31:14.457 INFO:teuthology.misc.health.smithi001.stderr:2: v1:172.21.15.1:6790/0 mon.noname-b 2024-03-25T18:31:14.457 INFO:teuthology.misc.health.smithi001.stderr: 2024-03-25T18:31:14.458 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:31:14.456+0000 7f1c41274640 1 Processor -- start 2024-03-25T18:31:14.458 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:31:14.456+0000 7f1c41274640 1 -- start start 2024-03-25T18:31:14.458 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:31:14.456+0000 7f1c41274640 10 monclient: init 2024-03-25T18:31:14.458 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:31:14.456+0000 7f1c41274640 10 monclient: _reopen_session rank -1 2024-03-25T18:31:14.458 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:31:14.456+0000 7f1c41274640 10 monclient: _add_conns ranks=[1,0,2] 2024-03-25T18:31:14.458 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:31:14.456+0000 7f1c41274640 10 monclient(hunting): picked mon.noname-c con 0x7f1c3c0b4ff0 addr v1:172.21.15.115:6789/0 2024-03-25T18:31:14.458 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:31:14.456+0000 7f1c41274640 10 monclient(hunting): picked mon.noname-a con 0x7f1c3c0b4670 addr v1:172.21.15.1:6789/0 2024-03-25T18:31:14.458 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:31:14.456+0000 7f1c41274640 10 monclient(hunting): picked mon.noname-b con 0x7f1c3c13c480 addr v1:172.21.15.1:6790/0 2024-03-25T18:31:14.459 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:31:14.456+0000 7f1c41274640 1 -- --> v1:172.21.15.1:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1c3c0ab0e0 con 0x7f1c3c0b4670 2024-03-25T18:31:14.459 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:31:14.456+0000 7f1c41274640 1 -- --> v1:172.21.15.115:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1c3c0ab3c0 con 0x7f1c3c0b4ff0 2024-03-25T18:31:14.459 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:31:14.456+0000 7f1c41274640 1 -- --> v1:172.21.15.1:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1c3c0a9f30 con 0x7f1c3c13c480 2024-03-25T18:31:14.459 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:31:14.456+0000 7f1c41274640 10 monclient(hunting): _renew_subs 2024-03-25T18:31:14.459 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:31:14.456+0000 7f1c41274640 10 monclient(hunting): authenticate will time out at 7754.926468s 2024-03-25T18:31:14.466 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:31:14.465+0000 7f1c3b7fe640 1 --1- >> v1:172.21.15.1:6790/0 conn(0x7f1c3c13c480 0x7f1c3c14dc00 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.1:6790/0 says I am v1:172.21.15.1:38668/0 (socket says 172.21.15.1:38668) 2024-03-25T18:31:14.466 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:31:14.465+0000 7f1c3b7fe640 1 -- 172.21.15.1:0/874866995 learned_addr learned my addr 172.21.15.1:0/874866995 (peer_addr_for_me v1:172.21.15.1:0/0) 2024-03-25T18:31:14.473 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:31:14.472+0000 7f1c39ffb640 1 -- 172.21.15.1:0/874866995 <== mon.2 v1:172.21.15.1:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 56601619 0 0) 0x7f1c3c0a9f30 con 0x7f1c3c13c480 2024-03-25T18:31:14.473 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:31:14.472+0000 7f1c39ffb640 10 monclient(hunting): _negotiate 2024-03-25T18:31:14.473 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:31:14.472+0000 7f1c39ffb640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:31:14.473 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:31:14.472+0000 7f1c39ffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:31:14.473 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:31:14.472+0000 7f1c39ffb640 10 monclient(hunting): my global_id is 4424 2024-03-25T18:31:14.473 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:31:14.472+0000 7f1c39ffb640 1 -- 172.21.15.1:0/874866995 --> v1:172.21.15.1:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f1c24003520 con 0x7f1c3c13c480 2024-03-25T18:31:14.480 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:31:14.480+0000 7f1c39ffb640 1 -- 172.21.15.1:0/874866995 <== mon.2 v1:172.21.15.1:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3427389056 0 0) 0x7f1c24003520 con 0x7f1c3c13c480 2024-03-25T18:31:14.481 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:31:14.480+0000 7f1c39ffb640 1 -- 172.21.15.1:0/874866995 >> v1:172.21.15.115:6789/0 conn(0x7f1c3c0b4ff0 legacy=0x7f1c3c13bd70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:31:14.481 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:31:14.480+0000 7f1c39ffb640 1 -- 172.21.15.1:0/874866995 >> v1:172.21.15.1:6789/0 conn(0x7f1c3c0b4670 legacy=0x7f1c3c1425c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-25T18:31:14.481 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:31:14.480+0000 7f1c39ffb640 10 monclient: _finish_hunting 0 2024-03-25T18:31:14.481 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:31:14.480+0000 7f1c39ffb640 1 monclient: found mon.noname-b 2024-03-25T18:31:14.481 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:31:14.480+0000 7f1c39ffb640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:31:14.481 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:31:14.480+0000 7f1c39ffb640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.1:6790/0 2024-03-25T18:31:14.481 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:31:14.480+0000 7f1c39ffb640 1 -- 172.21.15.1:0/874866995 --> v1:172.21.15.1:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1c3c0a9b70 con 0x7f1c3c13c480 2024-03-25T18:31:14.481 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:31:14.480+0000 7f1c39ffb640 10 monclient: _finish_auth 0 2024-03-25T18:31:14.481 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:31:14.480+0000 7f1c39ffb640 10 monclient: _check_auth_tickets 2024-03-25T18:31:14.481 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:31:14.480+0000 7f1c39ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:31:14.481 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:31:14.480+0000 7f1c39ffb640 1 -- 172.21.15.1:0/874866995 <== mon.2 v1:172.21.15.1:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1586965025 0 0) 0x7f1c34001a10 con 0x7f1c3c13c480 2024-03-25T18:31:14.481 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:31:14.480+0000 7f1c41274640 5 monclient: authenticate success, global_id 4424 2024-03-25T18:31:14.481 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:31:14.480+0000 7f1c41274640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T18:31:14.481 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:31:14.480+0000 7f1c39ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:31:14.482 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:31:14.480+0000 7f1c39ffb640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-03-25T18:31:14.482 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:31:14.480+0000 7f1c39ffb640 10 monclient: dump: 2024-03-25T18:31:14.482 INFO:teuthology.misc.health.smithi001.stderr:epoch 1 2024-03-25T18:31:14.482 INFO:teuthology.misc.health.smithi001.stderr:fsid c8ea72cf-9e99-4383-9b52-6c23d365fdfc 2024-03-25T18:31:14.482 INFO:teuthology.misc.health.smithi001.stderr:last_changed 2024-03-25T18:25:50.510519+0000 2024-03-25T18:31:14.482 INFO:teuthology.misc.health.smithi001.stderr:created 2024-03-25T18:25:50.510519+0000 2024-03-25T18:31:14.482 INFO:teuthology.misc.health.smithi001.stderr:min_mon_release 19 (squid) 2024-03-25T18:31:14.482 INFO:teuthology.misc.health.smithi001.stderr:election_strategy: 3 2024-03-25T18:31:14.482 INFO:teuthology.misc.health.smithi001.stderr:0: v1:172.21.15.1:6789/0 mon.a 2024-03-25T18:31:14.482 INFO:teuthology.misc.health.smithi001.stderr:1: v1:172.21.15.115:6789/0 mon.b 2024-03-25T18:31:14.482 INFO:teuthology.misc.health.smithi001.stderr:2: v1:172.21.15.1:6790/0 mon.c 2024-03-25T18:31:14.482 INFO:teuthology.misc.health.smithi001.stderr: 2024-03-25T18:31:14.482 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:31:14.480+0000 7f1c41274640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T18:31:14.487 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:31:14.486+0000 7f1c39ffb640 1 -- 172.21.15.1:0/874866995 <== mon.2 v1:172.21.15.1:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f1c34001be0 con 0x7f1c3c13c480 2024-03-25T18:31:14.487 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:31:14.486+0000 7f1c39ffb640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:31:14.487 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:31:14.486+0000 7f1c39ffb640 1 -- 172.21.15.1:0/874866995 <== mon.2 v1:172.21.15.1:6790/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1586965025 0 0) 0x7f1c34002030 con 0x7f1c3c13c480 2024-03-25T18:31:14.487 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:31:14.486+0000 7f1c39ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:31:14.487 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:31:14.486+0000 7f1c39ffb640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-25T18:31:14.487 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:31:14.486+0000 7f1c39ffb640 10 monclient: dump: 2024-03-25T18:31:14.487 INFO:teuthology.misc.health.smithi001.stderr:epoch 1 2024-03-25T18:31:14.487 INFO:teuthology.misc.health.smithi001.stderr:fsid c8ea72cf-9e99-4383-9b52-6c23d365fdfc 2024-03-25T18:31:14.487 INFO:teuthology.misc.health.smithi001.stderr:last_changed 2024-03-25T18:25:50.510519+0000 2024-03-25T18:31:14.487 INFO:teuthology.misc.health.smithi001.stderr:created 2024-03-25T18:25:50.510519+0000 2024-03-25T18:31:14.487 INFO:teuthology.misc.health.smithi001.stderr:min_mon_release 19 (squid) 2024-03-25T18:31:14.487 INFO:teuthology.misc.health.smithi001.stderr:election_strategy: 3 2024-03-25T18:31:14.488 INFO:teuthology.misc.health.smithi001.stderr:0: v1:172.21.15.1:6789/0 mon.a 2024-03-25T18:31:14.488 INFO:teuthology.misc.health.smithi001.stderr:1: v1:172.21.15.115:6789/0 mon.b 2024-03-25T18:31:14.488 INFO:teuthology.misc.health.smithi001.stderr:2: v1:172.21.15.1:6790/0 mon.c 2024-03-25T18:31:14.488 INFO:teuthology.misc.health.smithi001.stderr: 2024-03-25T18:31:14.488 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:31:14.486+0000 7f1c41274640 10 monclient: get_monmap_and_config success 2024-03-25T18:31:14.488 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:31:14.486+0000 7f1c41274640 4 set_mon_vals no callback set 2024-03-25T18:31:14.489 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:31:14.487+0000 7f1c41274640 10 monclient: shutdown 2024-03-25T18:31:14.489 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:31:14.487+0000 7f1c41274640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:31:14.489 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:31:14.487+0000 7f1c41274640 1 -- 172.21.15.1:0/874866995 >> v1:172.21.15.1:6790/0 conn(0x7f1c3c13c480 legacy=0x7f1c3c14dc00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:31:14.489 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:31:14.487+0000 7f1c41274640 1 -- 172.21.15.1:0/874866995 shutdown_connections 2024-03-25T18:31:14.490 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:31:14.487+0000 7f1c41274640 1 -- 172.21.15.1:0/874866995 >> 172.21.15.1:0/874866995 conn(0x7f1c3c0617a0 msgr2=0x7f1c3c061bb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:31:14.490 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:31:14.487+0000 7f1c41274640 1 -- 172.21.15.1:0/874866995 shutdown_connections 2024-03-25T18:31:14.490 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:31:14.487+0000 7f1c41274640 1 -- 172.21.15.1:0/874866995 wait complete. 2024-03-25T18:31:14.490 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:31:14.488+0000 7f1c41274640 10 monclient: build_initial_monmap 2024-03-25T18:31:14.490 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:31:14.488+0000 7f1c41274640 10 monclient: monmap: 2024-03-25T18:31:14.490 INFO:teuthology.misc.health.smithi001.stderr:epoch 0 2024-03-25T18:31:14.490 INFO:teuthology.misc.health.smithi001.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T18:31:14.490 INFO:teuthology.misc.health.smithi001.stderr:last_changed 0.000000 2024-03-25T18:31:14.490 INFO:teuthology.misc.health.smithi001.stderr:created 0.000000 2024-03-25T18:31:14.490 INFO:teuthology.misc.health.smithi001.stderr:min_mon_release 0 (unknown) 2024-03-25T18:31:14.490 INFO:teuthology.misc.health.smithi001.stderr:election_strategy: 1 2024-03-25T18:31:14.490 INFO:teuthology.misc.health.smithi001.stderr:0: v1:172.21.15.1:6789/0 mon.noname-a 2024-03-25T18:31:14.490 INFO:teuthology.misc.health.smithi001.stderr:1: v1:172.21.15.115:6789/0 mon.noname-b 2024-03-25T18:31:14.491 INFO:teuthology.misc.health.smithi001.stderr:2: v1:172.21.15.1:6790/0 mon.noname-c 2024-03-25T18:31:14.491 INFO:teuthology.misc.health.smithi001.stderr: 2024-03-25T18:31:14.491 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:31:14.488+0000 7f1c41274640 1 librados: starting msgr at 2024-03-25T18:31:14.491 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:31:14.488+0000 7f1c41274640 1 librados: starting objecter 2024-03-25T18:31:14.491 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:31:14.488+0000 7f1c41274640 1 Processor -- start 2024-03-25T18:31:14.492 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:31:14.489+0000 7f1c41274640 1 -- start start 2024-03-25T18:31:14.492 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:31:14.489+0000 7f1c41274640 1 librados: setting wanted keys 2024-03-25T18:31:14.492 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:31:14.489+0000 7f1c41274640 1 librados: calling monclient init 2024-03-25T18:31:14.492 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:31:14.489+0000 7f1c41274640 10 monclient: init 2024-03-25T18:31:14.492 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:31:14.490+0000 7f1c41274640 10 monclient: _reopen_session rank -1 2024-03-25T18:31:14.492 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:31:14.490+0000 7f1c41274640 10 monclient: _add_conns ranks=[0,2,1] 2024-03-25T18:31:14.492 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:31:14.490+0000 7f1c41274640 10 monclient(hunting): picked mon.noname-a con 0x7f1c3c0b4670 addr v1:172.21.15.1:6789/0 2024-03-25T18:31:14.493 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:31:14.490+0000 7f1c41274640 10 monclient(hunting): picked mon.noname-c con 0x7f1c3c0b4ff0 addr v1:172.21.15.1:6790/0 2024-03-25T18:31:14.493 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:31:14.490+0000 7f1c41274640 10 monclient(hunting): picked mon.noname-b con 0x7f1c3c13c480 addr v1:172.21.15.115:6789/0 2024-03-25T18:31:14.493 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:31:14.490+0000 7f1c41274640 1 -- --> v1:172.21.15.1:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1c3c130d50 con 0x7f1c3c0b4670 2024-03-25T18:31:14.493 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:31:14.490+0000 7f1c41274640 1 -- --> v1:172.21.15.115:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1c3c144700 con 0x7f1c3c13c480 2024-03-25T18:31:14.493 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:31:14.490+0000 7f1c41274640 1 -- --> v1:172.21.15.1:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1c3c130ad0 con 0x7f1c3c0b4ff0 2024-03-25T18:31:14.493 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:31:14.490+0000 7f1c41274640 10 monclient(hunting): _renew_subs 2024-03-25T18:31:14.493 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:31:14.490+0000 7f1c41274640 10 monclient(hunting): authenticate will time out at 7754.960392s 2024-03-25T18:31:14.496 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:31:14.495+0000 7f1c3affd640 1 --1- >> v1:172.21.15.1:6789/0 conn(0x7f1c3c0b4670 0x7f1c3c12fcb0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.1:6789/0 says I am v1:172.21.15.1:37174/0 (socket says 172.21.15.1:37174) 2024-03-25T18:31:14.496 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:31:14.495+0000 7f1c3affd640 1 -- 172.21.15.1:0/217204209 learned_addr learned my addr 172.21.15.1:0/217204209 (peer_addr_for_me v1:172.21.15.1:0/0) 2024-03-25T18:31:14.499 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:31:14.498+0000 7f1c237fe640 10 client.?.objecter ms_handle_connect 0x7f1c3c0b4670 2024-03-25T18:31:14.499 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:31:14.498+0000 7f1c237fe640 10 client.?.objecter resend_mon_ops 2024-03-25T18:31:14.499 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:31:14.499+0000 7f1c237fe640 10 client.?.objecter ms_handle_connect 0x7f1c3c0b4ff0 2024-03-25T18:31:14.499 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:31:14.499+0000 7f1c237fe640 10 client.?.objecter resend_mon_ops 2024-03-25T18:31:14.505 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:31:14.504+0000 7f1c237fe640 1 -- 172.21.15.1:0/217204209 <== mon.2 v1:172.21.15.1:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1684884924 0 0) 0x7f1c3c130ad0 con 0x7f1c3c0b4ff0 2024-03-25T18:31:14.505 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:31:14.504+0000 7f1c237fe640 10 monclient(hunting): _negotiate 2024-03-25T18:31:14.505 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:31:14.504+0000 7f1c237fe640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:31:14.505 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:31:14.504+0000 7f1c237fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:31:14.505 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:31:14.504+0000 7f1c237fe640 10 monclient(hunting): my global_id is 4427 2024-03-25T18:31:14.505 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:31:14.504+0000 7f1c237fe640 1 -- 172.21.15.1:0/217204209 --> v1:172.21.15.1:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f1c28004eb0 con 0x7f1c3c0b4ff0 2024-03-25T18:31:14.506 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:31:14.505+0000 7f1c237fe640 10 client.?.objecter ms_handle_connect 0x7f1c3c13c480 2024-03-25T18:31:14.506 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:31:14.505+0000 7f1c237fe640 10 client.?.objecter resend_mon_ops 2024-03-25T18:31:14.508 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:31:14.507+0000 7f1c237fe640 1 -- 172.21.15.1:0/217204209 <== mon.0 v1:172.21.15.1:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1141244353 0 0) 0x7f1c3c0ab3c0 con 0x7f1c3c0b4670 2024-03-25T18:31:14.508 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:31:14.507+0000 7f1c237fe640 10 monclient(hunting): _negotiate 2024-03-25T18:31:14.508 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:31:14.507+0000 7f1c237fe640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:31:14.508 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:31:14.507+0000 7f1c237fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:31:14.508 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:31:14.507+0000 7f1c237fe640 10 monclient(hunting): my global_id is 4413 2024-03-25T18:31:14.508 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:31:14.507+0000 7f1c237fe640 1 -- 172.21.15.1:0/217204209 --> v1:172.21.15.1:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f1c3c130ad0 con 0x7f1c3c0b4670 2024-03-25T18:31:14.512 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:31:14.511+0000 7f1c237fe640 1 -- 172.21.15.1:0/217204209 <== mon.1 v1:172.21.15.115:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1564962568 0 0) 0x7f1c3c144700 con 0x7f1c3c13c480 2024-03-25T18:31:14.512 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:31:14.511+0000 7f1c237fe640 10 monclient(hunting): _negotiate 2024-03-25T18:31:14.512 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:31:14.511+0000 7f1c237fe640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:31:14.512 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:31:14.511+0000 7f1c237fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:31:14.512 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:31:14.511+0000 7f1c237fe640 10 monclient(hunting): my global_id is 4423 2024-03-25T18:31:14.512 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:31:14.511+0000 7f1c237fe640 1 -- 172.21.15.1:0/217204209 --> v1:172.21.15.115:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f1c3c0ab3c0 con 0x7f1c3c13c480 2024-03-25T18:31:14.517 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:31:14.516+0000 7f1c237fe640 1 -- 172.21.15.1:0/217204209 <== mon.0 v1:172.21.15.1:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2295321796 0 0) 0x7f1c3c130ad0 con 0x7f1c3c0b4670 2024-03-25T18:31:14.517 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:31:14.516+0000 7f1c237fe640 1 -- 172.21.15.1:0/217204209 --> v1:172.21.15.1:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f1c3c144700 con 0x7f1c3c0b4670 2024-03-25T18:31:14.517 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:31:14.516+0000 7f1c237fe640 1 -- 172.21.15.1:0/217204209 <== mon.0 v1:172.21.15.1:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1586965025 0 0) 0x7f1c2c001810 con 0x7f1c3c0b4670 2024-03-25T18:31:14.517 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:31:14.516+0000 7f1c237fe640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:31:14.517 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:31:14.516+0000 7f1c237fe640 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2024-03-25T18:31:14.517 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:31:14.516+0000 7f1c237fe640 10 monclient(hunting): dump: 2024-03-25T18:31:14.517 INFO:teuthology.misc.health.smithi001.stderr:epoch 1 2024-03-25T18:31:14.517 INFO:teuthology.misc.health.smithi001.stderr:fsid c8ea72cf-9e99-4383-9b52-6c23d365fdfc 2024-03-25T18:31:14.518 INFO:teuthology.misc.health.smithi001.stderr:last_changed 2024-03-25T18:25:50.510519+0000 2024-03-25T18:31:14.518 INFO:teuthology.misc.health.smithi001.stderr:created 2024-03-25T18:25:50.510519+0000 2024-03-25T18:31:14.518 INFO:teuthology.misc.health.smithi001.stderr:min_mon_release 19 (squid) 2024-03-25T18:31:14.518 INFO:teuthology.misc.health.smithi001.stderr:election_strategy: 3 2024-03-25T18:31:14.518 INFO:teuthology.misc.health.smithi001.stderr:0: v1:172.21.15.1:6789/0 mon.a 2024-03-25T18:31:14.518 INFO:teuthology.misc.health.smithi001.stderr:1: v1:172.21.15.115:6789/0 mon.b 2024-03-25T18:31:14.518 INFO:teuthology.misc.health.smithi001.stderr:2: v1:172.21.15.1:6790/0 mon.c 2024-03-25T18:31:14.518 INFO:teuthology.misc.health.smithi001.stderr: 2024-03-25T18:31:14.518 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:31:14.516+0000 7f1c237fe640 10 monclient(hunting): _finish_auth 0 2024-03-25T18:31:14.519 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:31:14.517+0000 7f1c237fe640 1 -- 172.21.15.1:0/217204209 <== mon.2 v1:172.21.15.1:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 4022018768 0 0) 0x7f1c28004eb0 con 0x7f1c3c0b4ff0 2024-03-25T18:31:14.519 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:31:14.517+0000 7f1c237fe640 1 -- 172.21.15.1:0/217204209 --> v1:172.21.15.1:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f1c3c130ad0 con 0x7f1c3c0b4ff0 2024-03-25T18:31:14.519 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:31:14.517+0000 7f1c237fe640 1 -- 172.21.15.1:0/217204209 <== mon.2 v1:172.21.15.1:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1586965025 0 0) 0x7f1c30001510 con 0x7f1c3c0b4ff0 2024-03-25T18:31:14.519 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:31:14.517+0000 7f1c237fe640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:31:14.519 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:31:14.517+0000 7f1c237fe640 10 monclient(hunting): got monmap 1 from mon.c (according to old e1) 2024-03-25T18:31:14.519 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:31:14.517+0000 7f1c237fe640 10 monclient(hunting): dump: 2024-03-25T18:31:14.519 INFO:teuthology.misc.health.smithi001.stderr:epoch 1 2024-03-25T18:31:14.519 INFO:teuthology.misc.health.smithi001.stderr:fsid c8ea72cf-9e99-4383-9b52-6c23d365fdfc 2024-03-25T18:31:14.519 INFO:teuthology.misc.health.smithi001.stderr:last_changed 2024-03-25T18:25:50.510519+0000 2024-03-25T18:31:14.519 INFO:teuthology.misc.health.smithi001.stderr:created 2024-03-25T18:25:50.510519+0000 2024-03-25T18:31:14.519 INFO:teuthology.misc.health.smithi001.stderr:min_mon_release 19 (squid) 2024-03-25T18:31:14.519 INFO:teuthology.misc.health.smithi001.stderr:election_strategy: 3 2024-03-25T18:31:14.520 INFO:teuthology.misc.health.smithi001.stderr:0: v1:172.21.15.1:6789/0 mon.a 2024-03-25T18:31:14.520 INFO:teuthology.misc.health.smithi001.stderr:1: v1:172.21.15.115:6789/0 mon.b 2024-03-25T18:31:14.520 INFO:teuthology.misc.health.smithi001.stderr:2: v1:172.21.15.1:6790/0 mon.c 2024-03-25T18:31:14.520 INFO:teuthology.misc.health.smithi001.stderr: 2024-03-25T18:31:14.522 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:31:14.521+0000 7f1c237fe640 1 -- 172.21.15.1:0/217204209 <== mon.2 v1:172.21.15.1:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 1236455005 0 0) 0x7f1c3c130ad0 con 0x7f1c3c0b4ff0 2024-03-25T18:31:14.522 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:31:14.521+0000 7f1c237fe640 1 -- 172.21.15.1:0/217204209 >> v1:172.21.15.115:6789/0 conn(0x7f1c3c13c480 legacy=0x7f1c3c1303c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:31:14.522 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:31:14.521+0000 7f1c237fe640 1 -- 172.21.15.1:0/217204209 >> v1:172.21.15.1:6789/0 conn(0x7f1c3c0b4670 legacy=0x7f1c3c12fcb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:31:14.522 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:31:14.521+0000 7f1c237fe640 10 monclient: _finish_hunting 0 2024-03-25T18:31:14.522 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:31:14.521+0000 7f1c237fe640 1 monclient: found mon.c 2024-03-25T18:31:14.523 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:31:14.521+0000 7f1c237fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:31:14.523 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:31:14.521+0000 7f1c237fe640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.1:6790/0 2024-03-25T18:31:14.523 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:31:14.521+0000 7f1c237fe640 1 -- 172.21.15.1:0/217204209 --> v1:172.21.15.1:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1c3c144970 con 0x7f1c3c0b4ff0 2024-03-25T18:31:14.523 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:31:14.521+0000 7f1c237fe640 10 monclient: _finish_auth 0 2024-03-25T18:31:14.523 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:31:14.521+0000 7f1c237fe640 10 monclient: _check_auth_tickets 2024-03-25T18:31:14.523 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:31:14.521+0000 7f1c237fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:31:14.523 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:31:14.521+0000 7f1c41274640 5 monclient: authenticate success, global_id 4427 2024-03-25T18:31:14.523 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:31:14.521+0000 7f1c41274640 10 monclient: _renew_subs 2024-03-25T18:31:14.523 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:31:14.521+0000 7f1c41274640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.1:6790/0 2024-03-25T18:31:14.523 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:31:14.521+0000 7f1c41274640 1 -- 172.21.15.1:0/217204209 --> v1:172.21.15.1:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1c3c144c50 con 0x7f1c3c0b4ff0 2024-03-25T18:31:14.523 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:31:14.521+0000 7f1c41274640 10 client.4427.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T18:31:14.523 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:31:14.521+0000 7f1c41274640 10 monclient: _renew_subs 2024-03-25T18:31:14.523 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:31:14.521+0000 7f1c41274640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.1:6790/0 2024-03-25T18:31:14.523 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:31:14.521+0000 7f1c41274640 1 -- 172.21.15.1:0/217204209 --> v1:172.21.15.1:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f1c3c13f880 con 0x7f1c3c0b4ff0 2024-03-25T18:31:14.523 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:31:14.521+0000 7f1c41274640 1 librados: init done 2024-03-25T18:31:14.524 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:31:14.522+0000 7f1c41274640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T18:31:14.524 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:31:14.522+0000 7f1c41274640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:31:14.524 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:31:14.522+0000 7f1c41274640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.1:6790/0 2024-03-25T18:31:14.524 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:31:14.522+0000 7f1c41274640 1 -- 172.21.15.1:0/217204209 --> v1:172.21.15.1:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1c3c143f40 con 0x7f1c3c0b4ff0 2024-03-25T18:31:14.836 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:31:14.835+0000 7f1c237fe640 1 -- 172.21.15.1:0/217204209 <== mon.2 v1:172.21.15.1:6790/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f1c30001a20 con 0x7f1c3c0b4ff0 2024-03-25T18:31:14.836 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:31:14.835+0000 7f1c237fe640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:31:14.836 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:31:14.835+0000 7f1c237fe640 1 -- 172.21.15.1:0/217204209 <== mon.2 v1:172.21.15.1:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1586965025 0 0) 0x7f1c30003910 con 0x7f1c3c0b4ff0 2024-03-25T18:31:14.836 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:31:14.835+0000 7f1c237fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:31:14.836 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:31:14.835+0000 7f1c237fe640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-25T18:31:14.837 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:31:14.835+0000 7f1c237fe640 10 monclient: dump: 2024-03-25T18:31:14.837 INFO:teuthology.misc.health.smithi001.stderr:epoch 1 2024-03-25T18:31:14.837 INFO:teuthology.misc.health.smithi001.stderr:fsid c8ea72cf-9e99-4383-9b52-6c23d365fdfc 2024-03-25T18:31:14.837 INFO:teuthology.misc.health.smithi001.stderr:last_changed 2024-03-25T18:25:50.510519+0000 2024-03-25T18:31:14.837 INFO:teuthology.misc.health.smithi001.stderr:created 2024-03-25T18:25:50.510519+0000 2024-03-25T18:31:14.837 INFO:teuthology.misc.health.smithi001.stderr:min_mon_release 19 (squid) 2024-03-25T18:31:14.837 INFO:teuthology.misc.health.smithi001.stderr:election_strategy: 3 2024-03-25T18:31:14.837 INFO:teuthology.misc.health.smithi001.stderr:0: v1:172.21.15.1:6789/0 mon.a 2024-03-25T18:31:14.837 INFO:teuthology.misc.health.smithi001.stderr:1: v1:172.21.15.115:6789/0 mon.b 2024-03-25T18:31:14.837 INFO:teuthology.misc.health.smithi001.stderr:2: v1:172.21.15.1:6790/0 mon.c 2024-03-25T18:31:14.837 INFO:teuthology.misc.health.smithi001.stderr: 2024-03-25T18:31:14.837 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:31:14.835+0000 7f1c397fa640 4 set_mon_vals no callback set 2024-03-25T18:31:14.838 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:31:14.836+0000 7f1c237fe640 1 -- 172.21.15.1:0/217204209 <== mon.2 v1:172.21.15.1:6790/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 21787975 0 0) 0x7f1c300198e0 con 0x7f1c3c0b4ff0 2024-03-25T18:31:14.838 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:31:14.837+0000 7f1c3affd640 10 monclient: get_auth_request con 0x7f1c2806dfb0 auth_method 0 2024-03-25T18:31:14.838 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:31:14.837+0000 7f1c237fe640 10 client.4427.objecter ms_handle_connect 0x7f1c2806dfb0 2024-03-25T18:31:14.844 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:31:14.843+0000 7f1c237fe640 1 -- 172.21.15.1:0/217204209 <== mon.2 v1:172.21.15.1:6790/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 996254230 0 0) 0x7f1c30053580 con 0x7f1c3c0b4ff0 2024-03-25T18:31:14.844 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:31:14.843+0000 7f1c237fe640 10 client.4427.objecter ms_dispatch 0x7f1c3c000b90 osd_map(10..10 src has 1..10) v4 2024-03-25T18:31:14.844 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:31:14.843+0000 7f1c237fe640 3 client.4427.objecter handle_osd_map got epochs [10,10] > 0 2024-03-25T18:31:14.844 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:31:14.843+0000 7f1c237fe640 3 client.4427.objecter handle_osd_map decoding full epoch 10 2024-03-25T18:31:14.844 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:31:14.843+0000 7f1c237fe640 20 client.4427.objecter dump_active .. 0 homeless 2024-03-25T18:31:14.844 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:31:14.843+0000 7f1c237fe640 1 -- 172.21.15.1:0/217204209 <== mon.2 v1:172.21.15.1:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f1c3004eec0 con 0x7f1c3c0b4ff0 2024-03-25T18:31:14.844 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:31:14.843+0000 7f1c237fe640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:31:14.844 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:31:14.843+0000 7f1c237fe640 10 monclient: _finish_command 1 = system:0 2024-03-25T18:31:15.063 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:31:15.062+0000 7f1c41274640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:31:15.063 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:31:15.062+0000 7f1c41274640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:31:15.063 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:31:15.062+0000 7f1c41274640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.1:6790/0 2024-03-25T18:31:15.063 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:31:15.062+0000 7f1c41274640 1 -- 172.21.15.1:0/217204209 --> v1:172.21.15.1:6790/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f1c3c0688c0 con 0x7f1c3c0b4ff0 2024-03-25T18:31:15.120 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:31:15.119+0000 7f1c237fe640 1 -- 172.21.15.1:0/217204209 <== mon.2 v1:172.21.15.1:6790/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 4092213348) 0x7f1c30050fc0 con 0x7f1c3c0b4ff0 2024-03-25T18:31:15.120 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:31:15.119+0000 7f1c237fe640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:31:15.120 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:31:15.119+0000 7f1c237fe640 10 monclient: _finish_command 2 = system:0 2024-03-25T18:31:15.120 INFO:teuthology.misc.health.smithi001.stdout: 2024-03-25T18:31:15.120 INFO:teuthology.misc.health.smithi001.stdout:{"epoch":10,"fsid":"c8ea72cf-9e99-4383-9b52-6c23d365fdfc","created":"2024-03-25T18:26:38.856638+0000","modified":"2024-03-25T18:26:44.224208+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T18:26:44.224208+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"b9f34922-3a84-49be-8426-807c1eceaefc","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"42bf306f-55e3-4634-9874-8b4b73ac6644","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"292cb23a-18a9-4207-a7cc-4d5727feaec0","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"45e0e9ab-898d-4673-9590-b4117bc5e84a","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"831a5a09-73ab-4a61-9872-088e76bdbc66","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"88cd903d-3b55-4840-9396-6e3cfe005282","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"89db7355-bc9d-46f4-8ab0-e8d9e744e026","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"cac47bee-0914-4f08-896f-348838882866","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T18:31:15.123 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:31:15.121+0000 7f1c41274640 10 librados: watch_flush enter 2024-03-25T18:31:15.123 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:31:15.121+0000 7f1c41274640 10 librados: watch_flush exit 2024-03-25T18:31:15.123 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:31:15.121+0000 7f1c41274640 20 client.4427.objecter shutdown clearing up homeless session... 2024-03-25T18:31:15.123 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:31:15.121+0000 7f1c41274640 10 client.4427.objecter successfully canceled tick 2024-03-25T18:31:15.123 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:31:15.121+0000 7f1c41274640 1 -- 172.21.15.1:0/217204209 >> v1:172.21.15.115:6800/4237379176 conn(0x7f1c2806dfb0 legacy=0x7f1c28070470 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:31:15.123 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:31:15.121+0000 7f1c41274640 10 monclient: shutdown 2024-03-25T18:31:15.123 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:31:15.121+0000 7f1c41274640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:31:15.123 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:31:15.121+0000 7f1c41274640 1 -- 172.21.15.1:0/217204209 >> v1:172.21.15.1:6790/0 conn(0x7f1c3c0b4ff0 legacy=0x7f1c3c13d940 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:31:15.123 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:31:15.121+0000 7f1c41274640 1 -- 172.21.15.1:0/217204209 shutdown_connections 2024-03-25T18:31:15.124 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:31:15.121+0000 7f1c41274640 1 -- 172.21.15.1:0/217204209 >> 172.21.15.1:0/217204209 conn(0x7f1c3c0617a0 msgr2=0x7f1c3c061b80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:31:15.124 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:31:15.121+0000 7f1c41274640 1 -- 172.21.15.1:0/217204209 shutdown_connections 2024-03-25T18:31:15.124 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:31:15.121+0000 7f1c41274640 1 -- 172.21.15.1:0/217204209 wait complete. 2024-03-25T18:31:15.124 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:31:15.121+0000 7f1c41274640 1 librados: shutdown 2024-03-25T18:31:15.135 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T18:31:16.681 INFO:tasks.ceph.mon.b.smithi115.stderr:==00:00:04:55.164 34836== Warning: unimplemented fcntl command: 1036 2024-03-25T18:31:16.682 INFO:tasks.ceph.mon.a.smithi001.stderr:==00:00:04:55.240 34876== Warning: unimplemented fcntl command: 1036 2024-03-25T18:31:16.690 INFO:tasks.ceph.mon.b.smithi115.stderr:==00:00:04:55.174 34836== Warning: unimplemented fcntl command: 1036 2024-03-25T18:31:16.695 INFO:tasks.ceph.mon.a.smithi001.stderr:==00:00:04:55.253 34876== Warning: unimplemented fcntl command: 1036 2024-03-25T18:31:16.723 INFO:tasks.ceph.mon.a.smithi001.stderr:==00:00:04:55.281 34876== Warning: unimplemented fcntl command: 1036 2024-03-25T18:31:16.723 INFO:tasks.ceph.mon.b.smithi115.stderr:==00:00:04:55.207 34836== Warning: unimplemented fcntl command: 1036 2024-03-25T18:31:16.749 INFO:tasks.ceph.mon.c.smithi001.stderr:==00:00:04:55.307 34877== Warning: unimplemented fcntl command: 1036 2024-03-25T18:31:16.758 INFO:tasks.ceph.mon.c.smithi001.stderr:==00:00:04:55.316 34877== Warning: unimplemented fcntl command: 1036 2024-03-25T18:31:16.787 INFO:tasks.ceph.mon.c.smithi001.stderr:==00:00:04:55.346 34877== Warning: unimplemented fcntl command: 1036 2024-03-25T18:31:21.942 DEBUG:teuthology.orchestra.run.smithi001:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T18:31:22.058 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:31:22.056+0000 7f63f83bc640 10 monclient: get_monmap_and_config 2024-03-25T18:31:22.058 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:31:22.056+0000 7f63f83bc640 10 monclient: build_initial_monmap 2024-03-25T18:31:22.058 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:31:22.056+0000 7f63f83bc640 10 monclient: monmap: 2024-03-25T18:31:22.058 INFO:teuthology.misc.health.smithi001.stderr:epoch 0 2024-03-25T18:31:22.058 INFO:teuthology.misc.health.smithi001.stderr:fsid c8ea72cf-9e99-4383-9b52-6c23d365fdfc 2024-03-25T18:31:22.058 INFO:teuthology.misc.health.smithi001.stderr:last_changed 2024-03-25T18:31:22.057843+0000 2024-03-25T18:31:22.058 INFO:teuthology.misc.health.smithi001.stderr:created 2024-03-25T18:31:22.057843+0000 2024-03-25T18:31:22.059 INFO:teuthology.misc.health.smithi001.stderr:min_mon_release 0 (unknown) 2024-03-25T18:31:22.059 INFO:teuthology.misc.health.smithi001.stderr:election_strategy: 3 2024-03-25T18:31:22.059 INFO:teuthology.misc.health.smithi001.stderr:0: v1:172.21.15.1:6789/0 mon.noname-a 2024-03-25T18:31:22.059 INFO:teuthology.misc.health.smithi001.stderr:1: v1:172.21.15.115:6789/0 mon.noname-c 2024-03-25T18:31:22.059 INFO:teuthology.misc.health.smithi001.stderr:2: v1:172.21.15.1:6790/0 mon.noname-b 2024-03-25T18:31:22.059 INFO:teuthology.misc.health.smithi001.stderr: 2024-03-25T18:31:22.060 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:31:22.056+0000 7f63f83bc640 1 Processor -- start 2024-03-25T18:31:22.060 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:31:22.056+0000 7f63f83bc640 1 -- start start 2024-03-25T18:31:22.060 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:31:22.056+0000 7f63f83bc640 10 monclient: init 2024-03-25T18:31:22.060 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:31:22.057+0000 7f63f83bc640 10 monclient: _reopen_session rank -1 2024-03-25T18:31:22.060 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:31:22.057+0000 7f63f83bc640 10 monclient: _add_conns ranks=[1,0,2] 2024-03-25T18:31:22.060 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:31:22.057+0000 7f63f83bc640 10 monclient(hunting): picked mon.noname-c con 0x7f63f00b4e70 addr v1:172.21.15.115:6789/0 2024-03-25T18:31:22.060 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:31:22.057+0000 7f63f83bc640 10 monclient(hunting): picked mon.noname-a con 0x7f63f00b44b0 addr v1:172.21.15.1:6789/0 2024-03-25T18:31:22.060 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:31:22.057+0000 7f63f83bc640 10 monclient(hunting): picked mon.noname-b con 0x7f63f00b1b00 addr v1:172.21.15.1:6790/0 2024-03-25T18:31:22.061 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:31:22.057+0000 7f63f83bc640 1 -- --> v1:172.21.15.1:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f63f00a96f0 con 0x7f63f00b44b0 2024-03-25T18:31:22.061 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:31:22.057+0000 7f63f83bc640 1 -- --> v1:172.21.15.115:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f63f00ab0e0 con 0x7f63f00b4e70 2024-03-25T18:31:22.061 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:31:22.057+0000 7f63f83bc640 1 -- --> v1:172.21.15.1:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f63f0145f70 con 0x7f63f00b1b00 2024-03-25T18:31:22.061 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:31:22.057+0000 7f63f83bc640 10 monclient(hunting): _renew_subs 2024-03-25T18:31:22.061 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:31:22.057+0000 7f63f83bc640 10 monclient(hunting): authenticate will time out at 7762.528483s 2024-03-25T18:31:22.068 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:31:22.066+0000 7f63f6158640 1 --1- >> v1:172.21.15.115:6789/0 conn(0x7f63f00b4e70 0x7f63f013f940 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.115:6789/0 says I am v1:172.21.15.1:40750/0 (socket says 172.21.15.1:40750) 2024-03-25T18:31:22.068 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:31:22.066+0000 7f63f6158640 1 -- 172.21.15.1:0/2115261023 learned_addr learned my addr 172.21.15.1:0/2115261023 (peer_addr_for_me v1:172.21.15.1:0/0) 2024-03-25T18:31:22.080 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:31:22.079+0000 7f63f5156640 1 -- 172.21.15.1:0/2115261023 <== mon.1 v1:172.21.15.115:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2015147813 0 0) 0x7f63f00ab0e0 con 0x7f63f00b4e70 2024-03-25T18:31:22.080 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:31:22.079+0000 7f63f5156640 10 monclient(hunting): _negotiate 2024-03-25T18:31:22.081 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:31:22.079+0000 7f63f5156640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:31:22.081 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:31:22.079+0000 7f63f5156640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:31:22.081 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:31:22.079+0000 7f63f5156640 10 monclient(hunting): my global_id is 4426 2024-03-25T18:31:22.081 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:31:22.079+0000 7f63f5156640 1 -- 172.21.15.1:0/2115261023 --> v1:172.21.15.115:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f63dc003520 con 0x7f63f00b4e70 2024-03-25T18:31:22.091 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:31:22.089+0000 7f63f5156640 1 -- 172.21.15.1:0/2115261023 <== mon.1 v1:172.21.15.115:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3960802252 0 0) 0x7f63dc003520 con 0x7f63f00b4e70 2024-03-25T18:31:22.091 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:31:22.089+0000 7f63f5156640 1 -- 172.21.15.1:0/2115261023 >> v1:172.21.15.1:6790/0 conn(0x7f63f00b1b00 legacy=0x7f63f014dc00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:31:22.091 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:31:22.089+0000 7f63f5156640 1 -- 172.21.15.1:0/2115261023 >> v1:172.21.15.1:6789/0 conn(0x7f63f00b44b0 legacy=0x7f63f0145860 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:31:22.091 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:31:22.089+0000 7f63f5156640 10 monclient: _finish_hunting 0 2024-03-25T18:31:22.091 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:31:22.089+0000 7f63f5156640 1 monclient: found mon.noname-c 2024-03-25T18:31:22.091 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:31:22.089+0000 7f63f5156640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:31:22.091 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:31:22.089+0000 7f63f5156640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.115:6789/0 2024-03-25T18:31:22.091 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:31:22.089+0000 7f63f5156640 1 -- 172.21.15.1:0/2115261023 --> v1:172.21.15.115:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f63f00a9b80 con 0x7f63f00b4e70 2024-03-25T18:31:22.091 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:31:22.089+0000 7f63f5156640 10 monclient: _finish_auth 0 2024-03-25T18:31:22.092 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:31:22.089+0000 7f63f5156640 10 monclient: _check_auth_tickets 2024-03-25T18:31:22.092 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:31:22.089+0000 7f63f5156640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:31:22.092 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:31:22.089+0000 7f63f5156640 1 -- 172.21.15.1:0/2115261023 <== mon.1 v1:172.21.15.115:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1586965025 0 0) 0x7f63ec001690 con 0x7f63f00b4e70 2024-03-25T18:31:22.092 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:31:22.089+0000 7f63f5156640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:31:22.092 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:31:22.089+0000 7f63f5156640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-03-25T18:31:22.092 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:31:22.089+0000 7f63f5156640 10 monclient: dump: 2024-03-25T18:31:22.092 INFO:teuthology.misc.health.smithi001.stderr:epoch 1 2024-03-25T18:31:22.092 INFO:teuthology.misc.health.smithi001.stderr:fsid c8ea72cf-9e99-4383-9b52-6c23d365fdfc 2024-03-25T18:31:22.092 INFO:teuthology.misc.health.smithi001.stderr:last_changed 2024-03-25T18:25:50.510519+0000 2024-03-25T18:31:22.092 INFO:teuthology.misc.health.smithi001.stderr:created 2024-03-25T18:25:50.510519+0000 2024-03-25T18:31:22.092 INFO:teuthology.misc.health.smithi001.stderr:min_mon_release 19 (squid) 2024-03-25T18:31:22.092 INFO:teuthology.misc.health.smithi001.stderr:election_strategy: 3 2024-03-25T18:31:22.092 INFO:teuthology.misc.health.smithi001.stderr:0: v1:172.21.15.1:6789/0 mon.a 2024-03-25T18:31:22.092 INFO:teuthology.misc.health.smithi001.stderr:1: v1:172.21.15.115:6789/0 mon.b 2024-03-25T18:31:22.092 INFO:teuthology.misc.health.smithi001.stderr:2: v1:172.21.15.1:6790/0 mon.c 2024-03-25T18:31:22.092 INFO:teuthology.misc.health.smithi001.stderr: 2024-03-25T18:31:22.093 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:31:22.089+0000 7f63f83bc640 5 monclient: authenticate success, global_id 4426 2024-03-25T18:31:22.093 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:31:22.089+0000 7f63f83bc640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T18:31:22.113 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:31:22.111+0000 7f63f5156640 1 -- 172.21.15.1:0/2115261023 <== mon.1 v1:172.21.15.115:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f63ec0017f0 con 0x7f63f00b4e70 2024-03-25T18:31:22.113 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:31:22.111+0000 7f63f5156640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:31:22.113 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:31:22.111+0000 7f63f5156640 1 -- 172.21.15.1:0/2115261023 <== mon.1 v1:172.21.15.115:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1586965025 0 0) 0x7f63ec001c00 con 0x7f63f00b4e70 2024-03-25T18:31:22.113 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:31:22.111+0000 7f63f5156640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:31:22.113 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:31:22.111+0000 7f63f5156640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-25T18:31:22.113 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:31:22.111+0000 7f63f5156640 10 monclient: dump: 2024-03-25T18:31:22.113 INFO:teuthology.misc.health.smithi001.stderr:epoch 1 2024-03-25T18:31:22.113 INFO:teuthology.misc.health.smithi001.stderr:fsid c8ea72cf-9e99-4383-9b52-6c23d365fdfc 2024-03-25T18:31:22.113 INFO:teuthology.misc.health.smithi001.stderr:last_changed 2024-03-25T18:25:50.510519+0000 2024-03-25T18:31:22.113 INFO:teuthology.misc.health.smithi001.stderr:created 2024-03-25T18:25:50.510519+0000 2024-03-25T18:31:22.114 INFO:teuthology.misc.health.smithi001.stderr:min_mon_release 19 (squid) 2024-03-25T18:31:22.114 INFO:teuthology.misc.health.smithi001.stderr:election_strategy: 3 2024-03-25T18:31:22.114 INFO:teuthology.misc.health.smithi001.stderr:0: v1:172.21.15.1:6789/0 mon.a 2024-03-25T18:31:22.114 INFO:teuthology.misc.health.smithi001.stderr:1: v1:172.21.15.115:6789/0 mon.b 2024-03-25T18:31:22.114 INFO:teuthology.misc.health.smithi001.stderr:2: v1:172.21.15.1:6790/0 mon.c 2024-03-25T18:31:22.114 INFO:teuthology.misc.health.smithi001.stderr: 2024-03-25T18:31:22.114 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:31:22.111+0000 7f63f83bc640 10 monclient: get_monmap_and_config success 2024-03-25T18:31:22.114 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:31:22.111+0000 7f63f83bc640 4 set_mon_vals no callback set 2024-03-25T18:31:22.116 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:31:22.112+0000 7f63f83bc640 10 monclient: shutdown 2024-03-25T18:31:22.116 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:31:22.112+0000 7f63f83bc640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:31:22.117 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:31:22.112+0000 7f63f83bc640 1 -- 172.21.15.1:0/2115261023 >> v1:172.21.15.115:6789/0 conn(0x7f63f00b4e70 legacy=0x7f63f013f940 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:31:22.117 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:31:22.112+0000 7f63f83bc640 1 -- 172.21.15.1:0/2115261023 shutdown_connections 2024-03-25T18:31:22.117 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:31:22.112+0000 7f63f83bc640 1 -- 172.21.15.1:0/2115261023 >> 172.21.15.1:0/2115261023 conn(0x7f63f00617a0 msgr2=0x7f63f0061bb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:31:22.117 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:31:22.112+0000 7f63f83bc640 1 -- 172.21.15.1:0/2115261023 shutdown_connections 2024-03-25T18:31:22.117 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:31:22.112+0000 7f63f83bc640 1 -- 172.21.15.1:0/2115261023 wait complete. 2024-03-25T18:31:22.117 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:31:22.113+0000 7f63f83bc640 10 monclient: build_initial_monmap 2024-03-25T18:31:22.117 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:31:22.113+0000 7f63f83bc640 10 monclient: monmap: 2024-03-25T18:31:22.117 INFO:teuthology.misc.health.smithi001.stderr:epoch 0 2024-03-25T18:31:22.117 INFO:teuthology.misc.health.smithi001.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T18:31:22.117 INFO:teuthology.misc.health.smithi001.stderr:last_changed 0.000000 2024-03-25T18:31:22.117 INFO:teuthology.misc.health.smithi001.stderr:created 0.000000 2024-03-25T18:31:22.117 INFO:teuthology.misc.health.smithi001.stderr:min_mon_release 0 (unknown) 2024-03-25T18:31:22.117 INFO:teuthology.misc.health.smithi001.stderr:election_strategy: 1 2024-03-25T18:31:22.117 INFO:teuthology.misc.health.smithi001.stderr:0: v1:172.21.15.1:6789/0 mon.noname-a 2024-03-25T18:31:22.117 INFO:teuthology.misc.health.smithi001.stderr:1: v1:172.21.15.115:6789/0 mon.noname-b 2024-03-25T18:31:22.118 INFO:teuthology.misc.health.smithi001.stderr:2: v1:172.21.15.1:6790/0 mon.noname-c 2024-03-25T18:31:22.118 INFO:teuthology.misc.health.smithi001.stderr: 2024-03-25T18:31:22.118 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:31:22.113+0000 7f63f83bc640 1 librados: starting msgr at 2024-03-25T18:31:22.118 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:31:22.113+0000 7f63f83bc640 1 librados: starting objecter 2024-03-25T18:31:22.118 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:31:22.113+0000 7f63f83bc640 1 Processor -- start 2024-03-25T18:31:22.118 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:31:22.113+0000 7f63f83bc640 1 -- start start 2024-03-25T18:31:22.118 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:31:22.113+0000 7f63f83bc640 1 librados: setting wanted keys 2024-03-25T18:31:22.118 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:31:22.113+0000 7f63f83bc640 1 librados: calling monclient init 2024-03-25T18:31:22.118 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:31:22.113+0000 7f63f83bc640 10 monclient: init 2024-03-25T18:31:22.118 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:31:22.114+0000 7f63f83bc640 10 monclient: _reopen_session rank -1 2024-03-25T18:31:22.118 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:31:22.114+0000 7f63f83bc640 10 monclient: _add_conns ranks=[1,2,0] 2024-03-25T18:31:22.118 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:31:22.114+0000 7f63f83bc640 10 monclient(hunting): picked mon.noname-b con 0x7f63f00b1b00 addr v1:172.21.15.115:6789/0 2024-03-25T18:31:22.118 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:31:22.114+0000 7f63f83bc640 10 monclient(hunting): picked mon.noname-c con 0x7f63f00b44b0 addr v1:172.21.15.1:6790/0 2024-03-25T18:31:22.118 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:31:22.114+0000 7f63f83bc640 10 monclient(hunting): picked mon.noname-a con 0x7f63f00b4e70 addr v1:172.21.15.1:6789/0 2024-03-25T18:31:22.118 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:31:22.114+0000 7f63f83bc640 1 -- --> v1:172.21.15.1:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f63f01d4540 con 0x7f63f00b4e70 2024-03-25T18:31:22.119 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:31:22.114+0000 7f63f83bc640 1 -- --> v1:172.21.15.115:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f63f0142730 con 0x7f63f00b1b00 2024-03-25T18:31:22.119 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:31:22.114+0000 7f63f83bc640 1 -- --> v1:172.21.15.1:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f63f0140160 con 0x7f63f00b44b0 2024-03-25T18:31:22.119 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:31:22.114+0000 7f63f83bc640 10 monclient(hunting): _renew_subs 2024-03-25T18:31:22.119 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:31:22.114+0000 7f63f83bc640 10 monclient(hunting): authenticate will time out at 7762.585634s 2024-03-25T18:31:22.125 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:31:22.123+0000 7f63f6158640 1 --1- >> v1:172.21.15.115:6789/0 conn(0x7f63f00b1b00 0x7f63f0140cd0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.115:6789/0 says I am v1:172.21.15.1:40780/0 (socket says 172.21.15.1:40780) 2024-03-25T18:31:22.125 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:31:22.123+0000 7f63f6158640 1 -- 172.21.15.1:0/546443042 learned_addr learned my addr 172.21.15.1:0/546443042 (peer_addr_for_me v1:172.21.15.1:0/0) 2024-03-25T18:31:22.126 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:31:22.125+0000 7f63e6ffd640 10 client.?.objecter ms_handle_connect 0x7f63f00b1b00 2024-03-25T18:31:22.126 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:31:22.125+0000 7f63e6ffd640 10 client.?.objecter resend_mon_ops 2024-03-25T18:31:22.135 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:31:22.133+0000 7f63e6ffd640 10 client.?.objecter ms_handle_connect 0x7f63f00b44b0 2024-03-25T18:31:22.135 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:31:22.133+0000 7f63e6ffd640 10 client.?.objecter resend_mon_ops 2024-03-25T18:31:22.136 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:31:22.134+0000 7f63e6ffd640 1 -- 172.21.15.1:0/546443042 <== mon.1 v1:172.21.15.115:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2988452990 0 0) 0x7f63f0142730 con 0x7f63f00b1b00 2024-03-25T18:31:22.136 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:31:22.134+0000 7f63e6ffd640 10 monclient(hunting): _negotiate 2024-03-25T18:31:22.136 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:31:22.134+0000 7f63e6ffd640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:31:22.136 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:31:22.134+0000 7f63e6ffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:31:22.136 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:31:22.134+0000 7f63e6ffd640 10 monclient(hunting): my global_id is 4429 2024-03-25T18:31:22.137 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:31:22.134+0000 7f63e6ffd640 1 -- 172.21.15.1:0/546443042 --> v1:172.21.15.115:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f63d4004eb0 con 0x7f63f00b1b00 2024-03-25T18:31:22.144 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:31:22.142+0000 7f63e6ffd640 10 client.?.objecter ms_handle_connect 0x7f63f00b4e70 2024-03-25T18:31:22.144 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:31:22.142+0000 7f63e6ffd640 10 client.?.objecter resend_mon_ops 2024-03-25T18:31:22.147 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:31:22.145+0000 7f63e6ffd640 1 -- 172.21.15.1:0/546443042 <== mon.2 v1:172.21.15.1:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 378870887 0 0) 0x7f63f0140160 con 0x7f63f00b44b0 2024-03-25T18:31:22.147 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:31:22.145+0000 7f63e6ffd640 10 monclient(hunting): _negotiate 2024-03-25T18:31:22.147 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:31:22.145+0000 7f63e6ffd640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:31:22.147 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:31:22.145+0000 7f63e6ffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:31:22.148 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:31:22.145+0000 7f63e6ffd640 10 monclient(hunting): my global_id is 4433 2024-03-25T18:31:22.148 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:31:22.145+0000 7f63e6ffd640 1 -- 172.21.15.1:0/546443042 --> v1:172.21.15.1:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f63f0142730 con 0x7f63f00b44b0 2024-03-25T18:31:22.158 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:31:22.156+0000 7f63e6ffd640 1 -- 172.21.15.1:0/546443042 <== mon.1 v1:172.21.15.115:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 173034262 0 0) 0x7f63d4004eb0 con 0x7f63f00b1b00 2024-03-25T18:31:22.158 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:31:22.156+0000 7f63e6ffd640 1 -- 172.21.15.1:0/546443042 --> v1:172.21.15.115:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f63f0140160 con 0x7f63f00b1b00 2024-03-25T18:31:22.158 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:31:22.156+0000 7f63e6ffd640 1 -- 172.21.15.1:0/546443042 <== mon.1 v1:172.21.15.115:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1586965025 0 0) 0x7f63ec001c00 con 0x7f63f00b1b00 2024-03-25T18:31:22.158 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:31:22.156+0000 7f63e6ffd640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:31:22.158 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:31:22.156+0000 7f63e6ffd640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-03-25T18:31:22.158 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:31:22.156+0000 7f63e6ffd640 10 monclient(hunting): dump: 2024-03-25T18:31:22.159 INFO:teuthology.misc.health.smithi001.stderr:epoch 1 2024-03-25T18:31:22.159 INFO:teuthology.misc.health.smithi001.stderr:fsid c8ea72cf-9e99-4383-9b52-6c23d365fdfc 2024-03-25T18:31:22.159 INFO:teuthology.misc.health.smithi001.stderr:last_changed 2024-03-25T18:25:50.510519+0000 2024-03-25T18:31:22.159 INFO:teuthology.misc.health.smithi001.stderr:created 2024-03-25T18:25:50.510519+0000 2024-03-25T18:31:22.159 INFO:teuthology.misc.health.smithi001.stderr:min_mon_release 19 (squid) 2024-03-25T18:31:22.159 INFO:teuthology.misc.health.smithi001.stderr:election_strategy: 3 2024-03-25T18:31:22.159 INFO:teuthology.misc.health.smithi001.stderr:0: v1:172.21.15.1:6789/0 mon.a 2024-03-25T18:31:22.159 INFO:teuthology.misc.health.smithi001.stderr:1: v1:172.21.15.115:6789/0 mon.b 2024-03-25T18:31:22.159 INFO:teuthology.misc.health.smithi001.stderr:2: v1:172.21.15.1:6790/0 mon.c 2024-03-25T18:31:22.159 INFO:teuthology.misc.health.smithi001.stderr: 2024-03-25T18:31:22.159 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:31:22.156+0000 7f63e6ffd640 10 monclient(hunting): _finish_auth 0 2024-03-25T18:31:22.161 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:31:22.159+0000 7f63e6ffd640 1 -- 172.21.15.1:0/546443042 <== mon.0 v1:172.21.15.1:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1204434895 0 0) 0x7f63f01d4540 con 0x7f63f00b4e70 2024-03-25T18:31:22.161 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:31:22.159+0000 7f63e6ffd640 10 monclient(hunting): _negotiate 2024-03-25T18:31:22.161 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:31:22.159+0000 7f63e6ffd640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:31:22.161 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:31:22.159+0000 7f63e6ffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:31:22.161 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:31:22.159+0000 7f63e6ffd640 10 monclient(hunting): my global_id is 4419 2024-03-25T18:31:22.161 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:31:22.159+0000 7f63e6ffd640 1 -- 172.21.15.1:0/546443042 --> v1:172.21.15.1:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f63d4004eb0 con 0x7f63f00b4e70 2024-03-25T18:31:22.172 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:31:22.170+0000 7f63e6ffd640 1 -- 172.21.15.1:0/546443042 <== mon.1 v1:172.21.15.115:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 2445298749 0 0) 0x7f63f0140160 con 0x7f63f00b1b00 2024-03-25T18:31:22.172 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:31:22.170+0000 7f63e6ffd640 1 -- 172.21.15.1:0/546443042 >> v1:172.21.15.1:6790/0 conn(0x7f63f00b44b0 legacy=0x7f63f01d49d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:31:22.172 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:31:22.170+0000 7f63e6ffd640 1 -- 172.21.15.1:0/546443042 >> v1:172.21.15.1:6789/0 conn(0x7f63f00b4e70 legacy=0x7f63f01d50e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:31:22.172 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:31:22.170+0000 7f63e6ffd640 10 monclient: _finish_hunting 0 2024-03-25T18:31:22.172 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:31:22.170+0000 7f63e6ffd640 1 monclient: found mon.b 2024-03-25T18:31:22.172 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:31:22.170+0000 7f63e6ffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:31:22.172 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:31:22.170+0000 7f63e6ffd640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.115:6789/0 2024-03-25T18:31:22.173 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:31:22.170+0000 7f63e6ffd640 1 -- 172.21.15.1:0/546443042 --> v1:172.21.15.115:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f63f01402f0 con 0x7f63f00b1b00 2024-03-25T18:31:22.173 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:31:22.170+0000 7f63e6ffd640 10 monclient: _finish_auth 0 2024-03-25T18:31:22.173 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:31:22.170+0000 7f63e6ffd640 10 monclient: _check_auth_tickets 2024-03-25T18:31:22.173 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:31:22.170+0000 7f63e6ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:31:22.173 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:31:22.170+0000 7f63f83bc640 5 monclient: authenticate success, global_id 4429 2024-03-25T18:31:22.173 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:31:22.170+0000 7f63f83bc640 10 monclient: _renew_subs 2024-03-25T18:31:22.173 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:31:22.170+0000 7f63f83bc640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.115:6789/0 2024-03-25T18:31:22.173 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:31:22.170+0000 7f63f83bc640 1 -- 172.21.15.1:0/546443042 --> v1:172.21.15.115:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f63f01413e0 con 0x7f63f00b1b00 2024-03-25T18:31:22.173 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:31:22.170+0000 7f63f83bc640 10 client.4429.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T18:31:22.173 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:31:22.170+0000 7f63f83bc640 10 monclient: _renew_subs 2024-03-25T18:31:22.173 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:31:22.170+0000 7f63f83bc640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.115:6789/0 2024-03-25T18:31:22.173 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:31:22.170+0000 7f63f83bc640 1 -- 172.21.15.1:0/546443042 --> v1:172.21.15.115:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f63f01e36e0 con 0x7f63f00b1b00 2024-03-25T18:31:22.173 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:31:22.170+0000 7f63f83bc640 1 librados: init done 2024-03-25T18:31:22.174 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:31:22.171+0000 7f63f83bc640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T18:31:22.174 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:31:22.171+0000 7f63f83bc640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:31:22.174 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:31:22.171+0000 7f63f83bc640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.115:6789/0 2024-03-25T18:31:22.174 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:31:22.171+0000 7f63f83bc640 1 -- 172.21.15.1:0/546443042 --> v1:172.21.15.115:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f63f004f730 con 0x7f63f00b1b00 2024-03-25T18:31:22.486 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:31:22.484+0000 7f63e6ffd640 1 -- 172.21.15.1:0/546443042 <== mon.1 v1:172.21.15.115:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f63ec004a90 con 0x7f63f00b1b00 2024-03-25T18:31:22.486 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:31:22.484+0000 7f63e6ffd640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:31:22.486 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:31:22.484+0000 7f63e6ffd640 1 -- 172.21.15.1:0/546443042 <== mon.1 v1:172.21.15.115:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1586965025 0 0) 0x7f63ec004da0 con 0x7f63f00b1b00 2024-03-25T18:31:22.487 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:31:22.484+0000 7f63e6ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:31:22.487 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:31:22.484+0000 7f63e6ffd640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-25T18:31:22.487 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:31:22.484+0000 7f63e6ffd640 10 monclient: dump: 2024-03-25T18:31:22.487 INFO:teuthology.misc.health.smithi001.stderr:epoch 1 2024-03-25T18:31:22.487 INFO:teuthology.misc.health.smithi001.stderr:fsid c8ea72cf-9e99-4383-9b52-6c23d365fdfc 2024-03-25T18:31:22.487 INFO:teuthology.misc.health.smithi001.stderr:last_changed 2024-03-25T18:25:50.510519+0000 2024-03-25T18:31:22.487 INFO:teuthology.misc.health.smithi001.stderr:created 2024-03-25T18:25:50.510519+0000 2024-03-25T18:31:22.487 INFO:teuthology.misc.health.smithi001.stderr:min_mon_release 19 (squid) 2024-03-25T18:31:22.487 INFO:teuthology.misc.health.smithi001.stderr:election_strategy: 3 2024-03-25T18:31:22.487 INFO:teuthology.misc.health.smithi001.stderr:0: v1:172.21.15.1:6789/0 mon.a 2024-03-25T18:31:22.487 INFO:teuthology.misc.health.smithi001.stderr:1: v1:172.21.15.115:6789/0 mon.b 2024-03-25T18:31:22.487 INFO:teuthology.misc.health.smithi001.stderr:2: v1:172.21.15.1:6790/0 mon.c 2024-03-25T18:31:22.487 INFO:teuthology.misc.health.smithi001.stderr: 2024-03-25T18:31:22.487 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:31:22.484+0000 7f63f4955640 4 set_mon_vals no callback set 2024-03-25T18:31:22.488 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:31:22.485+0000 7f63e6ffd640 1 -- 172.21.15.1:0/546443042 <== mon.1 v1:172.21.15.115:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 21787975 0 0) 0x7f63ec01ad70 con 0x7f63f00b1b00 2024-03-25T18:31:22.488 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:31:22.485+0000 7f63f5957640 10 monclient: get_auth_request con 0x7f63d406ddb0 auth_method 0 2024-03-25T18:31:22.488 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:31:22.486+0000 7f63e6ffd640 10 client.4429.objecter ms_handle_connect 0x7f63d406ddb0 2024-03-25T18:31:22.493 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:31:22.491+0000 7f63e6ffd640 1 -- 172.21.15.1:0/546443042 <== mon.1 v1:172.21.15.115:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 996254230 0 0) 0x7f63ec0548e0 con 0x7f63f00b1b00 2024-03-25T18:31:22.493 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:31:22.491+0000 7f63e6ffd640 10 client.4429.objecter ms_dispatch 0x7f63f0000b90 osd_map(10..10 src has 1..10) v4 2024-03-25T18:31:22.493 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:31:22.491+0000 7f63e6ffd640 3 client.4429.objecter handle_osd_map got epochs [10,10] > 0 2024-03-25T18:31:22.493 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:31:22.491+0000 7f63e6ffd640 3 client.4429.objecter handle_osd_map decoding full epoch 10 2024-03-25T18:31:22.493 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:31:22.491+0000 7f63e6ffd640 20 client.4429.objecter dump_active .. 0 homeless 2024-03-25T18:31:22.493 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:31:22.491+0000 7f63e6ffd640 1 -- 172.21.15.1:0/546443042 <== mon.1 v1:172.21.15.115:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f63ec0503f0 con 0x7f63f00b1b00 2024-03-25T18:31:22.493 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:31:22.491+0000 7f63e6ffd640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:31:22.493 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:31:22.491+0000 7f63e6ffd640 10 monclient: _finish_command 1 = system:0 2024-03-25T18:31:22.751 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:31:22.749+0000 7f63f83bc640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:31:22.751 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:31:22.749+0000 7f63f83bc640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:31:22.751 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:31:22.749+0000 7f63f83bc640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.115:6789/0 2024-03-25T18:31:22.751 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:31:22.749+0000 7f63f83bc640 1 -- 172.21.15.1:0/546443042 --> v1:172.21.15.115:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f63f00b23b0 con 0x7f63f00b1b00 2024-03-25T18:31:22.789 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:31:22.788+0000 7f63e6ffd640 1 -- 172.21.15.1:0/546443042 <== mon.1 v1:172.21.15.115:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 4092213348) 0x7f63ec052320 con 0x7f63f00b1b00 2024-03-25T18:31:22.789 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:31:22.788+0000 7f63e6ffd640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:31:22.790 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:31:22.788+0000 7f63e6ffd640 10 monclient: _finish_command 2 = system:0 2024-03-25T18:31:22.790 INFO:teuthology.misc.health.smithi001.stdout: 2024-03-25T18:31:22.790 INFO:teuthology.misc.health.smithi001.stdout:{"epoch":10,"fsid":"c8ea72cf-9e99-4383-9b52-6c23d365fdfc","created":"2024-03-25T18:26:38.856638+0000","modified":"2024-03-25T18:26:44.224208+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T18:26:44.224208+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"b9f34922-3a84-49be-8426-807c1eceaefc","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"42bf306f-55e3-4634-9874-8b4b73ac6644","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"292cb23a-18a9-4207-a7cc-4d5727feaec0","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"45e0e9ab-898d-4673-9590-b4117bc5e84a","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"831a5a09-73ab-4a61-9872-088e76bdbc66","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"88cd903d-3b55-4840-9396-6e3cfe005282","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"89db7355-bc9d-46f4-8ab0-e8d9e744e026","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"cac47bee-0914-4f08-896f-348838882866","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T18:31:22.792 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:31:22.790+0000 7f63f83bc640 10 librados: watch_flush enter 2024-03-25T18:31:22.792 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:31:22.790+0000 7f63f83bc640 10 librados: watch_flush exit 2024-03-25T18:31:22.792 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:31:22.790+0000 7f63f83bc640 20 client.4429.objecter shutdown clearing up homeless session... 2024-03-25T18:31:22.792 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:31:22.790+0000 7f63f83bc640 10 client.4429.objecter successfully canceled tick 2024-03-25T18:31:22.793 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:31:22.790+0000 7f63f83bc640 1 -- 172.21.15.1:0/546443042 >> v1:172.21.15.115:6800/4237379176 conn(0x7f63d406ddb0 legacy=0x7f63d4070270 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:31:22.793 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:31:22.790+0000 7f63f83bc640 10 monclient: shutdown 2024-03-25T18:31:22.793 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:31:22.790+0000 7f63f83bc640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:31:22.793 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:31:22.790+0000 7f63f83bc640 1 -- 172.21.15.1:0/546443042 >> v1:172.21.15.115:6789/0 conn(0x7f63f00b1b00 legacy=0x7f63f0140cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:31:22.793 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:31:22.790+0000 7f63f83bc640 1 -- 172.21.15.1:0/546443042 shutdown_connections 2024-03-25T18:31:22.793 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:31:22.790+0000 7f63f83bc640 1 -- 172.21.15.1:0/546443042 >> 172.21.15.1:0/546443042 conn(0x7f63f00617a0 msgr2=0x7f63f00b6590 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:31:22.793 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:31:22.790+0000 7f63f83bc640 1 -- 172.21.15.1:0/546443042 shutdown_connections 2024-03-25T18:31:22.793 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:31:22.790+0000 7f63f83bc640 1 -- 172.21.15.1:0/546443042 wait complete. 2024-03-25T18:31:22.793 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:31:22.790+0000 7f63f83bc640 1 librados: shutdown 2024-03-25T18:31:22.803 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T18:31:29.610 DEBUG:teuthology.orchestra.run.smithi001:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T18:31:29.730 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:31:29.728+0000 7f2b8cba3640 10 monclient: get_monmap_and_config 2024-03-25T18:31:29.730 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:31:29.728+0000 7f2b8cba3640 10 monclient: build_initial_monmap 2024-03-25T18:31:29.731 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:31:29.728+0000 7f2b8cba3640 10 monclient: monmap: 2024-03-25T18:31:29.731 INFO:teuthology.misc.health.smithi001.stderr:epoch 0 2024-03-25T18:31:29.731 INFO:teuthology.misc.health.smithi001.stderr:fsid c8ea72cf-9e99-4383-9b52-6c23d365fdfc 2024-03-25T18:31:29.731 INFO:teuthology.misc.health.smithi001.stderr:last_changed 2024-03-25T18:31:29.730385+0000 2024-03-25T18:31:29.731 INFO:teuthology.misc.health.smithi001.stderr:created 2024-03-25T18:31:29.730385+0000 2024-03-25T18:31:29.731 INFO:teuthology.misc.health.smithi001.stderr:min_mon_release 0 (unknown) 2024-03-25T18:31:29.731 INFO:teuthology.misc.health.smithi001.stderr:election_strategy: 3 2024-03-25T18:31:29.731 INFO:teuthology.misc.health.smithi001.stderr:0: v1:172.21.15.1:6789/0 mon.noname-a 2024-03-25T18:31:29.731 INFO:teuthology.misc.health.smithi001.stderr:1: v1:172.21.15.115:6789/0 mon.noname-c 2024-03-25T18:31:29.731 INFO:teuthology.misc.health.smithi001.stderr:2: v1:172.21.15.1:6790/0 mon.noname-b 2024-03-25T18:31:29.731 INFO:teuthology.misc.health.smithi001.stderr: 2024-03-25T18:31:29.733 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:31:29.729+0000 7f2b8cba3640 1 Processor -- start 2024-03-25T18:31:29.733 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:31:29.729+0000 7f2b8cba3640 1 -- start start 2024-03-25T18:31:29.733 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:31:29.729+0000 7f2b8cba3640 10 monclient: init 2024-03-25T18:31:29.733 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:31:29.730+0000 7f2b8cba3640 10 monclient: _reopen_session rank -1 2024-03-25T18:31:29.733 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:31:29.730+0000 7f2b8cba3640 10 monclient: _add_conns ranks=[1,0,2] 2024-03-25T18:31:29.733 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:31:29.730+0000 7f2b8cba3640 10 monclient(hunting): picked mon.noname-c con 0x7f2b880b4e70 addr v1:172.21.15.115:6789/0 2024-03-25T18:31:29.733 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:31:29.730+0000 7f2b8cba3640 10 monclient(hunting): picked mon.noname-a con 0x7f2b880b44b0 addr v1:172.21.15.1:6789/0 2024-03-25T18:31:29.734 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:31:29.730+0000 7f2b8cba3640 10 monclient(hunting): picked mon.noname-b con 0x7f2b880b1b00 addr v1:172.21.15.1:6790/0 2024-03-25T18:31:29.734 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:31:29.730+0000 7f2b8cba3640 1 -- --> v1:172.21.15.1:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2b880a96f0 con 0x7f2b880b44b0 2024-03-25T18:31:29.734 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:31:29.730+0000 7f2b8cba3640 1 -- --> v1:172.21.15.115:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2b880ab0e0 con 0x7f2b880b4e70 2024-03-25T18:31:29.734 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:31:29.730+0000 7f2b8cba3640 1 -- --> v1:172.21.15.1:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2b88145f70 con 0x7f2b880b1b00 2024-03-25T18:31:29.734 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:31:29.730+0000 7f2b8cba3640 10 monclient(hunting): _renew_subs 2024-03-25T18:31:29.734 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:31:29.730+0000 7f2b8cba3640 10 monclient(hunting): authenticate will time out at 7770.201514s 2024-03-25T18:31:29.734 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:31:29.732+0000 7f2b8659c640 1 --1- >> v1:172.21.15.115:6789/0 conn(0x7f2b880b4e70 0x7f2b8813f940 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.115:6789/0 says I am v1:172.21.15.1:35892/0 (socket says 172.21.15.1:35892) 2024-03-25T18:31:29.734 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:31:29.732+0000 7f2b8659c640 1 -- 172.21.15.1:0/2430602139 learned_addr learned my addr 172.21.15.1:0/2430602139 (peer_addr_for_me v1:172.21.15.1:0/0) 2024-03-25T18:31:29.809 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:31:29.807+0000 7f2b85d9b640 1 -- 172.21.15.1:0/2430602139 <== mon.1 v1:172.21.15.115:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 114092352 0 0) 0x7f2b880ab0e0 con 0x7f2b880b4e70 2024-03-25T18:31:29.809 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:31:29.807+0000 7f2b85d9b640 10 monclient(hunting): _negotiate 2024-03-25T18:31:29.809 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:31:29.807+0000 7f2b85d9b640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:31:29.809 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:31:29.807+0000 7f2b85d9b640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:31:29.809 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:31:29.807+0000 7f2b85d9b640 10 monclient(hunting): my global_id is 4432 2024-03-25T18:31:29.809 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:31:29.807+0000 7f2b85d9b640 1 -- 172.21.15.1:0/2430602139 --> v1:172.21.15.115:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f2b74003520 con 0x7f2b880b4e70 2024-03-25T18:31:29.809 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:31:29.807+0000 7f2b85d9b640 1 -- 172.21.15.1:0/2430602139 <== mon.0 v1:172.21.15.1:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3714217779 0 0) 0x7f2b880a96f0 con 0x7f2b880b44b0 2024-03-25T18:31:29.809 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:31:29.807+0000 7f2b85d9b640 10 monclient(hunting): _negotiate 2024-03-25T18:31:29.809 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:31:29.807+0000 7f2b85d9b640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:31:29.810 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:31:29.807+0000 7f2b85d9b640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:31:29.810 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:31:29.807+0000 7f2b85d9b640 10 monclient(hunting): my global_id is 4422 2024-03-25T18:31:29.810 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:31:29.807+0000 7f2b85d9b640 1 -- 172.21.15.1:0/2430602139 --> v1:172.21.15.1:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f2b880ab0e0 con 0x7f2b880b44b0 2024-03-25T18:31:29.814 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:31:29.812+0000 7f2b85d9b640 1 -- 172.21.15.1:0/2430602139 <== mon.2 v1:172.21.15.1:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2282309649 0 0) 0x7f2b88145f70 con 0x7f2b880b1b00 2024-03-25T18:31:29.814 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:31:29.812+0000 7f2b85d9b640 10 monclient(hunting): _negotiate 2024-03-25T18:31:29.814 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:31:29.812+0000 7f2b85d9b640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:31:29.814 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:31:29.812+0000 7f2b85d9b640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:31:29.814 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:31:29.812+0000 7f2b85d9b640 10 monclient(hunting): my global_id is 4436 2024-03-25T18:31:29.814 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:31:29.812+0000 7f2b85d9b640 1 -- 172.21.15.1:0/2430602139 --> v1:172.21.15.1:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f2b880a96f0 con 0x7f2b880b1b00 2024-03-25T18:31:29.824 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:31:29.822+0000 7f2b85d9b640 1 -- 172.21.15.1:0/2430602139 <== mon.0 v1:172.21.15.1:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1604359130 0 0) 0x7f2b880ab0e0 con 0x7f2b880b44b0 2024-03-25T18:31:29.824 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:31:29.822+0000 7f2b85d9b640 1 -- 172.21.15.1:0/2430602139 >> v1:172.21.15.1:6790/0 conn(0x7f2b880b1b00 legacy=0x7f2b8814dc00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:31:29.824 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:31:29.822+0000 7f2b85d9b640 1 -- 172.21.15.1:0/2430602139 >> v1:172.21.15.115:6789/0 conn(0x7f2b880b4e70 legacy=0x7f2b8813f940 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:31:29.824 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:31:29.822+0000 7f2b85d9b640 10 monclient: _finish_hunting 0 2024-03-25T18:31:29.824 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:31:29.822+0000 7f2b85d9b640 1 monclient: found mon.noname-a 2024-03-25T18:31:29.824 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:31:29.822+0000 7f2b85d9b640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:31:29.824 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:31:29.822+0000 7f2b85d9b640 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.1:6789/0 2024-03-25T18:31:29.824 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:31:29.822+0000 7f2b85d9b640 1 -- 172.21.15.1:0/2430602139 --> v1:172.21.15.1:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2b880a9b80 con 0x7f2b880b44b0 2024-03-25T18:31:29.824 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:31:29.822+0000 7f2b85d9b640 10 monclient: _finish_auth 0 2024-03-25T18:31:29.824 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:31:29.822+0000 7f2b85d9b640 10 monclient: _check_auth_tickets 2024-03-25T18:31:29.824 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:31:29.822+0000 7f2b85d9b640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:31:29.825 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:31:29.822+0000 7f2b85d9b640 1 -- 172.21.15.1:0/2430602139 <== mon.0 v1:172.21.15.1:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1586965025 0 0) 0x7f2b70001670 con 0x7f2b880b44b0 2024-03-25T18:31:29.825 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:31:29.822+0000 7f2b85d9b640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:31:29.825 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:31:29.822+0000 7f2b85d9b640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-03-25T18:31:29.825 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:31:29.822+0000 7f2b85d9b640 10 monclient: dump: 2024-03-25T18:31:29.825 INFO:teuthology.misc.health.smithi001.stderr:epoch 1 2024-03-25T18:31:29.825 INFO:teuthology.misc.health.smithi001.stderr:fsid c8ea72cf-9e99-4383-9b52-6c23d365fdfc 2024-03-25T18:31:29.825 INFO:teuthology.misc.health.smithi001.stderr:last_changed 2024-03-25T18:25:50.510519+0000 2024-03-25T18:31:29.825 INFO:teuthology.misc.health.smithi001.stderr:created 2024-03-25T18:25:50.510519+0000 2024-03-25T18:31:29.825 INFO:teuthology.misc.health.smithi001.stderr:min_mon_release 19 (squid) 2024-03-25T18:31:29.825 INFO:teuthology.misc.health.smithi001.stderr:election_strategy: 3 2024-03-25T18:31:29.825 INFO:teuthology.misc.health.smithi001.stderr:0: v1:172.21.15.1:6789/0 mon.a 2024-03-25T18:31:29.825 INFO:teuthology.misc.health.smithi001.stderr:1: v1:172.21.15.115:6789/0 mon.b 2024-03-25T18:31:29.825 INFO:teuthology.misc.health.smithi001.stderr:2: v1:172.21.15.1:6790/0 mon.c 2024-03-25T18:31:29.826 INFO:teuthology.misc.health.smithi001.stderr: 2024-03-25T18:31:29.826 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:31:29.822+0000 7f2b8cba3640 5 monclient: authenticate success, global_id 4422 2024-03-25T18:31:29.826 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:31:29.822+0000 7f2b8cba3640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T18:31:29.845 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:31:29.843+0000 7f2b85d9b640 1 -- 172.21.15.1:0/2430602139 <== mon.0 v1:172.21.15.1:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f2b700017d0 con 0x7f2b880b44b0 2024-03-25T18:31:29.845 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:31:29.843+0000 7f2b85d9b640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:31:29.845 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:31:29.843+0000 7f2b85d9b640 1 -- 172.21.15.1:0/2430602139 <== mon.0 v1:172.21.15.1:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1586965025 0 0) 0x7f2b70001be0 con 0x7f2b880b44b0 2024-03-25T18:31:29.846 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:31:29.843+0000 7f2b85d9b640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:31:29.846 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:31:29.843+0000 7f2b85d9b640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-25T18:31:29.846 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:31:29.843+0000 7f2b85d9b640 10 monclient: dump: 2024-03-25T18:31:29.846 INFO:teuthology.misc.health.smithi001.stderr:epoch 1 2024-03-25T18:31:29.846 INFO:teuthology.misc.health.smithi001.stderr:fsid c8ea72cf-9e99-4383-9b52-6c23d365fdfc 2024-03-25T18:31:29.846 INFO:teuthology.misc.health.smithi001.stderr:last_changed 2024-03-25T18:25:50.510519+0000 2024-03-25T18:31:29.846 INFO:teuthology.misc.health.smithi001.stderr:created 2024-03-25T18:25:50.510519+0000 2024-03-25T18:31:29.846 INFO:teuthology.misc.health.smithi001.stderr:min_mon_release 19 (squid) 2024-03-25T18:31:29.846 INFO:teuthology.misc.health.smithi001.stderr:election_strategy: 3 2024-03-25T18:31:29.846 INFO:teuthology.misc.health.smithi001.stderr:0: v1:172.21.15.1:6789/0 mon.a 2024-03-25T18:31:29.846 INFO:teuthology.misc.health.smithi001.stderr:1: v1:172.21.15.115:6789/0 mon.b 2024-03-25T18:31:29.847 INFO:teuthology.misc.health.smithi001.stderr:2: v1:172.21.15.1:6790/0 mon.c 2024-03-25T18:31:29.847 INFO:teuthology.misc.health.smithi001.stderr: 2024-03-25T18:31:29.847 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:31:29.843+0000 7f2b8cba3640 10 monclient: get_monmap_and_config success 2024-03-25T18:31:29.847 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:31:29.843+0000 7f2b8cba3640 4 set_mon_vals no callback set 2024-03-25T18:31:29.848 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:31:29.844+0000 7f2b8cba3640 10 monclient: shutdown 2024-03-25T18:31:29.848 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:31:29.844+0000 7f2b8cba3640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:31:29.848 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:31:29.844+0000 7f2b8cba3640 1 -- 172.21.15.1:0/2430602139 >> v1:172.21.15.1:6789/0 conn(0x7f2b880b44b0 legacy=0x7f2b88145860 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:31:29.848 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:31:29.844+0000 7f2b8cba3640 1 -- 172.21.15.1:0/2430602139 shutdown_connections 2024-03-25T18:31:29.849 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:31:29.844+0000 7f2b8cba3640 1 -- 172.21.15.1:0/2430602139 >> 172.21.15.1:0/2430602139 conn(0x7f2b880617a0 msgr2=0x7f2b88061bb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:31:29.849 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:31:29.845+0000 7f2b8cba3640 1 -- 172.21.15.1:0/2430602139 shutdown_connections 2024-03-25T18:31:29.849 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:31:29.845+0000 7f2b8cba3640 1 -- 172.21.15.1:0/2430602139 wait complete. 2024-03-25T18:31:29.849 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:31:29.846+0000 7f2b8cba3640 10 monclient: build_initial_monmap 2024-03-25T18:31:29.849 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:31:29.846+0000 7f2b8cba3640 10 monclient: monmap: 2024-03-25T18:31:29.849 INFO:teuthology.misc.health.smithi001.stderr:epoch 0 2024-03-25T18:31:29.849 INFO:teuthology.misc.health.smithi001.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T18:31:29.849 INFO:teuthology.misc.health.smithi001.stderr:last_changed 0.000000 2024-03-25T18:31:29.849 INFO:teuthology.misc.health.smithi001.stderr:created 0.000000 2024-03-25T18:31:29.849 INFO:teuthology.misc.health.smithi001.stderr:min_mon_release 0 (unknown) 2024-03-25T18:31:29.849 INFO:teuthology.misc.health.smithi001.stderr:election_strategy: 1 2024-03-25T18:31:29.849 INFO:teuthology.misc.health.smithi001.stderr:0: v1:172.21.15.1:6789/0 mon.noname-a 2024-03-25T18:31:29.850 INFO:teuthology.misc.health.smithi001.stderr:1: v1:172.21.15.115:6789/0 mon.noname-b 2024-03-25T18:31:29.850 INFO:teuthology.misc.health.smithi001.stderr:2: v1:172.21.15.1:6790/0 mon.noname-c 2024-03-25T18:31:29.850 INFO:teuthology.misc.health.smithi001.stderr: 2024-03-25T18:31:29.850 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:31:29.846+0000 7f2b8cba3640 1 librados: starting msgr at 2024-03-25T18:31:29.850 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:31:29.846+0000 7f2b8cba3640 1 librados: starting objecter 2024-03-25T18:31:29.850 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:31:29.846+0000 7f2b8cba3640 1 Processor -- start 2024-03-25T18:31:29.850 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:31:29.846+0000 7f2b8cba3640 1 -- start start 2024-03-25T18:31:29.850 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:31:29.846+0000 7f2b8cba3640 1 librados: setting wanted keys 2024-03-25T18:31:29.850 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:31:29.846+0000 7f2b8cba3640 1 librados: calling monclient init 2024-03-25T18:31:29.850 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:31:29.846+0000 7f2b8cba3640 10 monclient: init 2024-03-25T18:31:29.851 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:31:29.847+0000 7f2b8cba3640 10 monclient: _reopen_session rank -1 2024-03-25T18:31:29.851 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:31:29.847+0000 7f2b8cba3640 10 monclient: _add_conns ranks=[2,0,1] 2024-03-25T18:31:29.851 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:31:29.847+0000 7f2b8cba3640 10 monclient(hunting): picked mon.noname-c con 0x7f2b880b1b00 addr v1:172.21.15.1:6790/0 2024-03-25T18:31:29.852 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:31:29.847+0000 7f2b8cba3640 10 monclient(hunting): picked mon.noname-a con 0x7f2b880b44b0 addr v1:172.21.15.1:6789/0 2024-03-25T18:31:29.852 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:31:29.847+0000 7f2b8cba3640 10 monclient(hunting): picked mon.noname-b con 0x7f2b880b4e70 addr v1:172.21.15.115:6789/0 2024-03-25T18:31:29.852 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:31:29.847+0000 7f2b8cba3640 1 -- --> v1:172.21.15.1:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2b881430a0 con 0x7f2b880b44b0 2024-03-25T18:31:29.852 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:31:29.847+0000 7f2b8cba3640 1 -- --> v1:172.21.15.115:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2b88137ea0 con 0x7f2b880b4e70 2024-03-25T18:31:29.852 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:31:29.847+0000 7f2b8cba3640 1 -- --> v1:172.21.15.1:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2b881d7a90 con 0x7f2b880b1b00 2024-03-25T18:31:29.852 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:31:29.847+0000 7f2b8cba3640 10 monclient(hunting): _renew_subs 2024-03-25T18:31:29.852 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:31:29.847+0000 7f2b8cba3640 10 monclient(hunting): authenticate will time out at 7770.318572s 2024-03-25T18:31:29.862 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:31:29.860+0000 7f2b7ffff640 1 --1- >> v1:172.21.15.1:6789/0 conn(0x7f2b880b44b0 0x7f2b881d4b80 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.1:6789/0 says I am v1:172.21.15.1:38524/0 (socket says 172.21.15.1:38524) 2024-03-25T18:31:29.862 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:31:29.860+0000 7f2b7ffff640 1 -- 172.21.15.1:0/1789712887 learned_addr learned my addr 172.21.15.1:0/1789712887 (peer_addr_for_me v1:172.21.15.1:0/0) 2024-03-25T18:31:29.867 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:31:29.865+0000 7f2b7effd640 10 client.?.objecter ms_handle_connect 0x7f2b880b44b0 2024-03-25T18:31:29.867 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:31:29.865+0000 7f2b7effd640 10 client.?.objecter resend_mon_ops 2024-03-25T18:31:29.872 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:31:29.870+0000 7f2b7effd640 10 client.?.objecter ms_handle_connect 0x7f2b880b4e70 2024-03-25T18:31:29.872 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:31:29.870+0000 7f2b7effd640 10 client.?.objecter resend_mon_ops 2024-03-25T18:31:29.879 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:31:29.877+0000 7f2b7effd640 1 -- 172.21.15.1:0/1789712887 <== mon.0 v1:172.21.15.1:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1975720284 0 0) 0x7f2b881430a0 con 0x7f2b880b44b0 2024-03-25T18:31:29.879 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:31:29.877+0000 7f2b7effd640 10 monclient(hunting): _negotiate 2024-03-25T18:31:29.879 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:31:29.877+0000 7f2b7effd640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:31:29.879 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:31:29.877+0000 7f2b7effd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:31:29.879 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:31:29.877+0000 7f2b7effd640 10 monclient(hunting): my global_id is 4425 2024-03-25T18:31:29.879 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:31:29.877+0000 7f2b7effd640 1 -- 172.21.15.1:0/1789712887 --> v1:172.21.15.1:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f2b68004eb0 con 0x7f2b880b44b0 2024-03-25T18:31:29.880 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:31:29.878+0000 7f2b7effd640 10 client.?.objecter ms_handle_connect 0x7f2b880b1b00 2024-03-25T18:31:29.880 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:31:29.878+0000 7f2b7effd640 10 client.?.objecter resend_mon_ops 2024-03-25T18:31:29.891 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:31:29.889+0000 7f2b7effd640 1 -- 172.21.15.1:0/1789712887 <== mon.1 v1:172.21.15.115:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 252981368 0 0) 0x7f2b88137ea0 con 0x7f2b880b4e70 2024-03-25T18:31:29.891 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:31:29.889+0000 7f2b7effd640 10 monclient(hunting): _negotiate 2024-03-25T18:31:29.892 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:31:29.889+0000 7f2b7effd640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:31:29.892 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:31:29.889+0000 7f2b7effd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:31:29.892 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:31:29.889+0000 7f2b7effd640 10 monclient(hunting): my global_id is 4435 2024-03-25T18:31:29.892 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:31:29.890+0000 7f2b7effd640 1 -- 172.21.15.1:0/1789712887 --> v1:172.21.15.115:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f2b881430a0 con 0x7f2b880b4e70 2024-03-25T18:31:29.907 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:31:29.905+0000 7f2b7effd640 1 -- 172.21.15.1:0/1789712887 <== mon.1 v1:172.21.15.115:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2287238658 0 0) 0x7f2b881430a0 con 0x7f2b880b4e70 2024-03-25T18:31:29.907 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:31:29.905+0000 7f2b7effd640 1 -- 172.21.15.1:0/1789712887 --> v1:172.21.15.115:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f2b88137ea0 con 0x7f2b880b4e70 2024-03-25T18:31:29.907 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:31:29.905+0000 7f2b7effd640 1 -- 172.21.15.1:0/1789712887 <== mon.1 v1:172.21.15.115:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1586965025 0 0) 0x7f2b80001bf0 con 0x7f2b880b4e70 2024-03-25T18:31:29.908 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:31:29.905+0000 7f2b7effd640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:31:29.908 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:31:29.905+0000 7f2b7effd640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-03-25T18:31:29.908 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:31:29.905+0000 7f2b7effd640 10 monclient(hunting): dump: 2024-03-25T18:31:29.908 INFO:teuthology.misc.health.smithi001.stderr:epoch 1 2024-03-25T18:31:29.908 INFO:teuthology.misc.health.smithi001.stderr:fsid c8ea72cf-9e99-4383-9b52-6c23d365fdfc 2024-03-25T18:31:29.908 INFO:teuthology.misc.health.smithi001.stderr:last_changed 2024-03-25T18:25:50.510519+0000 2024-03-25T18:31:29.908 INFO:teuthology.misc.health.smithi001.stderr:created 2024-03-25T18:25:50.510519+0000 2024-03-25T18:31:29.908 INFO:teuthology.misc.health.smithi001.stderr:min_mon_release 19 (squid) 2024-03-25T18:31:29.908 INFO:teuthology.misc.health.smithi001.stderr:election_strategy: 3 2024-03-25T18:31:29.908 INFO:teuthology.misc.health.smithi001.stderr:0: v1:172.21.15.1:6789/0 mon.a 2024-03-25T18:31:29.908 INFO:teuthology.misc.health.smithi001.stderr:1: v1:172.21.15.115:6789/0 mon.b 2024-03-25T18:31:29.908 INFO:teuthology.misc.health.smithi001.stderr:2: v1:172.21.15.1:6790/0 mon.c 2024-03-25T18:31:29.908 INFO:teuthology.misc.health.smithi001.stderr: 2024-03-25T18:31:29.909 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:31:29.905+0000 7f2b7effd640 10 monclient(hunting): _finish_auth 0 2024-03-25T18:31:29.909 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:31:29.907+0000 7f2b7effd640 1 -- 172.21.15.1:0/1789712887 <== mon.2 v1:172.21.15.1:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3639182092 0 0) 0x7f2b881d7a90 con 0x7f2b880b1b00 2024-03-25T18:31:29.909 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:31:29.907+0000 7f2b7effd640 10 monclient(hunting): _negotiate 2024-03-25T18:31:29.909 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:31:29.907+0000 7f2b7effd640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:31:29.909 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:31:29.907+0000 7f2b7effd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:31:29.909 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:31:29.907+0000 7f2b7effd640 10 monclient(hunting): my global_id is 4439 2024-03-25T18:31:29.910 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:31:29.907+0000 7f2b7effd640 1 -- 172.21.15.1:0/1789712887 --> v1:172.21.15.1:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f2b881430a0 con 0x7f2b880b1b00 2024-03-25T18:31:29.913 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:31:29.911+0000 7f2b7effd640 1 -- 172.21.15.1:0/1789712887 <== mon.0 v1:172.21.15.1:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3066796108 0 0) 0x7f2b68004eb0 con 0x7f2b880b44b0 2024-03-25T18:31:29.913 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:31:29.911+0000 7f2b7effd640 1 -- 172.21.15.1:0/1789712887 --> v1:172.21.15.1:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f2b881d7a90 con 0x7f2b880b44b0 2024-03-25T18:31:29.913 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:31:29.911+0000 7f2b7effd640 1 -- 172.21.15.1:0/1789712887 <== mon.0 v1:172.21.15.1:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1586965025 0 0) 0x7f2b70001670 con 0x7f2b880b44b0 2024-03-25T18:31:29.913 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:31:29.911+0000 7f2b7effd640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:31:29.913 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:31:29.911+0000 7f2b7effd640 10 monclient(hunting): got monmap 1 from mon.a (according to old e1) 2024-03-25T18:31:29.913 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:31:29.911+0000 7f2b7effd640 10 monclient(hunting): dump: 2024-03-25T18:31:29.913 INFO:teuthology.misc.health.smithi001.stderr:epoch 1 2024-03-25T18:31:29.913 INFO:teuthology.misc.health.smithi001.stderr:fsid c8ea72cf-9e99-4383-9b52-6c23d365fdfc 2024-03-25T18:31:29.913 INFO:teuthology.misc.health.smithi001.stderr:last_changed 2024-03-25T18:25:50.510519+0000 2024-03-25T18:31:29.913 INFO:teuthology.misc.health.smithi001.stderr:created 2024-03-25T18:25:50.510519+0000 2024-03-25T18:31:29.914 INFO:teuthology.misc.health.smithi001.stderr:min_mon_release 19 (squid) 2024-03-25T18:31:29.914 INFO:teuthology.misc.health.smithi001.stderr:election_strategy: 3 2024-03-25T18:31:29.914 INFO:teuthology.misc.health.smithi001.stderr:0: v1:172.21.15.1:6789/0 mon.a 2024-03-25T18:31:29.914 INFO:teuthology.misc.health.smithi001.stderr:1: v1:172.21.15.115:6789/0 mon.b 2024-03-25T18:31:29.914 INFO:teuthology.misc.health.smithi001.stderr:2: v1:172.21.15.1:6790/0 mon.c 2024-03-25T18:31:29.914 INFO:teuthology.misc.health.smithi001.stderr: 2024-03-25T18:31:29.935 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:31:29.932+0000 7f2b7effd640 1 -- 172.21.15.1:0/1789712887 <== mon.1 v1:172.21.15.115:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 3680325102 0 0) 0x7f2b88137ea0 con 0x7f2b880b4e70 2024-03-25T18:31:29.935 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:31:29.932+0000 7f2b7effd640 1 -- 172.21.15.1:0/1789712887 >> v1:172.21.15.1:6790/0 conn(0x7f2b880b1b00 legacy=0x7f2b881d7fb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:31:29.935 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:31:29.932+0000 7f2b7effd640 1 -- 172.21.15.1:0/1789712887 >> v1:172.21.15.1:6789/0 conn(0x7f2b880b44b0 legacy=0x7f2b881d4b80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:31:29.935 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:31:29.932+0000 7f2b7effd640 10 monclient: _finish_hunting 0 2024-03-25T18:31:29.935 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:31:29.932+0000 7f2b7effd640 1 monclient: found mon.b 2024-03-25T18:31:29.935 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:31:29.932+0000 7f2b7effd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:31:29.935 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:31:29.932+0000 7f2b7effd640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.115:6789/0 2024-03-25T18:31:29.936 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:31:29.932+0000 7f2b7effd640 1 -- 172.21.15.1:0/1789712887 --> v1:172.21.15.115:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2b881d86c0 con 0x7f2b880b4e70 2024-03-25T18:31:29.936 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:31:29.932+0000 7f2b7effd640 10 monclient: _finish_auth 0 2024-03-25T18:31:29.936 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:31:29.932+0000 7f2b7effd640 10 monclient: _check_auth_tickets 2024-03-25T18:31:29.936 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:31:29.932+0000 7f2b7effd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:31:29.936 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:31:29.932+0000 7f2b8cba3640 5 monclient: authenticate success, global_id 4435 2024-03-25T18:31:29.936 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:31:29.932+0000 7f2b8cba3640 10 monclient: _renew_subs 2024-03-25T18:31:29.936 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:31:29.932+0000 7f2b8cba3640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.115:6789/0 2024-03-25T18:31:29.936 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:31:29.932+0000 7f2b8cba3640 1 -- 172.21.15.1:0/1789712887 --> v1:172.21.15.115:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2b881d4540 con 0x7f2b880b4e70 2024-03-25T18:31:29.936 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:31:29.932+0000 7f2b8cba3640 10 client.4435.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T18:31:29.936 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:31:29.932+0000 7f2b8cba3640 10 monclient: _renew_subs 2024-03-25T18:31:29.936 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:31:29.932+0000 7f2b8cba3640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.115:6789/0 2024-03-25T18:31:29.936 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:31:29.932+0000 7f2b8cba3640 1 -- 172.21.15.1:0/1789712887 --> v1:172.21.15.115:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f2b881d5470 con 0x7f2b880b4e70 2024-03-25T18:31:29.937 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:31:29.932+0000 7f2b8cba3640 1 librados: init done 2024-03-25T18:31:29.937 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:31:29.933+0000 7f2b8cba3640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T18:31:29.937 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:31:29.933+0000 7f2b8cba3640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:31:29.937 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:31:29.933+0000 7f2b8cba3640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.115:6789/0 2024-03-25T18:31:29.937 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:31:29.933+0000 7f2b8cba3640 1 -- 172.21.15.1:0/1789712887 --> v1:172.21.15.115:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f2b881d7c20 con 0x7f2b880b4e70 2024-03-25T18:31:30.205 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:31:30.203+0000 7f2b7effd640 1 -- 172.21.15.1:0/1789712887 <== mon.1 v1:172.21.15.115:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f2b80002080 con 0x7f2b880b4e70 2024-03-25T18:31:30.205 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:31:30.203+0000 7f2b7effd640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:31:30.206 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:31:30.203+0000 7f2b7effd640 1 -- 172.21.15.1:0/1789712887 <== mon.1 v1:172.21.15.115:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1586965025 0 0) 0x7f2b80003b30 con 0x7f2b880b4e70 2024-03-25T18:31:30.206 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:31:30.203+0000 7f2b7effd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:31:30.206 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:31:30.203+0000 7f2b7effd640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-25T18:31:30.206 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:31:30.203+0000 7f2b7effd640 10 monclient: dump: 2024-03-25T18:31:30.206 INFO:teuthology.misc.health.smithi001.stderr:epoch 1 2024-03-25T18:31:30.206 INFO:teuthology.misc.health.smithi001.stderr:fsid c8ea72cf-9e99-4383-9b52-6c23d365fdfc 2024-03-25T18:31:30.206 INFO:teuthology.misc.health.smithi001.stderr:last_changed 2024-03-25T18:25:50.510519+0000 2024-03-25T18:31:30.206 INFO:teuthology.misc.health.smithi001.stderr:created 2024-03-25T18:25:50.510519+0000 2024-03-25T18:31:30.206 INFO:teuthology.misc.health.smithi001.stderr:min_mon_release 19 (squid) 2024-03-25T18:31:30.206 INFO:teuthology.misc.health.smithi001.stderr:election_strategy: 3 2024-03-25T18:31:30.206 INFO:teuthology.misc.health.smithi001.stderr:0: v1:172.21.15.1:6789/0 mon.a 2024-03-25T18:31:30.206 INFO:teuthology.misc.health.smithi001.stderr:1: v1:172.21.15.115:6789/0 mon.b 2024-03-25T18:31:30.206 INFO:teuthology.misc.health.smithi001.stderr:2: v1:172.21.15.1:6790/0 mon.c 2024-03-25T18:31:30.207 INFO:teuthology.misc.health.smithi001.stderr: 2024-03-25T18:31:30.207 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:31:30.203+0000 7f2b8559a640 4 set_mon_vals no callback set 2024-03-25T18:31:30.207 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:31:30.204+0000 7f2b7effd640 1 -- 172.21.15.1:0/1789712887 <== mon.1 v1:172.21.15.115:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 21787975 0 0) 0x7f2b80003d10 con 0x7f2b880b4e70 2024-03-25T18:31:30.207 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:31:30.204+0000 7f2b8659c640 10 monclient: get_auth_request con 0x7f2b6806dfb0 auth_method 0 2024-03-25T18:31:30.207 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:31:30.205+0000 7f2b7effd640 10 client.4435.objecter ms_handle_connect 0x7f2b6806dfb0 2024-03-25T18:31:30.211 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:31:30.209+0000 7f2b7effd640 1 -- 172.21.15.1:0/1789712887 <== mon.1 v1:172.21.15.115:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 996254230 0 0) 0x7f2b80053870 con 0x7f2b880b4e70 2024-03-25T18:31:30.211 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:31:30.209+0000 7f2b7effd640 10 client.4435.objecter ms_dispatch 0x7f2b88000b90 osd_map(10..10 src has 1..10) v4 2024-03-25T18:31:30.211 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:31:30.209+0000 7f2b7effd640 3 client.4435.objecter handle_osd_map got epochs [10,10] > 0 2024-03-25T18:31:30.211 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:31:30.209+0000 7f2b7effd640 3 client.4435.objecter handle_osd_map decoding full epoch 10 2024-03-25T18:31:30.211 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:31:30.209+0000 7f2b7effd640 20 client.4435.objecter dump_active .. 0 homeless 2024-03-25T18:31:30.211 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:31:30.209+0000 7f2b7effd640 1 -- 172.21.15.1:0/1789712887 <== mon.1 v1:172.21.15.115:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f2b8004f1b0 con 0x7f2b880b4e70 2024-03-25T18:31:30.211 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:31:30.209+0000 7f2b7effd640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:31:30.212 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:31:30.209+0000 7f2b7effd640 10 monclient: _finish_command 1 = system:0 2024-03-25T18:31:30.480 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:31:30.478+0000 7f2b8cba3640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:31:30.480 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:31:30.478+0000 7f2b8cba3640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:31:30.480 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:31:30.478+0000 7f2b8cba3640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.115:6789/0 2024-03-25T18:31:30.480 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:31:30.478+0000 7f2b8cba3640 1 -- 172.21.15.1:0/1789712887 --> v1:172.21.15.115:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f2b881d8850 con 0x7f2b880b4e70 2024-03-25T18:31:30.519 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:31:30.517+0000 7f2b7effd640 1 -- 172.21.15.1:0/1789712887 <== mon.1 v1:172.21.15.115:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 4092213348) 0x7f2b800512b0 con 0x7f2b880b4e70 2024-03-25T18:31:30.519 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:31:30.517+0000 7f2b7effd640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:31:30.519 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:31:30.517+0000 7f2b7effd640 10 monclient: _finish_command 2 = system:0 2024-03-25T18:31:30.519 INFO:teuthology.misc.health.smithi001.stdout: 2024-03-25T18:31:30.519 INFO:teuthology.misc.health.smithi001.stdout:{"epoch":10,"fsid":"c8ea72cf-9e99-4383-9b52-6c23d365fdfc","created":"2024-03-25T18:26:38.856638+0000","modified":"2024-03-25T18:26:44.224208+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T18:26:44.224208+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"b9f34922-3a84-49be-8426-807c1eceaefc","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"42bf306f-55e3-4634-9874-8b4b73ac6644","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"292cb23a-18a9-4207-a7cc-4d5727feaec0","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"45e0e9ab-898d-4673-9590-b4117bc5e84a","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"831a5a09-73ab-4a61-9872-088e76bdbc66","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"88cd903d-3b55-4840-9396-6e3cfe005282","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"89db7355-bc9d-46f4-8ab0-e8d9e744e026","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"cac47bee-0914-4f08-896f-348838882866","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T18:31:30.522 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:31:30.519+0000 7f2b8cba3640 10 librados: watch_flush enter 2024-03-25T18:31:30.522 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:31:30.519+0000 7f2b8cba3640 10 librados: watch_flush exit 2024-03-25T18:31:30.522 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:31:30.519+0000 7f2b8cba3640 20 client.4435.objecter shutdown clearing up homeless session... 2024-03-25T18:31:30.522 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:31:30.519+0000 7f2b8cba3640 10 client.4435.objecter successfully canceled tick 2024-03-25T18:31:30.522 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:31:30.519+0000 7f2b8cba3640 1 -- 172.21.15.1:0/1789712887 >> v1:172.21.15.115:6800/4237379176 conn(0x7f2b6806dfb0 legacy=0x7f2b68070470 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:31:30.522 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:31:30.519+0000 7f2b8cba3640 10 monclient: shutdown 2024-03-25T18:31:30.522 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:31:30.519+0000 7f2b8cba3640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:31:30.522 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:31:30.519+0000 7f2b8cba3640 1 -- 172.21.15.1:0/1789712887 >> v1:172.21.15.115:6789/0 conn(0x7f2b880b4e70 legacy=0x7f2b88136420 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:31:30.522 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:31:30.520+0000 7f2b8cba3640 1 -- 172.21.15.1:0/1789712887 shutdown_connections 2024-03-25T18:31:30.522 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:31:30.520+0000 7f2b8cba3640 1 -- 172.21.15.1:0/1789712887 >> 172.21.15.1:0/1789712887 conn(0x7f2b880617a0 msgr2=0x7f2b88061b80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:31:30.522 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:31:30.520+0000 7f2b8cba3640 1 -- 172.21.15.1:0/1789712887 shutdown_connections 2024-03-25T18:31:30.522 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:31:30.520+0000 7f2b8cba3640 1 -- 172.21.15.1:0/1789712887 wait complete. 2024-03-25T18:31:30.522 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:31:30.520+0000 7f2b8cba3640 1 librados: shutdown 2024-03-25T18:31:30.533 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T18:31:36.877 INFO:tasks.ceph.mon.a.smithi001.stderr:==00:00:05:15.435 34876== Warning: unimplemented fcntl command: 1036 2024-03-25T18:31:36.881 INFO:tasks.ceph.mon.a.smithi001.stderr:==00:00:05:15.439 34876== Warning: unimplemented fcntl command: 1036 2024-03-25T18:31:36.912 INFO:tasks.ceph.mon.a.smithi001.stderr:==00:00:05:15.470 34876== Warning: unimplemented fcntl command: 1036 2024-03-25T18:31:36.927 INFO:tasks.ceph.mon.b.smithi115.stderr:==00:00:05:15.411 34836== Warning: unimplemented fcntl command: 1036 2024-03-25T18:31:36.929 INFO:tasks.ceph.mon.c.smithi001.stderr:==00:00:05:15.488 34877== Warning: unimplemented fcntl command: 1036 2024-03-25T18:31:36.932 INFO:tasks.ceph.mon.b.smithi115.stderr:==00:00:05:15.415 34836== Warning: unimplemented fcntl command: 1036 2024-03-25T18:31:36.944 INFO:tasks.ceph.mon.c.smithi001.stderr:==00:00:05:15.502 34877== Warning: unimplemented fcntl command: 1036 2024-03-25T18:31:36.972 INFO:tasks.ceph.mon.b.smithi115.stderr:==00:00:05:15.456 34836== Warning: unimplemented fcntl command: 1036 2024-03-25T18:31:36.978 INFO:tasks.ceph.mon.c.smithi001.stderr:==00:00:05:15.536 34877== Warning: unimplemented fcntl command: 1036 2024-03-25T18:31:37.339 DEBUG:teuthology.orchestra.run.smithi001:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T18:31:37.456 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:31:37.453+0000 7ff2bf4c0640 10 monclient: get_monmap_and_config 2024-03-25T18:31:37.456 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:31:37.453+0000 7ff2bf4c0640 10 monclient: build_initial_monmap 2024-03-25T18:31:37.456 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:31:37.453+0000 7ff2bf4c0640 10 monclient: monmap: 2024-03-25T18:31:37.456 INFO:teuthology.misc.health.smithi001.stderr:epoch 0 2024-03-25T18:31:37.456 INFO:teuthology.misc.health.smithi001.stderr:fsid c8ea72cf-9e99-4383-9b52-6c23d365fdfc 2024-03-25T18:31:37.456 INFO:teuthology.misc.health.smithi001.stderr:last_changed 2024-03-25T18:31:37.455523+0000 2024-03-25T18:31:37.456 INFO:teuthology.misc.health.smithi001.stderr:created 2024-03-25T18:31:37.455523+0000 2024-03-25T18:31:37.456 INFO:teuthology.misc.health.smithi001.stderr:min_mon_release 0 (unknown) 2024-03-25T18:31:37.456 INFO:teuthology.misc.health.smithi001.stderr:election_strategy: 3 2024-03-25T18:31:37.456 INFO:teuthology.misc.health.smithi001.stderr:0: v1:172.21.15.1:6789/0 mon.noname-a 2024-03-25T18:31:37.457 INFO:teuthology.misc.health.smithi001.stderr:1: v1:172.21.15.115:6789/0 mon.noname-c 2024-03-25T18:31:37.457 INFO:teuthology.misc.health.smithi001.stderr:2: v1:172.21.15.1:6790/0 mon.noname-b 2024-03-25T18:31:37.457 INFO:teuthology.misc.health.smithi001.stderr: 2024-03-25T18:31:37.457 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:31:37.454+0000 7ff2bf4c0640 1 Processor -- start 2024-03-25T18:31:37.458 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:31:37.454+0000 7ff2bf4c0640 1 -- start start 2024-03-25T18:31:37.458 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:31:37.454+0000 7ff2bf4c0640 10 monclient: init 2024-03-25T18:31:37.458 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:31:37.455+0000 7ff2bf4c0640 10 monclient: _reopen_session rank -1 2024-03-25T18:31:37.458 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:31:37.455+0000 7ff2bf4c0640 10 monclient: _add_conns ranks=[0,1,2] 2024-03-25T18:31:37.458 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:31:37.455+0000 7ff2bf4c0640 10 monclient(hunting): picked mon.noname-a con 0x7ff2b80b4ff0 addr v1:172.21.15.1:6789/0 2024-03-25T18:31:37.459 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:31:37.455+0000 7ff2bf4c0640 10 monclient(hunting): picked mon.noname-c con 0x7ff2b80b4670 addr v1:172.21.15.115:6789/0 2024-03-25T18:31:37.459 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:31:37.455+0000 7ff2bf4c0640 10 monclient(hunting): picked mon.noname-b con 0x7ff2b813c480 addr v1:172.21.15.1:6790/0 2024-03-25T18:31:37.459 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:31:37.455+0000 7ff2bf4c0640 1 -- --> v1:172.21.15.1:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff2b80ab0e0 con 0x7ff2b80b4ff0 2024-03-25T18:31:37.459 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:31:37.456+0000 7ff2bf4c0640 1 -- --> v1:172.21.15.115:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff2b80ab3c0 con 0x7ff2b80b4670 2024-03-25T18:31:37.459 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:31:37.456+0000 7ff2bf4c0640 1 -- --> v1:172.21.15.1:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff2b80a9f30 con 0x7ff2b813c480 2024-03-25T18:31:37.459 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:31:37.456+0000 7ff2bf4c0640 10 monclient(hunting): _renew_subs 2024-03-25T18:31:37.459 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:31:37.456+0000 7ff2bf4c0640 10 monclient(hunting): authenticate will time out at 7777.926745s 2024-03-25T18:31:37.462 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:31:37.460+0000 7ff2bd25c640 1 --1- >> v1:172.21.15.1:6789/0 conn(0x7ff2b80b4ff0 0x7ff2b813bd70 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.1:6789/0 says I am v1:172.21.15.1:37922/0 (socket says 172.21.15.1:37922) 2024-03-25T18:31:37.462 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:31:37.460+0000 7ff2bd25c640 1 -- 172.21.15.1:0/1600460048 learned_addr learned my addr 172.21.15.1:0/1600460048 (peer_addr_for_me v1:172.21.15.1:0/0) 2024-03-25T18:31:37.473 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:31:37.471+0000 7ff2affff640 1 -- 172.21.15.1:0/1600460048 <== mon.0 v1:172.21.15.1:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2483693629 0 0) 0x7ff2b80ab0e0 con 0x7ff2b80b4ff0 2024-03-25T18:31:37.473 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:31:37.472+0000 7ff2affff640 10 monclient(hunting): _negotiate 2024-03-25T18:31:37.473 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:31:37.472+0000 7ff2affff640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:31:37.473 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:31:37.472+0000 7ff2affff640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:31:37.473 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:31:37.472+0000 7ff2affff640 10 monclient(hunting): my global_id is 4428 2024-03-25T18:31:37.474 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:31:37.472+0000 7ff2affff640 1 -- 172.21.15.1:0/1600460048 --> v1:172.21.15.1:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ff2a4003520 con 0x7ff2b80b4ff0 2024-03-25T18:31:37.475 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:31:37.473+0000 7ff2affff640 1 -- 172.21.15.1:0/1600460048 <== mon.1 v1:172.21.15.115:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 115943356 0 0) 0x7ff2b80ab3c0 con 0x7ff2b80b4670 2024-03-25T18:31:37.475 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:31:37.473+0000 7ff2affff640 10 monclient(hunting): _negotiate 2024-03-25T18:31:37.475 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:31:37.473+0000 7ff2affff640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:31:37.475 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:31:37.473+0000 7ff2affff640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:31:37.475 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:31:37.473+0000 7ff2affff640 10 monclient(hunting): my global_id is 4438 2024-03-25T18:31:37.475 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:31:37.473+0000 7ff2affff640 1 -- 172.21.15.1:0/1600460048 --> v1:172.21.15.115:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ff2b80ab0e0 con 0x7ff2b80b4670 2024-03-25T18:31:37.480 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:31:37.478+0000 7ff2affff640 1 -- 172.21.15.1:0/1600460048 <== mon.0 v1:172.21.15.1:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1451013874 0 0) 0x7ff2a4003520 con 0x7ff2b80b4ff0 2024-03-25T18:31:37.480 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:31:37.478+0000 7ff2affff640 1 -- 172.21.15.1:0/1600460048 >> v1:172.21.15.1:6790/0 conn(0x7ff2b813c480 legacy=0x7ff2b814dc00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:31:37.480 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:31:37.478+0000 7ff2affff640 1 -- 172.21.15.1:0/1600460048 >> v1:172.21.15.115:6789/0 conn(0x7ff2b80b4670 legacy=0x7ff2b81425c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:31:37.480 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:31:37.478+0000 7ff2affff640 10 monclient: _finish_hunting 0 2024-03-25T18:31:37.480 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:31:37.478+0000 7ff2affff640 1 monclient: found mon.noname-a 2024-03-25T18:31:37.481 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:31:37.478+0000 7ff2affff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:31:37.481 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:31:37.478+0000 7ff2affff640 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.1:6789/0 2024-03-25T18:31:37.481 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:31:37.478+0000 7ff2affff640 1 -- 172.21.15.1:0/1600460048 --> v1:172.21.15.1:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff2b80a9b70 con 0x7ff2b80b4ff0 2024-03-25T18:31:37.481 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:31:37.478+0000 7ff2affff640 10 monclient: _finish_auth 0 2024-03-25T18:31:37.481 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:31:37.478+0000 7ff2affff640 10 monclient: _check_auth_tickets 2024-03-25T18:31:37.481 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:31:37.478+0000 7ff2affff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:31:37.481 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:31:37.478+0000 7ff2affff640 1 -- 172.21.15.1:0/1600460048 <== mon.0 v1:172.21.15.1:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1586965025 0 0) 0x7ff2b4001690 con 0x7ff2b80b4ff0 2024-03-25T18:31:37.481 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:31:37.478+0000 7ff2bf4c0640 5 monclient: authenticate success, global_id 4428 2024-03-25T18:31:37.481 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:31:37.478+0000 7ff2bf4c0640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T18:31:37.481 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:31:37.478+0000 7ff2affff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:31:37.481 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:31:37.478+0000 7ff2affff640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-03-25T18:31:37.481 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:31:37.478+0000 7ff2affff640 10 monclient: dump: 2024-03-25T18:31:37.481 INFO:teuthology.misc.health.smithi001.stderr:epoch 1 2024-03-25T18:31:37.481 INFO:teuthology.misc.health.smithi001.stderr:fsid c8ea72cf-9e99-4383-9b52-6c23d365fdfc 2024-03-25T18:31:37.481 INFO:teuthology.misc.health.smithi001.stderr:last_changed 2024-03-25T18:25:50.510519+0000 2024-03-25T18:31:37.481 INFO:teuthology.misc.health.smithi001.stderr:created 2024-03-25T18:25:50.510519+0000 2024-03-25T18:31:37.481 INFO:teuthology.misc.health.smithi001.stderr:min_mon_release 19 (squid) 2024-03-25T18:31:37.482 INFO:teuthology.misc.health.smithi001.stderr:election_strategy: 3 2024-03-25T18:31:37.482 INFO:teuthology.misc.health.smithi001.stderr:0: v1:172.21.15.1:6789/0 mon.a 2024-03-25T18:31:37.482 INFO:teuthology.misc.health.smithi001.stderr:1: v1:172.21.15.115:6789/0 mon.b 2024-03-25T18:31:37.482 INFO:teuthology.misc.health.smithi001.stderr:2: v1:172.21.15.1:6790/0 mon.c 2024-03-25T18:31:37.482 INFO:teuthology.misc.health.smithi001.stderr: 2024-03-25T18:31:37.482 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:31:37.478+0000 7ff2bf4c0640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T18:31:37.487 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:31:37.485+0000 7ff2affff640 1 -- 172.21.15.1:0/1600460048 <== mon.0 v1:172.21.15.1:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7ff2b40017f0 con 0x7ff2b80b4ff0 2024-03-25T18:31:37.487 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:31:37.485+0000 7ff2affff640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:31:37.487 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:31:37.485+0000 7ff2affff640 1 -- 172.21.15.1:0/1600460048 <== mon.0 v1:172.21.15.1:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1586965025 0 0) 0x7ff2b4001c00 con 0x7ff2b80b4ff0 2024-03-25T18:31:37.487 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:31:37.485+0000 7ff2bf4c0640 10 monclient: get_monmap_and_config success 2024-03-25T18:31:37.487 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:31:37.485+0000 7ff2bf4c0640 4 set_mon_vals no callback set 2024-03-25T18:31:37.487 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:31:37.485+0000 7ff2affff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:31:37.488 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:31:37.485+0000 7ff2affff640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-25T18:31:37.488 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:31:37.485+0000 7ff2affff640 10 monclient: dump: 2024-03-25T18:31:37.488 INFO:teuthology.misc.health.smithi001.stderr:epoch 1 2024-03-25T18:31:37.488 INFO:teuthology.misc.health.smithi001.stderr:fsid c8ea72cf-9e99-4383-9b52-6c23d365fdfc 2024-03-25T18:31:37.488 INFO:teuthology.misc.health.smithi001.stderr:last_changed 2024-03-25T18:25:50.510519+0000 2024-03-25T18:31:37.488 INFO:teuthology.misc.health.smithi001.stderr:created 2024-03-25T18:25:50.510519+0000 2024-03-25T18:31:37.488 INFO:teuthology.misc.health.smithi001.stderr:min_mon_release 19 (squid) 2024-03-25T18:31:37.488 INFO:teuthology.misc.health.smithi001.stderr:election_strategy: 3 2024-03-25T18:31:37.488 INFO:teuthology.misc.health.smithi001.stderr:0: v1:172.21.15.1:6789/0 mon.a 2024-03-25T18:31:37.488 INFO:teuthology.misc.health.smithi001.stderr:1: v1:172.21.15.115:6789/0 mon.b 2024-03-25T18:31:37.488 INFO:teuthology.misc.health.smithi001.stderr:2: v1:172.21.15.1:6790/0 mon.c 2024-03-25T18:31:37.488 INFO:teuthology.misc.health.smithi001.stderr: 2024-03-25T18:31:37.490 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:31:37.486+0000 7ff2bf4c0640 10 monclient: shutdown 2024-03-25T18:31:37.491 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:31:37.486+0000 7ff2bf4c0640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:31:37.491 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:31:37.486+0000 7ff2bf4c0640 1 -- 172.21.15.1:0/1600460048 >> v1:172.21.15.1:6789/0 conn(0x7ff2b80b4ff0 legacy=0x7ff2b813bd70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:31:37.491 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:31:37.486+0000 7ff2bf4c0640 1 -- 172.21.15.1:0/1600460048 shutdown_connections 2024-03-25T18:31:37.491 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:31:37.486+0000 7ff2bf4c0640 1 -- 172.21.15.1:0/1600460048 >> 172.21.15.1:0/1600460048 conn(0x7ff2b80617a0 msgr2=0x7ff2b8061bb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:31:37.491 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:31:37.486+0000 7ff2bf4c0640 1 -- 172.21.15.1:0/1600460048 shutdown_connections 2024-03-25T18:31:37.491 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:31:37.486+0000 7ff2bf4c0640 1 -- 172.21.15.1:0/1600460048 wait complete. 2024-03-25T18:31:37.491 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:31:37.487+0000 7ff2bf4c0640 10 monclient: build_initial_monmap 2024-03-25T18:31:37.491 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:31:37.487+0000 7ff2bf4c0640 10 monclient: monmap: 2024-03-25T18:31:37.491 INFO:teuthology.misc.health.smithi001.stderr:epoch 0 2024-03-25T18:31:37.491 INFO:teuthology.misc.health.smithi001.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T18:31:37.491 INFO:teuthology.misc.health.smithi001.stderr:last_changed 0.000000 2024-03-25T18:31:37.491 INFO:teuthology.misc.health.smithi001.stderr:created 0.000000 2024-03-25T18:31:37.491 INFO:teuthology.misc.health.smithi001.stderr:min_mon_release 0 (unknown) 2024-03-25T18:31:37.491 INFO:teuthology.misc.health.smithi001.stderr:election_strategy: 1 2024-03-25T18:31:37.492 INFO:teuthology.misc.health.smithi001.stderr:0: v1:172.21.15.1:6789/0 mon.noname-a 2024-03-25T18:31:37.492 INFO:teuthology.misc.health.smithi001.stderr:1: v1:172.21.15.115:6789/0 mon.noname-b 2024-03-25T18:31:37.492 INFO:teuthology.misc.health.smithi001.stderr:2: v1:172.21.15.1:6790/0 mon.noname-c 2024-03-25T18:31:37.492 INFO:teuthology.misc.health.smithi001.stderr: 2024-03-25T18:31:37.492 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:31:37.487+0000 7ff2bf4c0640 1 librados: starting msgr at 2024-03-25T18:31:37.492 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:31:37.487+0000 7ff2bf4c0640 1 librados: starting objecter 2024-03-25T18:31:37.492 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:31:37.487+0000 7ff2bf4c0640 1 Processor -- start 2024-03-25T18:31:37.492 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:31:37.487+0000 7ff2bf4c0640 1 -- start start 2024-03-25T18:31:37.492 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:31:37.487+0000 7ff2bf4c0640 1 librados: setting wanted keys 2024-03-25T18:31:37.492 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:31:37.487+0000 7ff2bf4c0640 1 librados: calling monclient init 2024-03-25T18:31:37.492 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:31:37.487+0000 7ff2bf4c0640 10 monclient: init 2024-03-25T18:31:37.492 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:31:37.488+0000 7ff2bf4c0640 10 monclient: _reopen_session rank -1 2024-03-25T18:31:37.492 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:31:37.488+0000 7ff2bf4c0640 10 monclient: _add_conns ranks=[2,1,0] 2024-03-25T18:31:37.492 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:31:37.488+0000 7ff2bf4c0640 10 monclient(hunting): picked mon.noname-c con 0x7ff2b80b4670 addr v1:172.21.15.1:6790/0 2024-03-25T18:31:37.492 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:31:37.488+0000 7ff2bf4c0640 10 monclient(hunting): picked mon.noname-b con 0x7ff2b80b4ff0 addr v1:172.21.15.115:6789/0 2024-03-25T18:31:37.493 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:31:37.488+0000 7ff2bf4c0640 10 monclient(hunting): picked mon.noname-a con 0x7ff2b813c480 addr v1:172.21.15.1:6789/0 2024-03-25T18:31:37.493 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:31:37.488+0000 7ff2bf4c0640 1 -- --> v1:172.21.15.1:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff2b8135ea0 con 0x7ff2b813c480 2024-03-25T18:31:37.493 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:31:37.488+0000 7ff2bf4c0640 1 -- --> v1:172.21.15.115:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff2b80ac3d0 con 0x7ff2b80b4ff0 2024-03-25T18:31:37.493 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:31:37.488+0000 7ff2bf4c0640 1 -- --> v1:172.21.15.1:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff2b80ac6b0 con 0x7ff2b80b4670 2024-03-25T18:31:37.493 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:31:37.488+0000 7ff2bf4c0640 10 monclient(hunting): _renew_subs 2024-03-25T18:31:37.493 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:31:37.488+0000 7ff2bf4c0640 10 monclient(hunting): authenticate will time out at 7777.959248s 2024-03-25T18:31:37.493 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:31:37.490+0000 7ff2bca5b640 1 --1- >> v1:172.21.15.115:6789/0 conn(0x7ff2b80b4ff0 0x7ff2b80b0e40 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.115:6789/0 says I am v1:172.21.15.1:60134/0 (socket says 172.21.15.1:60134) 2024-03-25T18:31:37.493 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:31:37.490+0000 7ff2bca5b640 1 -- 172.21.15.1:0/1397676503 learned_addr learned my addr 172.21.15.1:0/1397676503 (peer_addr_for_me v1:172.21.15.1:0/0) 2024-03-25T18:31:37.494 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:31:37.491+0000 7ff2adffb640 10 client.?.objecter ms_handle_connect 0x7ff2b80b4ff0 2024-03-25T18:31:37.494 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:31:37.491+0000 7ff2adffb640 10 client.?.objecter resend_mon_ops 2024-03-25T18:31:37.499 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:31:37.497+0000 7ff2adffb640 1 -- 172.21.15.1:0/1397676503 <== mon.1 v1:172.21.15.115:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3217679045 0 0) 0x7ff2b80ac3d0 con 0x7ff2b80b4ff0 2024-03-25T18:31:37.499 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:31:37.497+0000 7ff2adffb640 10 monclient(hunting): _negotiate 2024-03-25T18:31:37.499 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:31:37.497+0000 7ff2adffb640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:31:37.499 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:31:37.497+0000 7ff2adffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:31:37.499 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:31:37.497+0000 7ff2adffb640 10 monclient(hunting): my global_id is 4441 2024-03-25T18:31:37.499 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:31:37.497+0000 7ff2adffb640 1 -- 172.21.15.1:0/1397676503 --> v1:172.21.15.115:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ff29c004eb0 con 0x7ff2b80b4ff0 2024-03-25T18:31:37.501 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:31:37.500+0000 7ff2adffb640 10 client.?.objecter ms_handle_connect 0x7ff2b813c480 2024-03-25T18:31:37.502 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:31:37.500+0000 7ff2adffb640 10 client.?.objecter resend_mon_ops 2024-03-25T18:31:37.508 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:31:37.506+0000 7ff2adffb640 1 -- 172.21.15.1:0/1397676503 <== mon.1 v1:172.21.15.115:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1863784276 0 0) 0x7ff29c004eb0 con 0x7ff2b80b4ff0 2024-03-25T18:31:37.508 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:31:37.506+0000 7ff2adffb640 1 -- 172.21.15.1:0/1397676503 --> v1:172.21.15.115:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7ff2b80ac3d0 con 0x7ff2b80b4ff0 2024-03-25T18:31:37.508 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:31:37.506+0000 7ff2adffb640 1 -- 172.21.15.1:0/1397676503 <== mon.1 v1:172.21.15.115:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1586965025 0 0) 0x7ff2a8001a10 con 0x7ff2b80b4ff0 2024-03-25T18:31:37.508 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:31:37.506+0000 7ff2adffb640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:31:37.508 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:31:37.506+0000 7ff2adffb640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-03-25T18:31:37.508 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:31:37.506+0000 7ff2adffb640 10 monclient(hunting): dump: 2024-03-25T18:31:37.508 INFO:teuthology.misc.health.smithi001.stderr:epoch 1 2024-03-25T18:31:37.508 INFO:teuthology.misc.health.smithi001.stderr:fsid c8ea72cf-9e99-4383-9b52-6c23d365fdfc 2024-03-25T18:31:37.508 INFO:teuthology.misc.health.smithi001.stderr:last_changed 2024-03-25T18:25:50.510519+0000 2024-03-25T18:31:37.508 INFO:teuthology.misc.health.smithi001.stderr:created 2024-03-25T18:25:50.510519+0000 2024-03-25T18:31:37.508 INFO:teuthology.misc.health.smithi001.stderr:min_mon_release 19 (squid) 2024-03-25T18:31:37.508 INFO:teuthology.misc.health.smithi001.stderr:election_strategy: 3 2024-03-25T18:31:37.509 INFO:teuthology.misc.health.smithi001.stderr:0: v1:172.21.15.1:6789/0 mon.a 2024-03-25T18:31:37.509 INFO:teuthology.misc.health.smithi001.stderr:1: v1:172.21.15.115:6789/0 mon.b 2024-03-25T18:31:37.509 INFO:teuthology.misc.health.smithi001.stderr:2: v1:172.21.15.1:6790/0 mon.c 2024-03-25T18:31:37.509 INFO:teuthology.misc.health.smithi001.stderr: 2024-03-25T18:31:37.509 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:31:37.506+0000 7ff2adffb640 10 monclient(hunting): _finish_auth 0 2024-03-25T18:31:37.509 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:31:37.506+0000 7ff2adffb640 1 -- 172.21.15.1:0/1397676503 <== mon.0 v1:172.21.15.1:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2725702128 0 0) 0x7ff2b8135ea0 con 0x7ff2b813c480 2024-03-25T18:31:37.509 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:31:37.506+0000 7ff2adffb640 10 monclient(hunting): _negotiate 2024-03-25T18:31:37.509 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:31:37.506+0000 7ff2adffb640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:31:37.509 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:31:37.506+0000 7ff2adffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:31:37.509 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:31:37.506+0000 7ff2adffb640 10 monclient(hunting): my global_id is 4431 2024-03-25T18:31:37.509 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:31:37.506+0000 7ff2adffb640 1 -- 172.21.15.1:0/1397676503 --> v1:172.21.15.1:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ff29c004eb0 con 0x7ff2b813c480 2024-03-25T18:31:37.512 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:31:37.510+0000 7ff2adffb640 1 -- 172.21.15.1:0/1397676503 <== mon.1 v1:172.21.15.115:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 3452251188 0 0) 0x7ff2b80ac3d0 con 0x7ff2b80b4ff0 2024-03-25T18:31:37.512 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:31:37.510+0000 7ff2adffb640 1 -- 172.21.15.1:0/1397676503 >> v1:172.21.15.1:6790/0 conn(0x7ff2b80b4670 legacy=0x7ff2b813b8d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:31:37.512 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:31:37.510+0000 7ff2adffb640 1 -- 172.21.15.1:0/1397676503 >> v1:172.21.15.1:6789/0 conn(0x7ff2b813c480 legacy=0x7ff2b81362d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:31:37.512 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:31:37.510+0000 7ff2adffb640 10 monclient: _finish_hunting 0 2024-03-25T18:31:37.512 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:31:37.510+0000 7ff2adffb640 1 monclient: found mon.b 2024-03-25T18:31:37.512 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:31:37.510+0000 7ff2adffb640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:31:37.512 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:31:37.510+0000 7ff2adffb640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.115:6789/0 2024-03-25T18:31:37.512 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:31:37.510+0000 7ff2adffb640 1 -- 172.21.15.1:0/1397676503 --> v1:172.21.15.115:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff2b813c110 con 0x7ff2b80b4ff0 2024-03-25T18:31:37.512 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:31:37.510+0000 7ff2adffb640 10 monclient: _finish_auth 0 2024-03-25T18:31:37.512 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:31:37.510+0000 7ff2adffb640 10 monclient: _check_auth_tickets 2024-03-25T18:31:37.512 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:31:37.510+0000 7ff2adffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:31:37.512 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:31:37.510+0000 7ff2bf4c0640 5 monclient: authenticate success, global_id 4441 2024-03-25T18:31:37.512 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:31:37.510+0000 7ff2bf4c0640 10 monclient: _renew_subs 2024-03-25T18:31:37.513 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:31:37.510+0000 7ff2bf4c0640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.115:6789/0 2024-03-25T18:31:37.513 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:31:37.510+0000 7ff2bf4c0640 1 -- 172.21.15.1:0/1397676503 --> v1:172.21.15.115:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff2b813ae10 con 0x7ff2b80b4ff0 2024-03-25T18:31:37.513 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:31:37.510+0000 7ff2bf4c0640 10 client.4441.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T18:31:37.513 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:31:37.510+0000 7ff2bf4c0640 10 monclient: _renew_subs 2024-03-25T18:31:37.513 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:31:37.510+0000 7ff2bf4c0640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.115:6789/0 2024-03-25T18:31:37.514 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:31:37.510+0000 7ff2bf4c0640 1 -- 172.21.15.1:0/1397676503 --> v1:172.21.15.115:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7ff2b80b1550 con 0x7ff2b80b4ff0 2024-03-25T18:31:37.514 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:31:37.510+0000 7ff2bf4c0640 1 librados: init done 2024-03-25T18:31:37.514 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:31:37.511+0000 7ff2bf4c0640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T18:31:37.514 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:31:37.511+0000 7ff2bf4c0640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:31:37.514 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:31:37.511+0000 7ff2bf4c0640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.115:6789/0 2024-03-25T18:31:37.514 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:31:37.511+0000 7ff2bf4c0640 1 -- 172.21.15.1:0/1397676503 --> v1:172.21.15.115:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff2b806dbd0 con 0x7ff2b80b4ff0 2024-03-25T18:31:37.783 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:31:37.781+0000 7ff2adffb640 1 -- 172.21.15.1:0/1397676503 <== mon.1 v1:172.21.15.115:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7ff2a8001cd0 con 0x7ff2b80b4ff0 2024-03-25T18:31:37.783 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:31:37.781+0000 7ff2adffb640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:31:37.783 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:31:37.781+0000 7ff2adffb640 1 -- 172.21.15.1:0/1397676503 <== mon.1 v1:172.21.15.115:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1586965025 0 0) 0x7ff2a8003a60 con 0x7ff2b80b4ff0 2024-03-25T18:31:37.783 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:31:37.781+0000 7ff2adffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:31:37.783 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:31:37.781+0000 7ff2adffb640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-25T18:31:37.784 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:31:37.781+0000 7ff2adffb640 10 monclient: dump: 2024-03-25T18:31:37.784 INFO:teuthology.misc.health.smithi001.stderr:epoch 1 2024-03-25T18:31:37.784 INFO:teuthology.misc.health.smithi001.stderr:fsid c8ea72cf-9e99-4383-9b52-6c23d365fdfc 2024-03-25T18:31:37.784 INFO:teuthology.misc.health.smithi001.stderr:last_changed 2024-03-25T18:25:50.510519+0000 2024-03-25T18:31:37.784 INFO:teuthology.misc.health.smithi001.stderr:created 2024-03-25T18:25:50.510519+0000 2024-03-25T18:31:37.784 INFO:teuthology.misc.health.smithi001.stderr:min_mon_release 19 (squid) 2024-03-25T18:31:37.784 INFO:teuthology.misc.health.smithi001.stderr:election_strategy: 3 2024-03-25T18:31:37.784 INFO:teuthology.misc.health.smithi001.stderr:0: v1:172.21.15.1:6789/0 mon.a 2024-03-25T18:31:37.784 INFO:teuthology.misc.health.smithi001.stderr:1: v1:172.21.15.115:6789/0 mon.b 2024-03-25T18:31:37.784 INFO:teuthology.misc.health.smithi001.stderr:2: v1:172.21.15.1:6790/0 mon.c 2024-03-25T18:31:37.784 INFO:teuthology.misc.health.smithi001.stderr: 2024-03-25T18:31:37.784 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:31:37.781+0000 7ff2af7fe640 4 set_mon_vals no callback set 2024-03-25T18:31:37.785 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:31:37.782+0000 7ff2adffb640 1 -- 172.21.15.1:0/1397676503 <== mon.1 v1:172.21.15.115:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 21787975 0 0) 0x7ff2a8003c80 con 0x7ff2b80b4ff0 2024-03-25T18:31:37.785 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:31:37.782+0000 7ff2bd25c640 10 monclient: get_auth_request con 0x7ff29c06dd30 auth_method 0 2024-03-25T18:31:37.785 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:31:37.783+0000 7ff2adffb640 10 client.4441.objecter ms_handle_connect 0x7ff29c06dd30 2024-03-25T18:31:37.788 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:31:37.786+0000 7ff2adffb640 1 -- 172.21.15.1:0/1397676503 <== mon.1 v1:172.21.15.115:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 996254230 0 0) 0x7ff2a80548f0 con 0x7ff2b80b4ff0 2024-03-25T18:31:37.789 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:31:37.786+0000 7ff2adffb640 10 client.4441.objecter ms_dispatch 0x7ff2b8000b90 osd_map(10..10 src has 1..10) v4 2024-03-25T18:31:37.789 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:31:37.786+0000 7ff2adffb640 3 client.4441.objecter handle_osd_map got epochs [10,10] > 0 2024-03-25T18:31:37.789 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:31:37.786+0000 7ff2adffb640 3 client.4441.objecter handle_osd_map decoding full epoch 10 2024-03-25T18:31:37.789 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:31:37.786+0000 7ff2adffb640 20 client.4441.objecter dump_active .. 0 homeless 2024-03-25T18:31:37.789 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:31:37.787+0000 7ff2adffb640 1 -- 172.21.15.1:0/1397676503 <== mon.1 v1:172.21.15.115:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7ff2a8050400 con 0x7ff2b80b4ff0 2024-03-25T18:31:37.789 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:31:37.787+0000 7ff2adffb640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:31:37.789 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:31:37.787+0000 7ff2adffb640 10 monclient: _finish_command 1 = system:0 2024-03-25T18:31:38.032 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:31:38.029+0000 7ff2bf4c0640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:31:38.032 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:31:38.029+0000 7ff2bf4c0640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:31:38.032 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:31:38.029+0000 7ff2bf4c0640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.115:6789/0 2024-03-25T18:31:38.032 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:31:38.029+0000 7ff2bf4c0640 1 -- 172.21.15.1:0/1397676503 --> v1:172.21.15.115:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7ff2b8146790 con 0x7ff2b80b4ff0 2024-03-25T18:31:38.053 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:31:38.051+0000 7ff2adffb640 1 -- 172.21.15.1:0/1397676503 <== mon.1 v1:172.21.15.115:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 4092213348) 0x7ff2a8052330 con 0x7ff2b80b4ff0 2024-03-25T18:31:38.053 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:31:38.051+0000 7ff2adffb640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:31:38.053 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:31:38.051+0000 7ff2adffb640 10 monclient: _finish_command 2 = system:0 2024-03-25T18:31:38.053 INFO:teuthology.misc.health.smithi001.stdout: 2024-03-25T18:31:38.053 INFO:teuthology.misc.health.smithi001.stdout:{"epoch":10,"fsid":"c8ea72cf-9e99-4383-9b52-6c23d365fdfc","created":"2024-03-25T18:26:38.856638+0000","modified":"2024-03-25T18:26:44.224208+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T18:26:44.224208+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"b9f34922-3a84-49be-8426-807c1eceaefc","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"42bf306f-55e3-4634-9874-8b4b73ac6644","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"292cb23a-18a9-4207-a7cc-4d5727feaec0","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"45e0e9ab-898d-4673-9590-b4117bc5e84a","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"831a5a09-73ab-4a61-9872-088e76bdbc66","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"88cd903d-3b55-4840-9396-6e3cfe005282","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"89db7355-bc9d-46f4-8ab0-e8d9e744e026","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"cac47bee-0914-4f08-896f-348838882866","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T18:31:38.056 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:31:38.054+0000 7ff2bf4c0640 10 librados: watch_flush enter 2024-03-25T18:31:38.056 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:31:38.054+0000 7ff2bf4c0640 10 librados: watch_flush exit 2024-03-25T18:31:38.056 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:31:38.054+0000 7ff2bf4c0640 20 client.4441.objecter shutdown clearing up homeless session... 2024-03-25T18:31:38.056 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:31:38.054+0000 7ff2bf4c0640 10 client.4441.objecter successfully canceled tick 2024-03-25T18:31:38.057 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:31:38.054+0000 7ff2bf4c0640 1 -- 172.21.15.1:0/1397676503 >> v1:172.21.15.115:6800/4237379176 conn(0x7ff29c06dd30 legacy=0x7ff29c0701f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:31:38.057 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:31:38.054+0000 7ff2bf4c0640 10 monclient: shutdown 2024-03-25T18:31:38.057 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:31:38.054+0000 7ff2bf4c0640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:31:38.057 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:31:38.054+0000 7ff2bf4c0640 1 -- 172.21.15.1:0/1397676503 >> v1:172.21.15.115:6789/0 conn(0x7ff2b80b4ff0 legacy=0x7ff2b80b0e40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:31:38.057 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:31:38.054+0000 7ff2bf4c0640 1 -- 172.21.15.1:0/1397676503 shutdown_connections 2024-03-25T18:31:38.057 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:31:38.054+0000 7ff2bf4c0640 1 -- 172.21.15.1:0/1397676503 >> 172.21.15.1:0/1397676503 conn(0x7ff2b80617a0 msgr2=0x7ff2b81407a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:31:38.057 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:31:38.054+0000 7ff2bf4c0640 1 -- 172.21.15.1:0/1397676503 shutdown_connections 2024-03-25T18:31:38.057 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:31:38.054+0000 7ff2bf4c0640 1 -- 172.21.15.1:0/1397676503 wait complete. 2024-03-25T18:31:38.057 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:31:38.054+0000 7ff2bf4c0640 1 librados: shutdown 2024-03-25T18:31:38.068 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T18:31:44.874 DEBUG:teuthology.orchestra.run.smithi001:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T18:31:44.989 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:31:44.987+0000 7fe9526a8640 10 monclient: get_monmap_and_config 2024-03-25T18:31:44.990 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:31:44.987+0000 7fe9526a8640 10 monclient: build_initial_monmap 2024-03-25T18:31:44.990 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:31:44.987+0000 7fe9526a8640 10 monclient: monmap: 2024-03-25T18:31:44.990 INFO:teuthology.misc.health.smithi001.stderr:epoch 0 2024-03-25T18:31:44.990 INFO:teuthology.misc.health.smithi001.stderr:fsid c8ea72cf-9e99-4383-9b52-6c23d365fdfc 2024-03-25T18:31:44.990 INFO:teuthology.misc.health.smithi001.stderr:last_changed 2024-03-25T18:31:44.989370+0000 2024-03-25T18:31:44.990 INFO:teuthology.misc.health.smithi001.stderr:created 2024-03-25T18:31:44.989370+0000 2024-03-25T18:31:44.990 INFO:teuthology.misc.health.smithi001.stderr:min_mon_release 0 (unknown) 2024-03-25T18:31:44.990 INFO:teuthology.misc.health.smithi001.stderr:election_strategy: 3 2024-03-25T18:31:44.990 INFO:teuthology.misc.health.smithi001.stderr:0: v1:172.21.15.1:6789/0 mon.noname-a 2024-03-25T18:31:44.990 INFO:teuthology.misc.health.smithi001.stderr:1: v1:172.21.15.115:6789/0 mon.noname-c 2024-03-25T18:31:44.990 INFO:teuthology.misc.health.smithi001.stderr:2: v1:172.21.15.1:6790/0 mon.noname-b 2024-03-25T18:31:44.991 INFO:teuthology.misc.health.smithi001.stderr: 2024-03-25T18:31:44.992 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:31:44.988+0000 7fe9526a8640 1 Processor -- start 2024-03-25T18:31:44.992 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:31:44.988+0000 7fe9526a8640 1 -- start start 2024-03-25T18:31:44.992 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:31:44.988+0000 7fe9526a8640 10 monclient: init 2024-03-25T18:31:44.992 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:31:44.989+0000 7fe9526a8640 10 monclient: _reopen_session rank -1 2024-03-25T18:31:44.992 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:31:44.989+0000 7fe9526a8640 10 monclient: _add_conns ranks=[2,1,0] 2024-03-25T18:31:44.992 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:31:44.989+0000 7fe9526a8640 10 monclient(hunting): picked mon.noname-b con 0x7fe94c0b4e70 addr v1:172.21.15.1:6790/0 2024-03-25T18:31:44.992 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:31:44.989+0000 7fe9526a8640 10 monclient(hunting): picked mon.noname-c con 0x7fe94c0b44b0 addr v1:172.21.15.115:6789/0 2024-03-25T18:31:44.992 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:31:44.989+0000 7fe9526a8640 10 monclient(hunting): picked mon.noname-a con 0x7fe94c0b1b00 addr v1:172.21.15.1:6789/0 2024-03-25T18:31:44.992 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:31:44.989+0000 7fe9526a8640 1 -- --> v1:172.21.15.1:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe94c0a96f0 con 0x7fe94c0b1b00 2024-03-25T18:31:44.992 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:31:44.989+0000 7fe9526a8640 1 -- --> v1:172.21.15.115:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe94c0ab0e0 con 0x7fe94c0b44b0 2024-03-25T18:31:44.993 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:31:44.989+0000 7fe9526a8640 1 -- --> v1:172.21.15.1:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe94c145f70 con 0x7fe94c0b4e70 2024-03-25T18:31:44.993 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:31:44.989+0000 7fe9526a8640 10 monclient(hunting): _renew_subs 2024-03-25T18:31:44.993 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:31:44.989+0000 7fe9526a8640 10 monclient(hunting): authenticate will time out at 7785.460283s 2024-03-25T18:31:45.000 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:31:44.999+0000 7fe943fff640 1 --1- >> v1:172.21.15.115:6789/0 conn(0x7fe94c0b44b0 0x7fe94c145860 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.115:6789/0 says I am v1:172.21.15.1:37066/0 (socket says 172.21.15.1:37066) 2024-03-25T18:31:45.001 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:31:44.999+0000 7fe943fff640 1 -- 172.21.15.1:0/1239459387 learned_addr learned my addr 172.21.15.1:0/1239459387 (peer_addr_for_me v1:172.21.15.1:0/0) 2024-03-25T18:31:45.014 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:31:45.012+0000 7fe94b7fe640 1 -- 172.21.15.1:0/1239459387 <== mon.1 v1:172.21.15.115:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2468181147 0 0) 0x7fe94c0ab0e0 con 0x7fe94c0b44b0 2024-03-25T18:31:45.014 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:31:45.012+0000 7fe94b7fe640 10 monclient(hunting): _negotiate 2024-03-25T18:31:45.014 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:31:45.012+0000 7fe94b7fe640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:31:45.014 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:31:45.012+0000 7fe94b7fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:31:45.014 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:31:45.012+0000 7fe94b7fe640 10 monclient(hunting): my global_id is 4444 2024-03-25T18:31:45.014 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:31:45.012+0000 7fe94b7fe640 1 -- 172.21.15.1:0/1239459387 --> v1:172.21.15.115:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fe938003520 con 0x7fe94c0b44b0 2024-03-25T18:31:45.025 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:31:45.023+0000 7fe94b7fe640 1 -- 172.21.15.1:0/1239459387 <== mon.0 v1:172.21.15.1:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3572444980 0 0) 0x7fe94c0a96f0 con 0x7fe94c0b1b00 2024-03-25T18:31:45.025 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:31:45.023+0000 7fe94b7fe640 10 monclient(hunting): _negotiate 2024-03-25T18:31:45.025 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:31:45.023+0000 7fe94b7fe640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:31:45.025 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:31:45.023+0000 7fe94b7fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:31:45.025 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:31:45.023+0000 7fe94b7fe640 10 monclient(hunting): my global_id is 4434 2024-03-25T18:31:45.025 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:31:45.023+0000 7fe94b7fe640 1 -- 172.21.15.1:0/1239459387 --> v1:172.21.15.1:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fe94c0ab0e0 con 0x7fe94c0b1b00 2024-03-25T18:31:45.025 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:31:45.023+0000 7fe94b7fe640 1 -- 172.21.15.1:0/1239459387 <== mon.2 v1:172.21.15.1:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2604295512 0 0) 0x7fe94c145f70 con 0x7fe94c0b4e70 2024-03-25T18:31:45.025 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:31:45.023+0000 7fe94b7fe640 10 monclient(hunting): _negotiate 2024-03-25T18:31:45.025 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:31:45.023+0000 7fe94b7fe640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:31:45.025 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:31:45.023+0000 7fe94b7fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:31:45.025 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:31:45.023+0000 7fe94b7fe640 10 monclient(hunting): my global_id is 4445 2024-03-25T18:31:45.025 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:31:45.023+0000 7fe94b7fe640 1 -- 172.21.15.1:0/1239459387 --> v1:172.21.15.1:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fe94c0a96f0 con 0x7fe94c0b4e70 2024-03-25T18:31:45.034 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:31:45.032+0000 7fe94b7fe640 1 -- 172.21.15.1:0/1239459387 <== mon.0 v1:172.21.15.1:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1126617494 0 0) 0x7fe94c0ab0e0 con 0x7fe94c0b1b00 2024-03-25T18:31:45.034 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:31:45.032+0000 7fe94b7fe640 1 -- 172.21.15.1:0/1239459387 >> v1:172.21.15.1:6790/0 conn(0x7fe94c0b4e70 legacy=0x7fe94c13f940 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:31:45.034 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:31:45.032+0000 7fe94b7fe640 1 -- 172.21.15.1:0/1239459387 >> v1:172.21.15.115:6789/0 conn(0x7fe94c0b44b0 legacy=0x7fe94c145860 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:31:45.034 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:31:45.032+0000 7fe94b7fe640 10 monclient: _finish_hunting 0 2024-03-25T18:31:45.034 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:31:45.032+0000 7fe94b7fe640 1 monclient: found mon.noname-a 2024-03-25T18:31:45.034 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:31:45.032+0000 7fe94b7fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:31:45.034 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:31:45.032+0000 7fe94b7fe640 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.1:6789/0 2024-03-25T18:31:45.034 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:31:45.032+0000 7fe94b7fe640 1 -- 172.21.15.1:0/1239459387 --> v1:172.21.15.1:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe94c0a9b80 con 0x7fe94c0b1b00 2024-03-25T18:31:45.034 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:31:45.032+0000 7fe94b7fe640 10 monclient: _finish_auth 0 2024-03-25T18:31:45.034 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:31:45.032+0000 7fe94b7fe640 10 monclient: _check_auth_tickets 2024-03-25T18:31:45.034 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:31:45.032+0000 7fe94b7fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:31:45.035 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:31:45.032+0000 7fe94b7fe640 1 -- 172.21.15.1:0/1239459387 <== mon.0 v1:172.21.15.1:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1586965025 0 0) 0x7fe9440019f0 con 0x7fe94c0b1b00 2024-03-25T18:31:45.035 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:31:45.032+0000 7fe94b7fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:31:45.035 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:31:45.032+0000 7fe94b7fe640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-03-25T18:31:45.035 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:31:45.032+0000 7fe94b7fe640 10 monclient: dump: 2024-03-25T18:31:45.035 INFO:teuthology.misc.health.smithi001.stderr:epoch 1 2024-03-25T18:31:45.035 INFO:teuthology.misc.health.smithi001.stderr:fsid c8ea72cf-9e99-4383-9b52-6c23d365fdfc 2024-03-25T18:31:45.035 INFO:teuthology.misc.health.smithi001.stderr:last_changed 2024-03-25T18:25:50.510519+0000 2024-03-25T18:31:45.035 INFO:teuthology.misc.health.smithi001.stderr:created 2024-03-25T18:25:50.510519+0000 2024-03-25T18:31:45.035 INFO:teuthology.misc.health.smithi001.stderr:min_mon_release 19 (squid) 2024-03-25T18:31:45.035 INFO:teuthology.misc.health.smithi001.stderr:election_strategy: 3 2024-03-25T18:31:45.035 INFO:teuthology.misc.health.smithi001.stderr:0: v1:172.21.15.1:6789/0 mon.a 2024-03-25T18:31:45.035 INFO:teuthology.misc.health.smithi001.stderr:1: v1:172.21.15.115:6789/0 mon.b 2024-03-25T18:31:45.035 INFO:teuthology.misc.health.smithi001.stderr:2: v1:172.21.15.1:6790/0 mon.c 2024-03-25T18:31:45.035 INFO:teuthology.misc.health.smithi001.stderr: 2024-03-25T18:31:45.035 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:31:45.032+0000 7fe9526a8640 5 monclient: authenticate success, global_id 4434 2024-03-25T18:31:45.036 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:31:45.032+0000 7fe9526a8640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T18:31:45.052 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:31:45.050+0000 7fe94b7fe640 1 -- 172.21.15.1:0/1239459387 <== mon.0 v1:172.21.15.1:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fe944001b50 con 0x7fe94c0b1b00 2024-03-25T18:31:45.052 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:31:45.050+0000 7fe94b7fe640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:31:45.052 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:31:45.050+0000 7fe94b7fe640 1 -- 172.21.15.1:0/1239459387 <== mon.0 v1:172.21.15.1:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1586965025 0 0) 0x7fe944001f60 con 0x7fe94c0b1b00 2024-03-25T18:31:45.052 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:31:45.050+0000 7fe9526a8640 10 monclient: get_monmap_and_config success 2024-03-25T18:31:45.052 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:31:45.050+0000 7fe9526a8640 4 set_mon_vals no callback set 2024-03-25T18:31:45.052 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:31:45.050+0000 7fe94b7fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:31:45.052 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:31:45.050+0000 7fe94b7fe640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-25T18:31:45.052 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:31:45.050+0000 7fe94b7fe640 10 monclient: dump: 2024-03-25T18:31:45.052 INFO:teuthology.misc.health.smithi001.stderr:epoch 1 2024-03-25T18:31:45.052 INFO:teuthology.misc.health.smithi001.stderr:fsid c8ea72cf-9e99-4383-9b52-6c23d365fdfc 2024-03-25T18:31:45.053 INFO:teuthology.misc.health.smithi001.stderr:last_changed 2024-03-25T18:25:50.510519+0000 2024-03-25T18:31:45.053 INFO:teuthology.misc.health.smithi001.stderr:created 2024-03-25T18:25:50.510519+0000 2024-03-25T18:31:45.053 INFO:teuthology.misc.health.smithi001.stderr:min_mon_release 19 (squid) 2024-03-25T18:31:45.053 INFO:teuthology.misc.health.smithi001.stderr:election_strategy: 3 2024-03-25T18:31:45.053 INFO:teuthology.misc.health.smithi001.stderr:0: v1:172.21.15.1:6789/0 mon.a 2024-03-25T18:31:45.053 INFO:teuthology.misc.health.smithi001.stderr:1: v1:172.21.15.115:6789/0 mon.b 2024-03-25T18:31:45.053 INFO:teuthology.misc.health.smithi001.stderr:2: v1:172.21.15.1:6790/0 mon.c 2024-03-25T18:31:45.053 INFO:teuthology.misc.health.smithi001.stderr: 2024-03-25T18:31:45.055 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:31:45.051+0000 7fe9526a8640 10 monclient: shutdown 2024-03-25T18:31:45.055 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:31:45.051+0000 7fe9526a8640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:31:45.055 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:31:45.051+0000 7fe9526a8640 1 -- 172.21.15.1:0/1239459387 >> v1:172.21.15.1:6789/0 conn(0x7fe94c0b1b00 legacy=0x7fe94c14dc00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:31:45.055 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:31:45.051+0000 7fe9526a8640 1 -- 172.21.15.1:0/1239459387 shutdown_connections 2024-03-25T18:31:45.056 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:31:45.051+0000 7fe9526a8640 1 -- 172.21.15.1:0/1239459387 >> 172.21.15.1:0/1239459387 conn(0x7fe94c0617a0 msgr2=0x7fe94c061bb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:31:45.056 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:31:45.051+0000 7fe9526a8640 1 -- 172.21.15.1:0/1239459387 shutdown_connections 2024-03-25T18:31:45.056 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:31:45.051+0000 7fe9526a8640 1 -- 172.21.15.1:0/1239459387 wait complete. 2024-03-25T18:31:45.056 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:31:45.052+0000 7fe9526a8640 10 monclient: build_initial_monmap 2024-03-25T18:31:45.056 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:31:45.052+0000 7fe9526a8640 10 monclient: monmap: 2024-03-25T18:31:45.056 INFO:teuthology.misc.health.smithi001.stderr:epoch 0 2024-03-25T18:31:45.056 INFO:teuthology.misc.health.smithi001.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T18:31:45.057 INFO:teuthology.misc.health.smithi001.stderr:last_changed 0.000000 2024-03-25T18:31:45.057 INFO:teuthology.misc.health.smithi001.stderr:created 0.000000 2024-03-25T18:31:45.057 INFO:teuthology.misc.health.smithi001.stderr:min_mon_release 0 (unknown) 2024-03-25T18:31:45.057 INFO:teuthology.misc.health.smithi001.stderr:election_strategy: 1 2024-03-25T18:31:45.057 INFO:teuthology.misc.health.smithi001.stderr:0: v1:172.21.15.1:6789/0 mon.noname-a 2024-03-25T18:31:45.057 INFO:teuthology.misc.health.smithi001.stderr:1: v1:172.21.15.115:6789/0 mon.noname-b 2024-03-25T18:31:45.057 INFO:teuthology.misc.health.smithi001.stderr:2: v1:172.21.15.1:6790/0 mon.noname-c 2024-03-25T18:31:45.057 INFO:teuthology.misc.health.smithi001.stderr: 2024-03-25T18:31:45.058 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:31:45.052+0000 7fe9526a8640 1 librados: starting msgr at 2024-03-25T18:31:45.058 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:31:45.052+0000 7fe9526a8640 1 librados: starting objecter 2024-03-25T18:31:45.058 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:31:45.052+0000 7fe9526a8640 1 Processor -- start 2024-03-25T18:31:45.058 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:31:45.052+0000 7fe9526a8640 1 -- start start 2024-03-25T18:31:45.058 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:31:45.052+0000 7fe9526a8640 1 librados: setting wanted keys 2024-03-25T18:31:45.058 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:31:45.052+0000 7fe9526a8640 1 librados: calling monclient init 2024-03-25T18:31:45.058 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:31:45.052+0000 7fe9526a8640 10 monclient: init 2024-03-25T18:31:45.058 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:31:45.053+0000 7fe9526a8640 10 monclient: _reopen_session rank -1 2024-03-25T18:31:45.058 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:31:45.053+0000 7fe9526a8640 10 monclient: _add_conns ranks=[2,0,1] 2024-03-25T18:31:45.058 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:31:45.053+0000 7fe9526a8640 10 monclient(hunting): picked mon.noname-c con 0x7fe94c0b1b00 addr v1:172.21.15.1:6790/0 2024-03-25T18:31:45.058 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:31:45.053+0000 7fe9526a8640 10 monclient(hunting): picked mon.noname-a con 0x7fe94c0b44b0 addr v1:172.21.15.1:6789/0 2024-03-25T18:31:45.058 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:31:45.053+0000 7fe9526a8640 10 monclient(hunting): picked mon.noname-b con 0x7fe94c0b4e70 addr v1:172.21.15.115:6789/0 2024-03-25T18:31:45.058 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:31:45.053+0000 7fe9526a8640 1 -- --> v1:172.21.15.1:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe94c1375f0 con 0x7fe94c0b44b0 2024-03-25T18:31:45.059 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:31:45.053+0000 7fe9526a8640 1 -- --> v1:172.21.15.115:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe94c1378d0 con 0x7fe94c0b4e70 2024-03-25T18:31:45.059 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:31:45.053+0000 7fe9526a8640 1 -- --> v1:172.21.15.1:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe94c138bd0 con 0x7fe94c0b1b00 2024-03-25T18:31:45.059 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:31:45.053+0000 7fe9526a8640 10 monclient(hunting): _renew_subs 2024-03-25T18:31:45.059 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:31:45.053+0000 7fe9526a8640 10 monclient(hunting): authenticate will time out at 7785.524091s 2024-03-25T18:31:45.069 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:31:45.067+0000 7fe943fff640 1 --1- >> v1:172.21.15.1:6789/0 conn(0x7fe94c0b44b0 0x7fe94c1d76e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.1:6789/0 says I am v1:172.21.15.1:33128/0 (socket says 172.21.15.1:33128) 2024-03-25T18:31:45.069 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:31:45.067+0000 7fe943fff640 1 -- 172.21.15.1:0/2469895755 learned_addr learned my addr 172.21.15.1:0/2469895755 (peer_addr_for_me v1:172.21.15.1:0/0) 2024-03-25T18:31:45.072 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:31:45.071+0000 7fe9497fa640 10 client.?.objecter ms_handle_connect 0x7fe94c0b4e70 2024-03-25T18:31:45.072 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:31:45.071+0000 7fe9497fa640 10 client.?.objecter resend_mon_ops 2024-03-25T18:31:45.073 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:31:45.071+0000 7fe9497fa640 10 client.?.objecter ms_handle_connect 0x7fe94c0b44b0 2024-03-25T18:31:45.073 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:31:45.071+0000 7fe9497fa640 10 client.?.objecter resend_mon_ops 2024-03-25T18:31:45.074 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:31:45.072+0000 7fe9497fa640 10 client.?.objecter ms_handle_connect 0x7fe94c0b1b00 2024-03-25T18:31:45.074 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:31:45.072+0000 7fe9497fa640 10 client.?.objecter resend_mon_ops 2024-03-25T18:31:45.082 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:31:45.080+0000 7fe9497fa640 1 -- 172.21.15.1:0/2469895755 <== mon.1 v1:172.21.15.115:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1169053738 0 0) 0x7fe94c1378d0 con 0x7fe94c0b4e70 2024-03-25T18:31:45.082 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:31:45.080+0000 7fe9497fa640 10 monclient(hunting): _negotiate 2024-03-25T18:31:45.082 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:31:45.080+0000 7fe9497fa640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:31:45.082 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:31:45.080+0000 7fe9497fa640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:31:45.082 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:31:45.080+0000 7fe9497fa640 10 monclient(hunting): my global_id is 4447 2024-03-25T18:31:45.082 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:31:45.080+0000 7fe9497fa640 1 -- 172.21.15.1:0/2469895755 --> v1:172.21.15.115:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fe92c004eb0 con 0x7fe94c0b4e70 2024-03-25T18:31:45.083 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:31:45.080+0000 7fe9497fa640 1 -- 172.21.15.1:0/2469895755 <== mon.0 v1:172.21.15.1:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2469549760 0 0) 0x7fe94c1375f0 con 0x7fe94c0b44b0 2024-03-25T18:31:45.083 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:31:45.080+0000 7fe9497fa640 10 monclient(hunting): _negotiate 2024-03-25T18:31:45.083 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:31:45.080+0000 7fe9497fa640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:31:45.083 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:31:45.080+0000 7fe9497fa640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:31:45.083 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:31:45.080+0000 7fe9497fa640 10 monclient(hunting): my global_id is 4437 2024-03-25T18:31:45.083 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:31:45.080+0000 7fe9497fa640 1 -- 172.21.15.1:0/2469895755 --> v1:172.21.15.1:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fe94c1378d0 con 0x7fe94c0b44b0 2024-03-25T18:31:45.083 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:31:45.081+0000 7fe9497fa640 1 -- 172.21.15.1:0/2469895755 <== mon.2 v1:172.21.15.1:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1411227836 0 0) 0x7fe94c138bd0 con 0x7fe94c0b1b00 2024-03-25T18:31:45.084 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:31:45.081+0000 7fe9497fa640 10 monclient(hunting): _negotiate 2024-03-25T18:31:45.084 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:31:45.081+0000 7fe9497fa640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:31:45.084 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:31:45.081+0000 7fe9497fa640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:31:45.084 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:31:45.081+0000 7fe9497fa640 10 monclient(hunting): my global_id is 4448 2024-03-25T18:31:45.084 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:31:45.081+0000 7fe9497fa640 1 -- 172.21.15.1:0/2469895755 --> v1:172.21.15.1:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fe94c1375f0 con 0x7fe94c0b1b00 2024-03-25T18:31:45.089 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:31:45.087+0000 7fe9497fa640 1 -- 172.21.15.1:0/2469895755 <== mon.1 v1:172.21.15.115:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2336938956 0 0) 0x7fe92c004eb0 con 0x7fe94c0b4e70 2024-03-25T18:31:45.089 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:31:45.087+0000 7fe9497fa640 1 -- 172.21.15.1:0/2469895755 --> v1:172.21.15.115:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fe94c138bd0 con 0x7fe94c0b4e70 2024-03-25T18:31:45.089 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:31:45.087+0000 7fe9497fa640 1 -- 172.21.15.1:0/2469895755 <== mon.1 v1:172.21.15.115:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1586965025 0 0) 0x7fe944001e60 con 0x7fe94c0b4e70 2024-03-25T18:31:45.089 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:31:45.087+0000 7fe9497fa640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:31:45.090 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:31:45.087+0000 7fe9497fa640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-03-25T18:31:45.090 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:31:45.087+0000 7fe9497fa640 10 monclient(hunting): dump: 2024-03-25T18:31:45.090 INFO:teuthology.misc.health.smithi001.stderr:epoch 1 2024-03-25T18:31:45.090 INFO:teuthology.misc.health.smithi001.stderr:fsid c8ea72cf-9e99-4383-9b52-6c23d365fdfc 2024-03-25T18:31:45.090 INFO:teuthology.misc.health.smithi001.stderr:last_changed 2024-03-25T18:25:50.510519+0000 2024-03-25T18:31:45.090 INFO:teuthology.misc.health.smithi001.stderr:created 2024-03-25T18:25:50.510519+0000 2024-03-25T18:31:45.090 INFO:teuthology.misc.health.smithi001.stderr:min_mon_release 19 (squid) 2024-03-25T18:31:45.090 INFO:teuthology.misc.health.smithi001.stderr:election_strategy: 3 2024-03-25T18:31:45.090 INFO:teuthology.misc.health.smithi001.stderr:0: v1:172.21.15.1:6789/0 mon.a 2024-03-25T18:31:45.090 INFO:teuthology.misc.health.smithi001.stderr:1: v1:172.21.15.115:6789/0 mon.b 2024-03-25T18:31:45.090 INFO:teuthology.misc.health.smithi001.stderr:2: v1:172.21.15.1:6790/0 mon.c 2024-03-25T18:31:45.090 INFO:teuthology.misc.health.smithi001.stderr: 2024-03-25T18:31:45.090 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:31:45.087+0000 7fe9497fa640 10 monclient(hunting): _finish_auth 0 2024-03-25T18:31:45.093 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:31:45.090+0000 7fe9497fa640 1 -- 172.21.15.1:0/2469895755 <== mon.2 v1:172.21.15.1:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 133904767 0 0) 0x7fe94c1375f0 con 0x7fe94c0b1b00 2024-03-25T18:31:45.093 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:31:45.090+0000 7fe9497fa640 1 -- 172.21.15.1:0/2469895755 --> v1:172.21.15.1:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fe92c004eb0 con 0x7fe94c0b1b00 2024-03-25T18:31:45.093 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:31:45.090+0000 7fe9497fa640 1 -- 172.21.15.1:0/2469895755 <== mon.2 v1:172.21.15.1:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1586965025 0 0) 0x7fe93c001790 con 0x7fe94c0b1b00 2024-03-25T18:31:45.093 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:31:45.090+0000 7fe9497fa640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:31:45.093 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:31:45.090+0000 7fe9497fa640 10 monclient(hunting): got monmap 1 from mon.c (according to old e1) 2024-03-25T18:31:45.093 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:31:45.090+0000 7fe9497fa640 10 monclient(hunting): dump: 2024-03-25T18:31:45.093 INFO:teuthology.misc.health.smithi001.stderr:epoch 1 2024-03-25T18:31:45.093 INFO:teuthology.misc.health.smithi001.stderr:fsid c8ea72cf-9e99-4383-9b52-6c23d365fdfc 2024-03-25T18:31:45.093 INFO:teuthology.misc.health.smithi001.stderr:last_changed 2024-03-25T18:25:50.510519+0000 2024-03-25T18:31:45.093 INFO:teuthology.misc.health.smithi001.stderr:created 2024-03-25T18:25:50.510519+0000 2024-03-25T18:31:45.094 INFO:teuthology.misc.health.smithi001.stderr:min_mon_release 19 (squid) 2024-03-25T18:31:45.094 INFO:teuthology.misc.health.smithi001.stderr:election_strategy: 3 2024-03-25T18:31:45.094 INFO:teuthology.misc.health.smithi001.stderr:0: v1:172.21.15.1:6789/0 mon.a 2024-03-25T18:31:45.094 INFO:teuthology.misc.health.smithi001.stderr:1: v1:172.21.15.115:6789/0 mon.b 2024-03-25T18:31:45.094 INFO:teuthology.misc.health.smithi001.stderr:2: v1:172.21.15.1:6790/0 mon.c 2024-03-25T18:31:45.094 INFO:teuthology.misc.health.smithi001.stderr: 2024-03-25T18:31:45.094 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:31:45.090+0000 7fe9497fa640 1 -- 172.21.15.1:0/2469895755 <== mon.1 v1:172.21.15.115:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 2674603526 0 0) 0x7fe94c138bd0 con 0x7fe94c0b4e70 2024-03-25T18:31:45.094 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:31:45.090+0000 7fe9497fa640 1 -- 172.21.15.1:0/2469895755 >> v1:172.21.15.1:6790/0 conn(0x7fe94c0b1b00 legacy=0x7fe94c1382b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:31:45.094 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:31:45.090+0000 7fe9497fa640 1 -- 172.21.15.1:0/2469895755 >> v1:172.21.15.1:6789/0 conn(0x7fe94c0b44b0 legacy=0x7fe94c1d76e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:31:45.094 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:31:45.090+0000 7fe9497fa640 10 monclient: _finish_hunting 0 2024-03-25T18:31:45.094 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:31:45.090+0000 7fe9497fa640 1 monclient: found mon.b 2024-03-25T18:31:45.094 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:31:45.090+0000 7fe9497fa640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:31:45.094 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:31:45.090+0000 7fe9497fa640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.115:6789/0 2024-03-25T18:31:45.094 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:31:45.090+0000 7fe9497fa640 1 -- 172.21.15.1:0/2469895755 --> v1:172.21.15.115:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe94c139560 con 0x7fe94c0b4e70 2024-03-25T18:31:45.094 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:31:45.090+0000 7fe9497fa640 10 monclient: _finish_auth 0 2024-03-25T18:31:45.095 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:31:45.090+0000 7fe9497fa640 10 monclient: _check_auth_tickets 2024-03-25T18:31:45.095 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:31:45.090+0000 7fe9497fa640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:31:45.095 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:31:45.090+0000 7fe9526a8640 5 monclient: authenticate success, global_id 4447 2024-03-25T18:31:45.095 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:31:45.090+0000 7fe9526a8640 10 monclient: _renew_subs 2024-03-25T18:31:45.095 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:31:45.090+0000 7fe9526a8640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.115:6789/0 2024-03-25T18:31:45.095 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:31:45.090+0000 7fe9526a8640 1 -- 172.21.15.1:0/2469895755 --> v1:172.21.15.115:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe94c1397b0 con 0x7fe94c0b4e70 2024-03-25T18:31:45.095 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:31:45.091+0000 7fe9526a8640 10 client.4447.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T18:31:45.095 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:31:45.091+0000 7fe9526a8640 10 monclient: _renew_subs 2024-03-25T18:31:45.095 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:31:45.091+0000 7fe9526a8640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.115:6789/0 2024-03-25T18:31:45.095 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:31:45.091+0000 7fe9526a8640 1 -- 172.21.15.1:0/2469895755 --> v1:172.21.15.115:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7fe94c1e3210 con 0x7fe94c0b4e70 2024-03-25T18:31:45.095 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:31:45.091+0000 7fe9526a8640 1 librados: init done 2024-03-25T18:31:45.096 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:31:45.092+0000 7fe9526a8640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T18:31:45.096 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:31:45.092+0000 7fe9526a8640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:31:45.096 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:31:45.092+0000 7fe9526a8640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.115:6789/0 2024-03-25T18:31:45.096 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:31:45.092+0000 7fe9526a8640 1 -- 172.21.15.1:0/2469895755 --> v1:172.21.15.115:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe94c04f730 con 0x7fe94c0b4e70 2024-03-25T18:31:45.371 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:31:45.369+0000 7fe9497fa640 1 -- 172.21.15.1:0/2469895755 <== mon.1 v1:172.21.15.115:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fe944003c20 con 0x7fe94c0b4e70 2024-03-25T18:31:45.371 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:31:45.369+0000 7fe9497fa640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:31:45.371 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:31:45.369+0000 7fe9497fa640 1 -- 172.21.15.1:0/2469895755 <== mon.1 v1:172.21.15.115:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1586965025 0 0) 0x7fe944003f30 con 0x7fe94c0b4e70 2024-03-25T18:31:45.371 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:31:45.369+0000 7fe9497fa640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:31:45.371 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:31:45.369+0000 7fe9497fa640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-25T18:31:45.371 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:31:45.369+0000 7fe9497fa640 10 monclient: dump: 2024-03-25T18:31:45.371 INFO:teuthology.misc.health.smithi001.stderr:epoch 1 2024-03-25T18:31:45.371 INFO:teuthology.misc.health.smithi001.stderr:fsid c8ea72cf-9e99-4383-9b52-6c23d365fdfc 2024-03-25T18:31:45.372 INFO:teuthology.misc.health.smithi001.stderr:last_changed 2024-03-25T18:25:50.510519+0000 2024-03-25T18:31:45.372 INFO:teuthology.misc.health.smithi001.stderr:created 2024-03-25T18:25:50.510519+0000 2024-03-25T18:31:45.372 INFO:teuthology.misc.health.smithi001.stderr:min_mon_release 19 (squid) 2024-03-25T18:31:45.372 INFO:teuthology.misc.health.smithi001.stderr:election_strategy: 3 2024-03-25T18:31:45.372 INFO:teuthology.misc.health.smithi001.stderr:0: v1:172.21.15.1:6789/0 mon.a 2024-03-25T18:31:45.372 INFO:teuthology.misc.health.smithi001.stderr:1: v1:172.21.15.115:6789/0 mon.b 2024-03-25T18:31:45.372 INFO:teuthology.misc.health.smithi001.stderr:2: v1:172.21.15.1:6790/0 mon.c 2024-03-25T18:31:45.372 INFO:teuthology.misc.health.smithi001.stderr: 2024-03-25T18:31:45.372 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:31:45.369+0000 7fe94affd640 4 set_mon_vals no callback set 2024-03-25T18:31:45.372 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:31:45.370+0000 7fe9497fa640 1 -- 172.21.15.1:0/2469895755 <== mon.1 v1:172.21.15.115:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 21787975 0 0) 0x7fe944019f00 con 0x7fe94c0b4e70 2024-03-25T18:31:45.373 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:31:45.371+0000 7fe94bfff640 10 monclient: get_auth_request con 0x7fe92c06ded0 auth_method 0 2024-03-25T18:31:45.373 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:31:45.371+0000 7fe9497fa640 10 client.4447.objecter ms_handle_connect 0x7fe92c06ded0 2024-03-25T18:31:45.377 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:31:45.376+0000 7fe9497fa640 1 -- 172.21.15.1:0/2469895755 <== mon.1 v1:172.21.15.115:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 996254230 0 0) 0x7fe9440539a0 con 0x7fe94c0b4e70 2024-03-25T18:31:45.377 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:31:45.376+0000 7fe9497fa640 10 client.4447.objecter ms_dispatch 0x7fe94c000b90 osd_map(10..10 src has 1..10) v4 2024-03-25T18:31:45.377 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:31:45.376+0000 7fe9497fa640 3 client.4447.objecter handle_osd_map got epochs [10,10] > 0 2024-03-25T18:31:45.378 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:31:45.376+0000 7fe9497fa640 3 client.4447.objecter handle_osd_map decoding full epoch 10 2024-03-25T18:31:45.378 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:31:45.376+0000 7fe9497fa640 20 client.4447.objecter dump_active .. 0 homeless 2024-03-25T18:31:45.379 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:31:45.377+0000 7fe9497fa640 1 -- 172.21.15.1:0/2469895755 <== mon.1 v1:172.21.15.115:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7fe94404f4b0 con 0x7fe94c0b4e70 2024-03-25T18:31:45.379 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:31:45.377+0000 7fe9497fa640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:31:45.379 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:31:45.377+0000 7fe9497fa640 10 monclient: _finish_command 1 = system:0 2024-03-25T18:31:45.638 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:31:45.636+0000 7fe9526a8640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:31:45.638 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:31:45.636+0000 7fe9526a8640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:31:45.638 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:31:45.636+0000 7fe9526a8640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.115:6789/0 2024-03-25T18:31:45.638 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:31:45.636+0000 7fe9526a8640 1 -- 172.21.15.1:0/2469895755 --> v1:172.21.15.115:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fe94c0aa090 con 0x7fe94c0b4e70 2024-03-25T18:31:45.670 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:31:45.668+0000 7fe9497fa640 1 -- 172.21.15.1:0/2469895755 <== mon.1 v1:172.21.15.115:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 4092213348) 0x7fe9440513e0 con 0x7fe94c0b4e70 2024-03-25T18:31:45.670 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:31:45.668+0000 7fe9497fa640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:31:45.670 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:31:45.668+0000 7fe9497fa640 10 monclient: _finish_command 2 = system:0 2024-03-25T18:31:45.670 INFO:teuthology.misc.health.smithi001.stdout: 2024-03-25T18:31:45.670 INFO:teuthology.misc.health.smithi001.stdout:{"epoch":10,"fsid":"c8ea72cf-9e99-4383-9b52-6c23d365fdfc","created":"2024-03-25T18:26:38.856638+0000","modified":"2024-03-25T18:26:44.224208+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T18:26:44.224208+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"b9f34922-3a84-49be-8426-807c1eceaefc","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"42bf306f-55e3-4634-9874-8b4b73ac6644","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"292cb23a-18a9-4207-a7cc-4d5727feaec0","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"45e0e9ab-898d-4673-9590-b4117bc5e84a","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"831a5a09-73ab-4a61-9872-088e76bdbc66","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"88cd903d-3b55-4840-9396-6e3cfe005282","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"89db7355-bc9d-46f4-8ab0-e8d9e744e026","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"cac47bee-0914-4f08-896f-348838882866","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T18:31:45.672 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:31:45.670+0000 7fe9526a8640 10 librados: watch_flush enter 2024-03-25T18:31:45.672 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:31:45.670+0000 7fe9526a8640 10 librados: watch_flush exit 2024-03-25T18:31:45.672 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:31:45.670+0000 7fe9526a8640 20 client.4447.objecter shutdown clearing up homeless session... 2024-03-25T18:31:45.672 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:31:45.670+0000 7fe9526a8640 10 client.4447.objecter successfully canceled tick 2024-03-25T18:31:45.672 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:31:45.670+0000 7fe9526a8640 1 -- 172.21.15.1:0/2469895755 >> v1:172.21.15.115:6800/4237379176 conn(0x7fe92c06ded0 legacy=0x7fe92c070390 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:31:45.672 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:31:45.670+0000 7fe9526a8640 10 monclient: shutdown 2024-03-25T18:31:45.672 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:31:45.670+0000 7fe9526a8640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:31:45.672 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:31:45.670+0000 7fe9526a8640 1 -- 172.21.15.1:0/2469895755 >> v1:172.21.15.115:6789/0 conn(0x7fe94c0b4e70 legacy=0x7fe94c1d7df0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:31:45.672 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:31:45.670+0000 7fe9526a8640 1 -- 172.21.15.1:0/2469895755 shutdown_connections 2024-03-25T18:31:45.672 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:31:45.670+0000 7fe9526a8640 1 -- 172.21.15.1:0/2469895755 >> 172.21.15.1:0/2469895755 conn(0x7fe94c0617a0 msgr2=0x7fe94c13edc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:31:45.673 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:31:45.670+0000 7fe9526a8640 1 -- 172.21.15.1:0/2469895755 shutdown_connections 2024-03-25T18:31:45.673 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:31:45.670+0000 7fe9526a8640 1 -- 172.21.15.1:0/2469895755 wait complete. 2024-03-25T18:31:45.673 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:31:45.671+0000 7fe9526a8640 1 librados: shutdown 2024-03-25T18:31:45.684 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T18:31:52.490 DEBUG:teuthology.orchestra.run.smithi001:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T18:31:52.605 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:31:52.603+0000 7fc924dba640 10 monclient: get_monmap_and_config 2024-03-25T18:31:52.605 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:31:52.603+0000 7fc924dba640 10 monclient: build_initial_monmap 2024-03-25T18:31:52.605 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:31:52.603+0000 7fc924dba640 10 monclient: monmap: 2024-03-25T18:31:52.605 INFO:teuthology.misc.health.smithi001.stderr:epoch 0 2024-03-25T18:31:52.605 INFO:teuthology.misc.health.smithi001.stderr:fsid c8ea72cf-9e99-4383-9b52-6c23d365fdfc 2024-03-25T18:31:52.605 INFO:teuthology.misc.health.smithi001.stderr:last_changed 2024-03-25T18:31:52.604868+0000 2024-03-25T18:31:52.605 INFO:teuthology.misc.health.smithi001.stderr:created 2024-03-25T18:31:52.604868+0000 2024-03-25T18:31:52.605 INFO:teuthology.misc.health.smithi001.stderr:min_mon_release 0 (unknown) 2024-03-25T18:31:52.605 INFO:teuthology.misc.health.smithi001.stderr:election_strategy: 3 2024-03-25T18:31:52.606 INFO:teuthology.misc.health.smithi001.stderr:0: v1:172.21.15.1:6789/0 mon.noname-a 2024-03-25T18:31:52.606 INFO:teuthology.misc.health.smithi001.stderr:1: v1:172.21.15.115:6789/0 mon.noname-c 2024-03-25T18:31:52.606 INFO:teuthology.misc.health.smithi001.stderr:2: v1:172.21.15.1:6790/0 mon.noname-b 2024-03-25T18:31:52.606 INFO:teuthology.misc.health.smithi001.stderr: 2024-03-25T18:31:52.607 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:31:52.603+0000 7fc924dba640 1 Processor -- start 2024-03-25T18:31:52.607 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:31:52.604+0000 7fc924dba640 1 -- start start 2024-03-25T18:31:52.607 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:31:52.604+0000 7fc924dba640 10 monclient: init 2024-03-25T18:31:52.607 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:31:52.604+0000 7fc924dba640 10 monclient: _reopen_session rank -1 2024-03-25T18:31:52.607 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:31:52.604+0000 7fc924dba640 10 monclient: _add_conns ranks=[0,2,1] 2024-03-25T18:31:52.607 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:31:52.604+0000 7fc924dba640 10 monclient(hunting): picked mon.noname-a con 0x7fc9200b4ff0 addr v1:172.21.15.1:6789/0 2024-03-25T18:31:52.607 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:31:52.604+0000 7fc924dba640 10 monclient(hunting): picked mon.noname-b con 0x7fc9200b4670 addr v1:172.21.15.1:6790/0 2024-03-25T18:31:52.607 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:31:52.604+0000 7fc924dba640 10 monclient(hunting): picked mon.noname-c con 0x7fc92013c480 addr v1:172.21.15.115:6789/0 2024-03-25T18:31:52.608 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:31:52.604+0000 7fc924dba640 1 -- --> v1:172.21.15.1:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc9200ab0e0 con 0x7fc9200b4ff0 2024-03-25T18:31:52.608 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:31:52.604+0000 7fc924dba640 1 -- --> v1:172.21.15.115:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc9200ab3c0 con 0x7fc92013c480 2024-03-25T18:31:52.608 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:31:52.604+0000 7fc924dba640 1 -- --> v1:172.21.15.1:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc9200a9f30 con 0x7fc9200b4670 2024-03-25T18:31:52.608 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:31:52.604+0000 7fc924dba640 10 monclient(hunting): _renew_subs 2024-03-25T18:31:52.608 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:31:52.604+0000 7fc924dba640 10 monclient(hunting): authenticate will time out at 7793.075524s 2024-03-25T18:31:52.620 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:31:52.618+0000 7fc91e59c640 1 --1- >> v1:172.21.15.1:6789/0 conn(0x7fc9200b4ff0 0x7fc92013bd70 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.1:6789/0 says I am v1:172.21.15.1:50890/0 (socket says 172.21.15.1:50890) 2024-03-25T18:31:52.620 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:31:52.618+0000 7fc91e59c640 1 -- 172.21.15.1:0/1642702718 learned_addr learned my addr 172.21.15.1:0/1642702718 (peer_addr_for_me v1:172.21.15.1:0/0) 2024-03-25T18:31:52.641 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:31:52.639+0000 7fc91d59a640 1 -- 172.21.15.1:0/1642702718 <== mon.0 v1:172.21.15.1:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 725648988 0 0) 0x7fc9200ab0e0 con 0x7fc9200b4ff0 2024-03-25T18:31:52.641 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:31:52.639+0000 7fc91d59a640 10 monclient(hunting): _negotiate 2024-03-25T18:31:52.641 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:31:52.639+0000 7fc91d59a640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:31:52.641 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:31:52.639+0000 7fc91d59a640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:31:52.641 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:31:52.639+0000 7fc91d59a640 10 monclient(hunting): my global_id is 4440 2024-03-25T18:31:52.641 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:31:52.639+0000 7fc91d59a640 1 -- 172.21.15.1:0/1642702718 --> v1:172.21.15.1:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fc908003520 con 0x7fc9200b4ff0 2024-03-25T18:31:52.641 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:31:52.639+0000 7fc91d59a640 1 -- 172.21.15.1:0/1642702718 <== mon.1 v1:172.21.15.115:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1316833790 0 0) 0x7fc9200ab3c0 con 0x7fc92013c480 2024-03-25T18:31:52.641 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:31:52.639+0000 7fc91d59a640 10 monclient(hunting): _negotiate 2024-03-25T18:31:52.641 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:31:52.639+0000 7fc91d59a640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:31:52.641 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:31:52.639+0000 7fc91d59a640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:31:52.641 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:31:52.639+0000 7fc91d59a640 10 monclient(hunting): my global_id is 4450 2024-03-25T18:31:52.642 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:31:52.639+0000 7fc91d59a640 1 -- 172.21.15.1:0/1642702718 --> v1:172.21.15.115:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fc9200ab0e0 con 0x7fc92013c480 2024-03-25T18:31:52.651 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:31:52.648+0000 7fc91d59a640 1 -- 172.21.15.1:0/1642702718 <== mon.1 v1:172.21.15.115:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 392066641 0 0) 0x7fc9200ab0e0 con 0x7fc92013c480 2024-03-25T18:31:52.651 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:31:52.648+0000 7fc91d59a640 1 -- 172.21.15.1:0/1642702718 >> v1:172.21.15.1:6790/0 conn(0x7fc9200b4670 legacy=0x7fc9201425c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:31:52.651 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:31:52.649+0000 7fc91d59a640 1 -- 172.21.15.1:0/1642702718 >> v1:172.21.15.1:6789/0 conn(0x7fc9200b4ff0 legacy=0x7fc92013bd70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:31:52.651 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:31:52.649+0000 7fc91d59a640 10 monclient: _finish_hunting 0 2024-03-25T18:31:52.651 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:31:52.649+0000 7fc91d59a640 1 monclient: found mon.noname-c 2024-03-25T18:31:52.651 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:31:52.649+0000 7fc91d59a640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:31:52.651 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:31:52.649+0000 7fc91d59a640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.115:6789/0 2024-03-25T18:31:52.651 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:31:52.649+0000 7fc91d59a640 1 -- 172.21.15.1:0/1642702718 --> v1:172.21.15.115:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc9200a9b70 con 0x7fc92013c480 2024-03-25T18:31:52.652 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:31:52.649+0000 7fc91d59a640 10 monclient: _finish_auth 0 2024-03-25T18:31:52.652 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:31:52.649+0000 7fc91d59a640 10 monclient: _check_auth_tickets 2024-03-25T18:31:52.652 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:31:52.649+0000 7fc91d59a640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:31:52.652 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:31:52.649+0000 7fc91d59a640 1 -- 172.21.15.1:0/1642702718 <== mon.1 v1:172.21.15.115:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1586965025 0 0) 0x7fc9180019f0 con 0x7fc92013c480 2024-03-25T18:31:52.652 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:31:52.649+0000 7fc924dba640 5 monclient: authenticate success, global_id 4450 2024-03-25T18:31:52.652 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:31:52.649+0000 7fc924dba640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T18:31:52.652 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:31:52.649+0000 7fc91d59a640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:31:52.652 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:31:52.649+0000 7fc91d59a640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-03-25T18:31:52.652 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:31:52.649+0000 7fc91d59a640 10 monclient: dump: 2024-03-25T18:31:52.652 INFO:teuthology.misc.health.smithi001.stderr:epoch 1 2024-03-25T18:31:52.652 INFO:teuthology.misc.health.smithi001.stderr:fsid c8ea72cf-9e99-4383-9b52-6c23d365fdfc 2024-03-25T18:31:52.652 INFO:teuthology.misc.health.smithi001.stderr:last_changed 2024-03-25T18:25:50.510519+0000 2024-03-25T18:31:52.653 INFO:teuthology.misc.health.smithi001.stderr:created 2024-03-25T18:25:50.510519+0000 2024-03-25T18:31:52.653 INFO:teuthology.misc.health.smithi001.stderr:min_mon_release 19 (squid) 2024-03-25T18:31:52.653 INFO:teuthology.misc.health.smithi001.stderr:election_strategy: 3 2024-03-25T18:31:52.653 INFO:teuthology.misc.health.smithi001.stderr:0: v1:172.21.15.1:6789/0 mon.a 2024-03-25T18:31:52.653 INFO:teuthology.misc.health.smithi001.stderr:1: v1:172.21.15.115:6789/0 mon.b 2024-03-25T18:31:52.653 INFO:teuthology.misc.health.smithi001.stderr:2: v1:172.21.15.1:6790/0 mon.c 2024-03-25T18:31:52.653 INFO:teuthology.misc.health.smithi001.stderr: 2024-03-25T18:31:52.653 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:31:52.649+0000 7fc924dba640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T18:31:52.680 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:31:52.678+0000 7fc91d59a640 1 -- 172.21.15.1:0/1642702718 <== mon.1 v1:172.21.15.115:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fc918001b50 con 0x7fc92013c480 2024-03-25T18:31:52.680 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:31:52.678+0000 7fc91d59a640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:31:52.680 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:31:52.678+0000 7fc91d59a640 1 -- 172.21.15.1:0/1642702718 <== mon.1 v1:172.21.15.115:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1586965025 0 0) 0x7fc918001f20 con 0x7fc92013c480 2024-03-25T18:31:52.680 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:31:52.678+0000 7fc924dba640 10 monclient: get_monmap_and_config success 2024-03-25T18:31:52.680 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:31:52.678+0000 7fc924dba640 4 set_mon_vals no callback set 2024-03-25T18:31:52.680 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:31:52.678+0000 7fc91d59a640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:31:52.680 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:31:52.678+0000 7fc91d59a640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-25T18:31:52.680 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:31:52.678+0000 7fc91d59a640 10 monclient: dump: 2024-03-25T18:31:52.680 INFO:teuthology.misc.health.smithi001.stderr:epoch 1 2024-03-25T18:31:52.680 INFO:teuthology.misc.health.smithi001.stderr:fsid c8ea72cf-9e99-4383-9b52-6c23d365fdfc 2024-03-25T18:31:52.680 INFO:teuthology.misc.health.smithi001.stderr:last_changed 2024-03-25T18:25:50.510519+0000 2024-03-25T18:31:52.681 INFO:teuthology.misc.health.smithi001.stderr:created 2024-03-25T18:25:50.510519+0000 2024-03-25T18:31:52.681 INFO:teuthology.misc.health.smithi001.stderr:min_mon_release 19 (squid) 2024-03-25T18:31:52.681 INFO:teuthology.misc.health.smithi001.stderr:election_strategy: 3 2024-03-25T18:31:52.681 INFO:teuthology.misc.health.smithi001.stderr:0: v1:172.21.15.1:6789/0 mon.a 2024-03-25T18:31:52.681 INFO:teuthology.misc.health.smithi001.stderr:1: v1:172.21.15.115:6789/0 mon.b 2024-03-25T18:31:52.681 INFO:teuthology.misc.health.smithi001.stderr:2: v1:172.21.15.1:6790/0 mon.c 2024-03-25T18:31:52.681 INFO:teuthology.misc.health.smithi001.stderr: 2024-03-25T18:31:52.682 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:31:52.678+0000 7fc924dba640 10 monclient: shutdown 2024-03-25T18:31:52.682 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:31:52.678+0000 7fc924dba640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:31:52.683 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:31:52.679+0000 7fc924dba640 1 -- 172.21.15.1:0/1642702718 >> v1:172.21.15.115:6789/0 conn(0x7fc92013c480 legacy=0x7fc92014dc00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:31:52.683 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:31:52.679+0000 7fc924dba640 1 -- 172.21.15.1:0/1642702718 shutdown_connections 2024-03-25T18:31:52.683 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:31:52.679+0000 7fc924dba640 1 -- 172.21.15.1:0/1642702718 >> 172.21.15.1:0/1642702718 conn(0x7fc9200617a0 msgr2=0x7fc920061bb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:31:52.683 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:31:52.679+0000 7fc924dba640 1 -- 172.21.15.1:0/1642702718 shutdown_connections 2024-03-25T18:31:52.683 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:31:52.679+0000 7fc924dba640 1 -- 172.21.15.1:0/1642702718 wait complete. 2024-03-25T18:31:52.683 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:31:52.680+0000 7fc924dba640 10 monclient: build_initial_monmap 2024-03-25T18:31:52.683 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:31:52.680+0000 7fc924dba640 10 monclient: monmap: 2024-03-25T18:31:52.683 INFO:teuthology.misc.health.smithi001.stderr:epoch 0 2024-03-25T18:31:52.683 INFO:teuthology.misc.health.smithi001.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T18:31:52.683 INFO:teuthology.misc.health.smithi001.stderr:last_changed 0.000000 2024-03-25T18:31:52.683 INFO:teuthology.misc.health.smithi001.stderr:created 0.000000 2024-03-25T18:31:52.683 INFO:teuthology.misc.health.smithi001.stderr:min_mon_release 0 (unknown) 2024-03-25T18:31:52.684 INFO:teuthology.misc.health.smithi001.stderr:election_strategy: 1 2024-03-25T18:31:52.684 INFO:teuthology.misc.health.smithi001.stderr:0: v1:172.21.15.1:6789/0 mon.noname-a 2024-03-25T18:31:52.684 INFO:teuthology.misc.health.smithi001.stderr:1: v1:172.21.15.115:6789/0 mon.noname-b 2024-03-25T18:31:52.684 INFO:teuthology.misc.health.smithi001.stderr:2: v1:172.21.15.1:6790/0 mon.noname-c 2024-03-25T18:31:52.684 INFO:teuthology.misc.health.smithi001.stderr: 2024-03-25T18:31:52.684 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:31:52.680+0000 7fc924dba640 1 librados: starting msgr at 2024-03-25T18:31:52.684 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:31:52.680+0000 7fc924dba640 1 librados: starting objecter 2024-03-25T18:31:52.684 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:31:52.681+0000 7fc924dba640 1 Processor -- start 2024-03-25T18:31:52.684 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:31:52.681+0000 7fc924dba640 1 -- start start 2024-03-25T18:31:52.684 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:31:52.681+0000 7fc924dba640 1 librados: setting wanted keys 2024-03-25T18:31:52.684 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:31:52.681+0000 7fc924dba640 1 librados: calling monclient init 2024-03-25T18:31:52.684 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:31:52.681+0000 7fc924dba640 10 monclient: init 2024-03-25T18:31:52.686 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:31:52.681+0000 7fc924dba640 10 monclient: _reopen_session rank -1 2024-03-25T18:31:52.686 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:31:52.681+0000 7fc924dba640 10 monclient: _add_conns ranks=[2,1,0] 2024-03-25T18:31:52.686 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:31:52.681+0000 7fc924dba640 10 monclient(hunting): picked mon.noname-c con 0x7fc9200b4670 addr v1:172.21.15.1:6790/0 2024-03-25T18:31:52.686 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:31:52.681+0000 7fc924dba640 10 monclient(hunting): picked mon.noname-b con 0x7fc9200b4ff0 addr v1:172.21.15.115:6789/0 2024-03-25T18:31:52.686 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:31:52.681+0000 7fc924dba640 10 monclient(hunting): picked mon.noname-a con 0x7fc92013c480 addr v1:172.21.15.1:6789/0 2024-03-25T18:31:52.686 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:31:52.681+0000 7fc924dba640 1 -- --> v1:172.21.15.1:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc920144870 con 0x7fc92013c480 2024-03-25T18:31:52.686 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:31:52.681+0000 7fc924dba640 1 -- --> v1:172.21.15.115:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc920144ae0 con 0x7fc9200b4ff0 2024-03-25T18:31:52.686 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:31:52.681+0000 7fc924dba640 1 -- --> v1:172.21.15.1:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc92013ec30 con 0x7fc9200b4670 2024-03-25T18:31:52.686 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:31:52.682+0000 7fc924dba640 10 monclient(hunting): _renew_subs 2024-03-25T18:31:52.687 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:31:52.682+0000 7fc924dba640 10 monclient(hunting): authenticate will time out at 7793.152668s 2024-03-25T18:31:52.695 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:31:52.693+0000 7fc91dd9b640 1 --1- >> v1:172.21.15.115:6789/0 conn(0x7fc9200b4ff0 0x7fc92013d300 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.115:6789/0 says I am v1:172.21.15.1:50532/0 (socket says 172.21.15.1:50532) 2024-03-25T18:31:52.695 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:31:52.693+0000 7fc91dd9b640 1 -- 172.21.15.1:0/2659080491 learned_addr learned my addr 172.21.15.1:0/2659080491 (peer_addr_for_me v1:172.21.15.1:0/0) 2024-03-25T18:31:52.698 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:31:52.696+0000 7fc90effd640 10 client.?.objecter ms_handle_connect 0x7fc9200b4ff0 2024-03-25T18:31:52.698 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:31:52.696+0000 7fc90effd640 10 client.?.objecter resend_mon_ops 2024-03-25T18:31:52.701 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:31:52.699+0000 7fc90effd640 10 client.?.objecter ms_handle_connect 0x7fc9200b4670 2024-03-25T18:31:52.701 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:31:52.699+0000 7fc90effd640 10 client.?.objecter resend_mon_ops 2024-03-25T18:31:52.709 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:31:52.707+0000 7fc90effd640 1 -- 172.21.15.1:0/2659080491 <== mon.1 v1:172.21.15.115:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1158386242 0 0) 0x7fc920144ae0 con 0x7fc9200b4ff0 2024-03-25T18:31:52.709 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:31:52.707+0000 7fc90effd640 10 monclient(hunting): _negotiate 2024-03-25T18:31:52.709 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:31:52.707+0000 7fc90effd640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:31:52.709 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:31:52.707+0000 7fc90effd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:31:52.710 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:31:52.707+0000 7fc90effd640 10 monclient(hunting): my global_id is 4453 2024-03-25T18:31:52.710 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:31:52.707+0000 7fc90effd640 1 -- 172.21.15.1:0/2659080491 --> v1:172.21.15.115:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fc900004eb0 con 0x7fc9200b4ff0 2024-03-25T18:31:52.713 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:31:52.711+0000 7fc90effd640 1 -- 172.21.15.1:0/2659080491 <== mon.2 v1:172.21.15.1:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1566599131 0 0) 0x7fc92013ec30 con 0x7fc9200b4670 2024-03-25T18:31:52.713 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:31:52.711+0000 7fc90effd640 10 monclient(hunting): _negotiate 2024-03-25T18:31:52.713 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:31:52.711+0000 7fc90effd640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:31:52.713 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:31:52.711+0000 7fc90effd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:31:52.713 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:31:52.711+0000 7fc90effd640 10 monclient(hunting): my global_id is 4454 2024-03-25T18:31:52.713 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:31:52.711+0000 7fc90effd640 1 -- 172.21.15.1:0/2659080491 --> v1:172.21.15.1:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fc920144ae0 con 0x7fc9200b4670 2024-03-25T18:31:52.713 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:31:52.711+0000 7fc90effd640 10 client.?.objecter ms_handle_connect 0x7fc92013c480 2024-03-25T18:31:52.713 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:31:52.711+0000 7fc90effd640 10 client.?.objecter resend_mon_ops 2024-03-25T18:31:52.730 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:31:52.728+0000 7fc90effd640 1 -- 172.21.15.1:0/2659080491 <== mon.1 v1:172.21.15.115:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 548085862 0 0) 0x7fc900004eb0 con 0x7fc9200b4ff0 2024-03-25T18:31:52.730 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:31:52.728+0000 7fc90effd640 1 -- 172.21.15.1:0/2659080491 --> v1:172.21.15.115:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fc92013ec30 con 0x7fc9200b4ff0 2024-03-25T18:31:52.730 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:31:52.728+0000 7fc90effd640 1 -- 172.21.15.1:0/2659080491 <== mon.1 v1:172.21.15.115:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1586965025 0 0) 0x7fc9140019d0 con 0x7fc9200b4ff0 2024-03-25T18:31:52.730 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:31:52.728+0000 7fc90effd640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:31:52.730 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:31:52.728+0000 7fc90effd640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-03-25T18:31:52.730 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:31:52.728+0000 7fc90effd640 10 monclient(hunting): dump: 2024-03-25T18:31:52.730 INFO:teuthology.misc.health.smithi001.stderr:epoch 1 2024-03-25T18:31:52.730 INFO:teuthology.misc.health.smithi001.stderr:fsid c8ea72cf-9e99-4383-9b52-6c23d365fdfc 2024-03-25T18:31:52.730 INFO:teuthology.misc.health.smithi001.stderr:last_changed 2024-03-25T18:25:50.510519+0000 2024-03-25T18:31:52.731 INFO:teuthology.misc.health.smithi001.stderr:created 2024-03-25T18:25:50.510519+0000 2024-03-25T18:31:52.731 INFO:teuthology.misc.health.smithi001.stderr:min_mon_release 19 (squid) 2024-03-25T18:31:52.731 INFO:teuthology.misc.health.smithi001.stderr:election_strategy: 3 2024-03-25T18:31:52.731 INFO:teuthology.misc.health.smithi001.stderr:0: v1:172.21.15.1:6789/0 mon.a 2024-03-25T18:31:52.731 INFO:teuthology.misc.health.smithi001.stderr:1: v1:172.21.15.115:6789/0 mon.b 2024-03-25T18:31:52.731 INFO:teuthology.misc.health.smithi001.stderr:2: v1:172.21.15.1:6790/0 mon.c 2024-03-25T18:31:52.731 INFO:teuthology.misc.health.smithi001.stderr: 2024-03-25T18:31:52.731 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:31:52.728+0000 7fc90effd640 10 monclient(hunting): _finish_auth 0 2024-03-25T18:31:52.735 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:31:52.733+0000 7fc90effd640 1 -- 172.21.15.1:0/2659080491 <== mon.2 v1:172.21.15.1:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2203471683 0 0) 0x7fc920144ae0 con 0x7fc9200b4670 2024-03-25T18:31:52.735 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:31:52.733+0000 7fc90effd640 1 -- 172.21.15.1:0/2659080491 --> v1:172.21.15.1:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fc900004eb0 con 0x7fc9200b4670 2024-03-25T18:31:52.735 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:31:52.733+0000 7fc90effd640 1 -- 172.21.15.1:0/2659080491 <== mon.2 v1:172.21.15.1:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1586965025 0 0) 0x7fc910001950 con 0x7fc9200b4670 2024-03-25T18:31:52.735 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:31:52.733+0000 7fc90effd640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:31:52.735 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:31:52.733+0000 7fc90effd640 10 monclient(hunting): got monmap 1 from mon.c (according to old e1) 2024-03-25T18:31:52.735 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:31:52.733+0000 7fc90effd640 10 monclient(hunting): dump: 2024-03-25T18:31:52.735 INFO:teuthology.misc.health.smithi001.stderr:epoch 1 2024-03-25T18:31:52.735 INFO:teuthology.misc.health.smithi001.stderr:fsid c8ea72cf-9e99-4383-9b52-6c23d365fdfc 2024-03-25T18:31:52.735 INFO:teuthology.misc.health.smithi001.stderr:last_changed 2024-03-25T18:25:50.510519+0000 2024-03-25T18:31:52.735 INFO:teuthology.misc.health.smithi001.stderr:created 2024-03-25T18:25:50.510519+0000 2024-03-25T18:31:52.736 INFO:teuthology.misc.health.smithi001.stderr:min_mon_release 19 (squid) 2024-03-25T18:31:52.736 INFO:teuthology.misc.health.smithi001.stderr:election_strategy: 3 2024-03-25T18:31:52.736 INFO:teuthology.misc.health.smithi001.stderr:0: v1:172.21.15.1:6789/0 mon.a 2024-03-25T18:31:52.736 INFO:teuthology.misc.health.smithi001.stderr:1: v1:172.21.15.115:6789/0 mon.b 2024-03-25T18:31:52.736 INFO:teuthology.misc.health.smithi001.stderr:2: v1:172.21.15.1:6790/0 mon.c 2024-03-25T18:31:52.736 INFO:teuthology.misc.health.smithi001.stderr: 2024-03-25T18:31:52.742 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:31:52.740+0000 7fc90effd640 1 -- 172.21.15.1:0/2659080491 <== mon.2 v1:172.21.15.1:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 1003671665 0 0) 0x7fc900004eb0 con 0x7fc9200b4670 2024-03-25T18:31:52.742 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:31:52.740+0000 7fc90effd640 1 -- 172.21.15.1:0/2659080491 >> v1:172.21.15.115:6789/0 conn(0x7fc9200b4ff0 legacy=0x7fc92013d300 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:31:52.742 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:31:52.740+0000 7fc90effd640 1 -- 172.21.15.1:0/2659080491 >> v1:172.21.15.1:6789/0 conn(0x7fc92013c480 legacy=0x7fc9201e2380 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:31:52.742 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:31:52.740+0000 7fc90effd640 10 monclient: _finish_hunting 0 2024-03-25T18:31:52.742 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:31:52.740+0000 7fc90effd640 1 monclient: found mon.c 2024-03-25T18:31:52.742 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:31:52.740+0000 7fc90effd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:31:52.743 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:31:52.740+0000 7fc90effd640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.1:6790/0 2024-03-25T18:31:52.743 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:31:52.740+0000 7fc90effd640 1 -- 172.21.15.1:0/2659080491 --> v1:172.21.15.1:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc920143680 con 0x7fc9200b4670 2024-03-25T18:31:52.743 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:31:52.740+0000 7fc90effd640 10 monclient: _finish_auth 0 2024-03-25T18:31:52.743 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:31:52.740+0000 7fc90effd640 10 monclient: _check_auth_tickets 2024-03-25T18:31:52.743 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:31:52.740+0000 7fc90effd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:31:52.743 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:31:52.740+0000 7fc924dba640 5 monclient: authenticate success, global_id 4454 2024-03-25T18:31:52.743 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:31:52.740+0000 7fc924dba640 10 monclient: _renew_subs 2024-03-25T18:31:52.743 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:31:52.740+0000 7fc924dba640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.1:6790/0 2024-03-25T18:31:52.743 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:31:52.740+0000 7fc924dba640 1 -- 172.21.15.1:0/2659080491 --> v1:172.21.15.1:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc920143930 con 0x7fc9200b4670 2024-03-25T18:31:52.743 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:31:52.740+0000 7fc924dba640 10 client.4454.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T18:31:52.743 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:31:52.740+0000 7fc924dba640 10 monclient: _renew_subs 2024-03-25T18:31:52.743 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:31:52.740+0000 7fc924dba640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.1:6790/0 2024-03-25T18:31:52.743 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:31:52.740+0000 7fc924dba640 1 -- 172.21.15.1:0/2659080491 --> v1:172.21.15.1:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7fc9201460f0 con 0x7fc9200b4670 2024-03-25T18:31:52.743 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:31:52.740+0000 7fc924dba640 1 librados: init done 2024-03-25T18:31:52.744 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:31:52.741+0000 7fc924dba640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T18:31:52.744 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:31:52.741+0000 7fc924dba640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:31:52.744 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:31:52.741+0000 7fc924dba640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.1:6790/0 2024-03-25T18:31:52.744 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:31:52.741+0000 7fc924dba640 1 -- 172.21.15.1:0/2659080491 --> v1:172.21.15.1:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc92006dbd0 con 0x7fc9200b4670 2024-03-25T18:31:53.046 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:31:53.044+0000 7fc90effd640 1 -- 172.21.15.1:0/2659080491 <== mon.2 v1:172.21.15.1:6790/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fc910001d60 con 0x7fc9200b4670 2024-03-25T18:31:53.046 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:31:53.044+0000 7fc90effd640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:31:53.046 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:31:53.044+0000 7fc90effd640 1 -- 172.21.15.1:0/2659080491 <== mon.2 v1:172.21.15.1:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1586965025 0 0) 0x7fc910004a00 con 0x7fc9200b4670 2024-03-25T18:31:53.046 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:31:53.044+0000 7fc90effd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:31:53.046 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:31:53.044+0000 7fc90effd640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-25T18:31:53.046 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:31:53.044+0000 7fc90effd640 10 monclient: dump: 2024-03-25T18:31:53.046 INFO:teuthology.misc.health.smithi001.stderr:epoch 1 2024-03-25T18:31:53.047 INFO:teuthology.misc.health.smithi001.stderr:fsid c8ea72cf-9e99-4383-9b52-6c23d365fdfc 2024-03-25T18:31:53.047 INFO:teuthology.misc.health.smithi001.stderr:last_changed 2024-03-25T18:25:50.510519+0000 2024-03-25T18:31:53.047 INFO:teuthology.misc.health.smithi001.stderr:created 2024-03-25T18:25:50.510519+0000 2024-03-25T18:31:53.047 INFO:teuthology.misc.health.smithi001.stderr:min_mon_release 19 (squid) 2024-03-25T18:31:53.047 INFO:teuthology.misc.health.smithi001.stderr:election_strategy: 3 2024-03-25T18:31:53.047 INFO:teuthology.misc.health.smithi001.stderr:0: v1:172.21.15.1:6789/0 mon.a 2024-03-25T18:31:53.047 INFO:teuthology.misc.health.smithi001.stderr:1: v1:172.21.15.115:6789/0 mon.b 2024-03-25T18:31:53.047 INFO:teuthology.misc.health.smithi001.stderr:2: v1:172.21.15.1:6790/0 mon.c 2024-03-25T18:31:53.047 INFO:teuthology.misc.health.smithi001.stderr: 2024-03-25T18:31:53.047 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:31:53.044+0000 7fc91cd99640 4 set_mon_vals no callback set 2024-03-25T18:31:53.048 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:31:53.045+0000 7fc90effd640 1 -- 172.21.15.1:0/2659080491 <== mon.2 v1:172.21.15.1:6790/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 21787975 0 0) 0x7fc910004c20 con 0x7fc9200b4670 2024-03-25T18:31:53.048 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:31:53.045+0000 7fc91dd9b640 10 monclient: get_auth_request con 0x7fc90006de80 auth_method 0 2024-03-25T18:31:53.048 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:31:53.045+0000 7fc90effd640 10 client.4454.objecter ms_handle_connect 0x7fc90006de80 2024-03-25T18:31:53.054 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:31:53.052+0000 7fc90effd640 1 -- 172.21.15.1:0/2659080491 <== mon.2 v1:172.21.15.1:6790/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 996254230 0 0) 0x7fc910054800 con 0x7fc9200b4670 2024-03-25T18:31:53.054 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:31:53.052+0000 7fc90effd640 10 client.4454.objecter ms_dispatch 0x7fc920000b90 osd_map(10..10 src has 1..10) v4 2024-03-25T18:31:53.054 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:31:53.052+0000 7fc90effd640 3 client.4454.objecter handle_osd_map got epochs [10,10] > 0 2024-03-25T18:31:53.054 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:31:53.052+0000 7fc90effd640 3 client.4454.objecter handle_osd_map decoding full epoch 10 2024-03-25T18:31:53.054 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:31:53.052+0000 7fc90effd640 20 client.4454.objecter dump_active .. 0 homeless 2024-03-25T18:31:53.054 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:31:53.052+0000 7fc90effd640 1 -- 172.21.15.1:0/2659080491 <== mon.2 v1:172.21.15.1:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7fc910050140 con 0x7fc9200b4670 2024-03-25T18:31:53.054 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:31:53.052+0000 7fc90effd640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:31:53.054 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:31:53.052+0000 7fc90effd640 10 monclient: _finish_command 1 = system:0 2024-03-25T18:31:53.301 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:31:53.299+0000 7fc924dba640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:31:53.301 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:31:53.299+0000 7fc924dba640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:31:53.301 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:31:53.299+0000 7fc924dba640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.1:6790/0 2024-03-25T18:31:53.301 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:31:53.299+0000 7fc924dba640 1 -- 172.21.15.1:0/2659080491 --> v1:172.21.15.1:6790/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fc9200688c0 con 0x7fc9200b4670 2024-03-25T18:31:53.370 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:31:53.368+0000 7fc90effd640 1 -- 172.21.15.1:0/2659080491 <== mon.2 v1:172.21.15.1:6790/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 4092213348) 0x7fc910052240 con 0x7fc9200b4670 2024-03-25T18:31:53.370 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:31:53.368+0000 7fc90effd640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:31:53.370 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:31:53.368+0000 7fc90effd640 10 monclient: _finish_command 2 = system:0 2024-03-25T18:31:53.370 INFO:teuthology.misc.health.smithi001.stdout: 2024-03-25T18:31:53.371 INFO:teuthology.misc.health.smithi001.stdout:{"epoch":10,"fsid":"c8ea72cf-9e99-4383-9b52-6c23d365fdfc","created":"2024-03-25T18:26:38.856638+0000","modified":"2024-03-25T18:26:44.224208+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T18:26:44.224208+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"b9f34922-3a84-49be-8426-807c1eceaefc","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"42bf306f-55e3-4634-9874-8b4b73ac6644","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"292cb23a-18a9-4207-a7cc-4d5727feaec0","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"45e0e9ab-898d-4673-9590-b4117bc5e84a","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"831a5a09-73ab-4a61-9872-088e76bdbc66","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"88cd903d-3b55-4840-9396-6e3cfe005282","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"89db7355-bc9d-46f4-8ab0-e8d9e744e026","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"cac47bee-0914-4f08-896f-348838882866","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T18:31:53.373 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:31:53.370+0000 7fc924dba640 10 librados: watch_flush enter 2024-03-25T18:31:53.373 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:31:53.370+0000 7fc924dba640 10 librados: watch_flush exit 2024-03-25T18:31:53.373 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:31:53.370+0000 7fc924dba640 20 client.4454.objecter shutdown clearing up homeless session... 2024-03-25T18:31:53.373 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:31:53.370+0000 7fc924dba640 10 client.4454.objecter successfully canceled tick 2024-03-25T18:31:53.373 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:31:53.370+0000 7fc924dba640 1 -- 172.21.15.1:0/2659080491 >> v1:172.21.15.115:6800/4237379176 conn(0x7fc90006de80 legacy=0x7fc900070340 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:31:53.373 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:31:53.370+0000 7fc924dba640 10 monclient: shutdown 2024-03-25T18:31:53.373 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:31:53.370+0000 7fc924dba640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:31:53.373 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:31:53.370+0000 7fc924dba640 1 -- 172.21.15.1:0/2659080491 >> v1:172.21.15.1:6790/0 conn(0x7fc9200b4670 legacy=0x7fc92013cbf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:31:53.373 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:31:53.370+0000 7fc924dba640 1 -- 172.21.15.1:0/2659080491 shutdown_connections 2024-03-25T18:31:53.374 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:31:53.370+0000 7fc924dba640 1 -- 172.21.15.1:0/2659080491 >> 172.21.15.1:0/2659080491 conn(0x7fc9200617a0 msgr2=0x7fc9201405c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:31:53.374 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:31:53.371+0000 7fc924dba640 1 -- 172.21.15.1:0/2659080491 shutdown_connections 2024-03-25T18:31:53.374 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:31:53.371+0000 7fc924dba640 1 -- 172.21.15.1:0/2659080491 wait complete. 2024-03-25T18:31:53.374 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:31:53.371+0000 7fc924dba640 1 librados: shutdown 2024-03-25T18:31:53.385 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T18:31:56.898 INFO:tasks.ceph.mon.a.smithi001.stderr:==00:00:05:35.457 34876== Warning: unimplemented fcntl command: 1036 2024-03-25T18:31:56.904 INFO:tasks.ceph.mon.a.smithi001.stderr:==00:00:05:35.462 34876== Warning: unimplemented fcntl command: 1036 2024-03-25T18:31:56.938 INFO:tasks.ceph.mon.a.smithi001.stderr:==00:00:05:35.496 34876== Warning: unimplemented fcntl command: 1036 2024-03-25T18:31:56.949 INFO:tasks.ceph.mon.c.smithi001.stderr:==00:00:05:35.508 34877== Warning: unimplemented fcntl command: 1036 2024-03-25T18:31:56.955 INFO:tasks.ceph.mon.b.smithi115.stderr:==00:00:05:35.438 34836== Warning: unimplemented fcntl command: 1036 2024-03-25T18:31:56.959 INFO:tasks.ceph.mon.b.smithi115.stderr:==00:00:05:35.442 34836== Warning: unimplemented fcntl command: 1036 2024-03-25T18:31:56.960 INFO:tasks.ceph.mon.c.smithi001.stderr:==00:00:05:35.518 34877== Warning: unimplemented fcntl command: 1036 2024-03-25T18:31:56.985 INFO:tasks.ceph.mon.c.smithi001.stderr:==00:00:05:35.543 34877== Warning: unimplemented fcntl command: 1036 2024-03-25T18:31:56.987 INFO:tasks.ceph.mon.b.smithi115.stderr:==00:00:05:35.471 34836== Warning: unimplemented fcntl command: 1036 2024-03-25T18:32:00.192 DEBUG:teuthology.orchestra.run.smithi001:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T18:32:00.313 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:32:00.311+0000 7f629457b640 10 monclient: get_monmap_and_config 2024-03-25T18:32:00.313 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:32:00.312+0000 7f629457b640 10 monclient: build_initial_monmap 2024-03-25T18:32:00.313 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:32:00.312+0000 7f629457b640 10 monclient: monmap: 2024-03-25T18:32:00.313 INFO:teuthology.misc.health.smithi001.stderr:epoch 0 2024-03-25T18:32:00.313 INFO:teuthology.misc.health.smithi001.stderr:fsid c8ea72cf-9e99-4383-9b52-6c23d365fdfc 2024-03-25T18:32:00.313 INFO:teuthology.misc.health.smithi001.stderr:last_changed 2024-03-25T18:32:00.312590+0000 2024-03-25T18:32:00.313 INFO:teuthology.misc.health.smithi001.stderr:created 2024-03-25T18:32:00.312590+0000 2024-03-25T18:32:00.313 INFO:teuthology.misc.health.smithi001.stderr:min_mon_release 0 (unknown) 2024-03-25T18:32:00.313 INFO:teuthology.misc.health.smithi001.stderr:election_strategy: 3 2024-03-25T18:32:00.313 INFO:teuthology.misc.health.smithi001.stderr:0: v1:172.21.15.1:6789/0 mon.noname-a 2024-03-25T18:32:00.313 INFO:teuthology.misc.health.smithi001.stderr:1: v1:172.21.15.115:6789/0 mon.noname-c 2024-03-25T18:32:00.314 INFO:teuthology.misc.health.smithi001.stderr:2: v1:172.21.15.1:6790/0 mon.noname-b 2024-03-25T18:32:00.314 INFO:teuthology.misc.health.smithi001.stderr: 2024-03-25T18:32:00.315 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:32:00.312+0000 7f629457b640 1 Processor -- start 2024-03-25T18:32:00.315 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:32:00.312+0000 7f629457b640 1 -- start start 2024-03-25T18:32:00.315 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:32:00.312+0000 7f629457b640 10 monclient: init 2024-03-25T18:32:00.315 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:32:00.313+0000 7f629457b640 10 monclient: _reopen_session rank -1 2024-03-25T18:32:00.315 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:32:00.313+0000 7f629457b640 10 monclient: _add_conns ranks=[2,1,0] 2024-03-25T18:32:00.315 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:32:00.313+0000 7f629457b640 10 monclient(hunting): picked mon.noname-b con 0x7f628c0b4ff0 addr v1:172.21.15.1:6790/0 2024-03-25T18:32:00.315 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:32:00.313+0000 7f629457b640 10 monclient(hunting): picked mon.noname-c con 0x7f628c0b4670 addr v1:172.21.15.115:6789/0 2024-03-25T18:32:00.315 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:32:00.313+0000 7f629457b640 10 monclient(hunting): picked mon.noname-a con 0x7f628c13c480 addr v1:172.21.15.1:6789/0 2024-03-25T18:32:00.315 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:32:00.313+0000 7f629457b640 1 -- --> v1:172.21.15.1:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f628c0ab0e0 con 0x7f628c13c480 2024-03-25T18:32:00.315 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:32:00.313+0000 7f629457b640 1 -- --> v1:172.21.15.115:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f628c0ab3c0 con 0x7f628c0b4670 2024-03-25T18:32:00.315 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:32:00.313+0000 7f629457b640 1 -- --> v1:172.21.15.1:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f628c0a9f30 con 0x7f628c0b4ff0 2024-03-25T18:32:00.316 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:32:00.313+0000 7f629457b640 10 monclient(hunting): _renew_subs 2024-03-25T18:32:00.316 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:32:00.313+0000 7f629457b640 10 monclient(hunting): authenticate will time out at 7800.783480s 2024-03-25T18:32:00.324 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:32:00.323+0000 7f6291b16640 1 --1- >> v1:172.21.15.115:6789/0 conn(0x7f628c0b4670 0x7f628c1425c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.115:6789/0 says I am v1:172.21.15.1:54984/0 (socket says 172.21.15.1:54984) 2024-03-25T18:32:00.324 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:32:00.323+0000 7f6291b16640 1 -- 172.21.15.1:0/1383290471 learned_addr learned my addr 172.21.15.1:0/1383290471 (peer_addr_for_me v1:172.21.15.1:0/0) 2024-03-25T18:32:00.338 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:32:00.337+0000 7f6291315640 1 -- 172.21.15.1:0/1383290471 <== mon.0 v1:172.21.15.1:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1809471711 0 0) 0x7f628c0ab0e0 con 0x7f628c13c480 2024-03-25T18:32:00.338 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:32:00.337+0000 7f6291315640 10 monclient(hunting): _negotiate 2024-03-25T18:32:00.338 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:32:00.337+0000 7f6291315640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:32:00.338 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:32:00.337+0000 7f6291315640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:32:00.338 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:32:00.337+0000 7f6291315640 10 monclient(hunting): my global_id is 4446 2024-03-25T18:32:00.338 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:32:00.337+0000 7f6291315640 1 -- 172.21.15.1:0/1383290471 --> v1:172.21.15.1:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f6278003520 con 0x7f628c13c480 2024-03-25T18:32:00.341 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:32:00.340+0000 7f6291315640 1 -- 172.21.15.1:0/1383290471 <== mon.1 v1:172.21.15.115:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1780027821 0 0) 0x7f628c0ab3c0 con 0x7f628c0b4670 2024-03-25T18:32:00.341 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:32:00.340+0000 7f6291315640 10 monclient(hunting): _negotiate 2024-03-25T18:32:00.341 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:32:00.340+0000 7f6291315640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:32:00.341 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:32:00.340+0000 7f6291315640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:32:00.341 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:32:00.340+0000 7f6291315640 10 monclient(hunting): my global_id is 4456 2024-03-25T18:32:00.341 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:32:00.340+0000 7f6291315640 1 -- 172.21.15.1:0/1383290471 --> v1:172.21.15.115:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f628c0ab0e0 con 0x7f628c0b4670 2024-03-25T18:32:00.349 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:32:00.348+0000 7f6291315640 1 -- 172.21.15.1:0/1383290471 <== mon.0 v1:172.21.15.1:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3642485887 0 0) 0x7f6278003520 con 0x7f628c13c480 2024-03-25T18:32:00.349 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:32:00.348+0000 7f6291315640 1 -- 172.21.15.1:0/1383290471 >> v1:172.21.15.1:6790/0 conn(0x7f628c0b4ff0 legacy=0x7f628c13bd70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:32:00.349 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:32:00.348+0000 7f6291315640 1 -- 172.21.15.1:0/1383290471 >> v1:172.21.15.115:6789/0 conn(0x7f628c0b4670 legacy=0x7f628c1425c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:32:00.349 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:32:00.348+0000 7f6291315640 10 monclient: _finish_hunting 0 2024-03-25T18:32:00.349 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:32:00.348+0000 7f6291315640 1 monclient: found mon.noname-a 2024-03-25T18:32:00.349 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:32:00.348+0000 7f6291315640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:32:00.350 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:32:00.348+0000 7f6291315640 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.1:6789/0 2024-03-25T18:32:00.350 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:32:00.348+0000 7f6291315640 1 -- 172.21.15.1:0/1383290471 --> v1:172.21.15.1:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f628c0a9b70 con 0x7f628c13c480 2024-03-25T18:32:00.350 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:32:00.348+0000 7f6291315640 10 monclient: _finish_auth 0 2024-03-25T18:32:00.350 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:32:00.348+0000 7f6291315640 10 monclient: _check_auth_tickets 2024-03-25T18:32:00.350 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:32:00.348+0000 7f6291315640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:32:00.350 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:32:00.348+0000 7f6291315640 1 -- 172.21.15.1:0/1383290471 <== mon.0 v1:172.21.15.1:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1586965025 0 0) 0x7f6284001a10 con 0x7f628c13c480 2024-03-25T18:32:00.350 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:32:00.348+0000 7f6291315640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:32:00.350 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:32:00.348+0000 7f6291315640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-03-25T18:32:00.350 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:32:00.348+0000 7f6291315640 10 monclient: dump: 2024-03-25T18:32:00.350 INFO:teuthology.misc.health.smithi001.stderr:epoch 1 2024-03-25T18:32:00.350 INFO:teuthology.misc.health.smithi001.stderr:fsid c8ea72cf-9e99-4383-9b52-6c23d365fdfc 2024-03-25T18:32:00.350 INFO:teuthology.misc.health.smithi001.stderr:last_changed 2024-03-25T18:25:50.510519+0000 2024-03-25T18:32:00.350 INFO:teuthology.misc.health.smithi001.stderr:created 2024-03-25T18:25:50.510519+0000 2024-03-25T18:32:00.350 INFO:teuthology.misc.health.smithi001.stderr:min_mon_release 19 (squid) 2024-03-25T18:32:00.351 INFO:teuthology.misc.health.smithi001.stderr:election_strategy: 3 2024-03-25T18:32:00.351 INFO:teuthology.misc.health.smithi001.stderr:0: v1:172.21.15.1:6789/0 mon.a 2024-03-25T18:32:00.351 INFO:teuthology.misc.health.smithi001.stderr:1: v1:172.21.15.115:6789/0 mon.b 2024-03-25T18:32:00.351 INFO:teuthology.misc.health.smithi001.stderr:2: v1:172.21.15.1:6790/0 mon.c 2024-03-25T18:32:00.351 INFO:teuthology.misc.health.smithi001.stderr: 2024-03-25T18:32:00.351 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:32:00.348+0000 7f629457b640 5 monclient: authenticate success, global_id 4446 2024-03-25T18:32:00.351 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:32:00.348+0000 7f629457b640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T18:32:00.357 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:32:00.357+0000 7f6291315640 1 -- 172.21.15.1:0/1383290471 <== mon.0 v1:172.21.15.1:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f6284001b70 con 0x7f628c13c480 2024-03-25T18:32:00.358 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:32:00.357+0000 7f6291315640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:32:00.358 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:32:00.357+0000 7f6291315640 1 -- 172.21.15.1:0/1383290471 <== mon.0 v1:172.21.15.1:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1586965025 0 0) 0x7f6284001f80 con 0x7f628c13c480 2024-03-25T18:32:00.358 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:32:00.357+0000 7f629457b640 10 monclient: get_monmap_and_config success 2024-03-25T18:32:00.358 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:32:00.357+0000 7f629457b640 4 set_mon_vals no callback set 2024-03-25T18:32:00.358 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:32:00.357+0000 7f6291315640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:32:00.358 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:32:00.357+0000 7f6291315640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-25T18:32:00.358 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:32:00.357+0000 7f6291315640 10 monclient: dump: 2024-03-25T18:32:00.358 INFO:teuthology.misc.health.smithi001.stderr:epoch 1 2024-03-25T18:32:00.358 INFO:teuthology.misc.health.smithi001.stderr:fsid c8ea72cf-9e99-4383-9b52-6c23d365fdfc 2024-03-25T18:32:00.358 INFO:teuthology.misc.health.smithi001.stderr:last_changed 2024-03-25T18:25:50.510519+0000 2024-03-25T18:32:00.358 INFO:teuthology.misc.health.smithi001.stderr:created 2024-03-25T18:25:50.510519+0000 2024-03-25T18:32:00.358 INFO:teuthology.misc.health.smithi001.stderr:min_mon_release 19 (squid) 2024-03-25T18:32:00.358 INFO:teuthology.misc.health.smithi001.stderr:election_strategy: 3 2024-03-25T18:32:00.358 INFO:teuthology.misc.health.smithi001.stderr:0: v1:172.21.15.1:6789/0 mon.a 2024-03-25T18:32:00.359 INFO:teuthology.misc.health.smithi001.stderr:1: v1:172.21.15.115:6789/0 mon.b 2024-03-25T18:32:00.359 INFO:teuthology.misc.health.smithi001.stderr:2: v1:172.21.15.1:6790/0 mon.c 2024-03-25T18:32:00.359 INFO:teuthology.misc.health.smithi001.stderr: 2024-03-25T18:32:00.360 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:32:00.357+0000 7f629457b640 10 monclient: shutdown 2024-03-25T18:32:00.360 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:32:00.357+0000 7f629457b640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:32:00.360 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:32:00.357+0000 7f629457b640 1 -- 172.21.15.1:0/1383290471 >> v1:172.21.15.1:6789/0 conn(0x7f628c13c480 legacy=0x7f628c14dc00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:32:00.360 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:32:00.358+0000 7f629457b640 1 -- 172.21.15.1:0/1383290471 shutdown_connections 2024-03-25T18:32:00.360 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:32:00.358+0000 7f629457b640 1 -- 172.21.15.1:0/1383290471 >> 172.21.15.1:0/1383290471 conn(0x7f628c0617a0 msgr2=0x7f628c061bb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:32:00.360 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:32:00.358+0000 7f629457b640 1 -- 172.21.15.1:0/1383290471 shutdown_connections 2024-03-25T18:32:00.360 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:32:00.358+0000 7f629457b640 1 -- 172.21.15.1:0/1383290471 wait complete. 2024-03-25T18:32:00.360 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:32:00.358+0000 7f629457b640 10 monclient: build_initial_monmap 2024-03-25T18:32:00.360 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:32:00.358+0000 7f629457b640 10 monclient: monmap: 2024-03-25T18:32:00.360 INFO:teuthology.misc.health.smithi001.stderr:epoch 0 2024-03-25T18:32:00.360 INFO:teuthology.misc.health.smithi001.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T18:32:00.361 INFO:teuthology.misc.health.smithi001.stderr:last_changed 0.000000 2024-03-25T18:32:00.361 INFO:teuthology.misc.health.smithi001.stderr:created 0.000000 2024-03-25T18:32:00.361 INFO:teuthology.misc.health.smithi001.stderr:min_mon_release 0 (unknown) 2024-03-25T18:32:00.361 INFO:teuthology.misc.health.smithi001.stderr:election_strategy: 1 2024-03-25T18:32:00.361 INFO:teuthology.misc.health.smithi001.stderr:0: v1:172.21.15.1:6789/0 mon.noname-a 2024-03-25T18:32:00.361 INFO:teuthology.misc.health.smithi001.stderr:1: v1:172.21.15.115:6789/0 mon.noname-b 2024-03-25T18:32:00.361 INFO:teuthology.misc.health.smithi001.stderr:2: v1:172.21.15.1:6790/0 mon.noname-c 2024-03-25T18:32:00.361 INFO:teuthology.misc.health.smithi001.stderr: 2024-03-25T18:32:00.361 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:32:00.359+0000 7f629457b640 1 librados: starting msgr at 2024-03-25T18:32:00.361 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:32:00.359+0000 7f629457b640 1 librados: starting objecter 2024-03-25T18:32:00.361 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:32:00.359+0000 7f629457b640 1 Processor -- start 2024-03-25T18:32:00.361 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:32:00.359+0000 7f629457b640 1 -- start start 2024-03-25T18:32:00.361 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:32:00.359+0000 7f629457b640 1 librados: setting wanted keys 2024-03-25T18:32:00.361 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:32:00.359+0000 7f629457b640 1 librados: calling monclient init 2024-03-25T18:32:00.361 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:32:00.359+0000 7f629457b640 10 monclient: init 2024-03-25T18:32:00.362 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:32:00.360+0000 7f629457b640 10 monclient: _reopen_session rank -1 2024-03-25T18:32:00.362 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:32:00.360+0000 7f629457b640 10 monclient: _add_conns ranks=[0,1,2] 2024-03-25T18:32:00.362 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:32:00.360+0000 7f629457b640 10 monclient(hunting): picked mon.noname-a con 0x7f628c0b4670 addr v1:172.21.15.1:6789/0 2024-03-25T18:32:00.363 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:32:00.360+0000 7f629457b640 10 monclient(hunting): picked mon.noname-b con 0x7f628c0b4ff0 addr v1:172.21.15.115:6789/0 2024-03-25T18:32:00.363 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:32:00.360+0000 7f629457b640 10 monclient(hunting): picked mon.noname-c con 0x7f628c13c480 addr v1:172.21.15.1:6790/0 2024-03-25T18:32:00.363 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:32:00.360+0000 7f629457b640 1 -- --> v1:172.21.15.1:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f628c0aec70 con 0x7f628c0b4670 2024-03-25T18:32:00.363 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:32:00.360+0000 7f629457b640 1 -- --> v1:172.21.15.115:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f628c0b07a0 con 0x7f628c0b4ff0 2024-03-25T18:32:00.363 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:32:00.360+0000 7f629457b640 1 -- --> v1:172.21.15.1:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f628c0b0a80 con 0x7f628c13c480 2024-03-25T18:32:00.363 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:32:00.360+0000 7f629457b640 10 monclient(hunting): _renew_subs 2024-03-25T18:32:00.363 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:32:00.360+0000 7f629457b640 10 monclient(hunting): authenticate will time out at 7800.830349s 2024-03-25T18:32:00.376 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:32:00.375+0000 7f6291b16640 1 --1- >> v1:172.21.15.115:6789/0 conn(0x7f628c0b4ff0 0x7f628c13a2c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.115:6789/0 says I am v1:172.21.15.1:55010/0 (socket says 172.21.15.1:55010) 2024-03-25T18:32:00.376 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:32:00.375+0000 7f6291b16640 1 -- 172.21.15.1:0/3640402332 learned_addr learned my addr 172.21.15.1:0/3640402332 (peer_addr_for_me v1:172.21.15.1:0/0) 2024-03-25T18:32:00.376 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:32:00.376+0000 7f6282ffd640 10 client.?.objecter ms_handle_connect 0x7f628c0b4ff0 2024-03-25T18:32:00.376 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:32:00.376+0000 7f6282ffd640 10 client.?.objecter resend_mon_ops 2024-03-25T18:32:00.386 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:32:00.385+0000 7f6282ffd640 10 client.?.objecter ms_handle_connect 0x7f628c0b4670 2024-03-25T18:32:00.386 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:32:00.385+0000 7f6282ffd640 10 client.?.objecter resend_mon_ops 2024-03-25T18:32:00.388 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:32:00.387+0000 7f6282ffd640 10 client.?.objecter ms_handle_connect 0x7f628c13c480 2024-03-25T18:32:00.388 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:32:00.387+0000 7f6282ffd640 10 client.?.objecter resend_mon_ops 2024-03-25T18:32:00.390 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:32:00.390+0000 7f6282ffd640 1 -- 172.21.15.1:0/3640402332 <== mon.1 v1:172.21.15.115:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2008707100 0 0) 0x7f628c0b07a0 con 0x7f628c0b4ff0 2024-03-25T18:32:00.391 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:32:00.390+0000 7f6282ffd640 10 monclient(hunting): _negotiate 2024-03-25T18:32:00.391 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:32:00.390+0000 7f6282ffd640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:32:00.391 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:32:00.390+0000 7f6282ffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:32:00.391 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:32:00.390+0000 7f6282ffd640 10 monclient(hunting): my global_id is 4459 2024-03-25T18:32:00.391 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:32:00.390+0000 7f6282ffd640 1 -- 172.21.15.1:0/3640402332 --> v1:172.21.15.115:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f6270004eb0 con 0x7f628c0b4ff0 2024-03-25T18:32:00.398 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:32:00.396+0000 7f6282ffd640 1 -- 172.21.15.1:0/3640402332 <== mon.0 v1:172.21.15.1:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4124210 0 0) 0x7f628c0aec70 con 0x7f628c0b4670 2024-03-25T18:32:00.398 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:32:00.396+0000 7f6282ffd640 10 monclient(hunting): _negotiate 2024-03-25T18:32:00.398 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:32:00.396+0000 7f6282ffd640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:32:00.398 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:32:00.396+0000 7f6282ffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:32:00.398 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:32:00.396+0000 7f6282ffd640 10 monclient(hunting): my global_id is 4449 2024-03-25T18:32:00.398 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:32:00.396+0000 7f6282ffd640 1 -- 172.21.15.1:0/3640402332 --> v1:172.21.15.1:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f628c0b07a0 con 0x7f628c0b4670 2024-03-25T18:32:00.409 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:32:00.408+0000 7f6282ffd640 1 -- 172.21.15.1:0/3640402332 <== mon.1 v1:172.21.15.115:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2512113479 0 0) 0x7f6270004eb0 con 0x7f628c0b4ff0 2024-03-25T18:32:00.410 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:32:00.409+0000 7f6282ffd640 1 -- 172.21.15.1:0/3640402332 --> v1:172.21.15.115:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f628c0aec70 con 0x7f628c0b4ff0 2024-03-25T18:32:00.410 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:32:00.409+0000 7f6282ffd640 1 -- 172.21.15.1:0/3640402332 <== mon.1 v1:172.21.15.115:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1586965025 0 0) 0x7f627c0019b0 con 0x7f628c0b4ff0 2024-03-25T18:32:00.410 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:32:00.409+0000 7f6282ffd640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:32:00.410 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:32:00.409+0000 7f6282ffd640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-03-25T18:32:00.410 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:32:00.409+0000 7f6282ffd640 10 monclient(hunting): dump: 2024-03-25T18:32:00.410 INFO:teuthology.misc.health.smithi001.stderr:epoch 1 2024-03-25T18:32:00.410 INFO:teuthology.misc.health.smithi001.stderr:fsid c8ea72cf-9e99-4383-9b52-6c23d365fdfc 2024-03-25T18:32:00.410 INFO:teuthology.misc.health.smithi001.stderr:last_changed 2024-03-25T18:25:50.510519+0000 2024-03-25T18:32:00.410 INFO:teuthology.misc.health.smithi001.stderr:created 2024-03-25T18:25:50.510519+0000 2024-03-25T18:32:00.410 INFO:teuthology.misc.health.smithi001.stderr:min_mon_release 19 (squid) 2024-03-25T18:32:00.410 INFO:teuthology.misc.health.smithi001.stderr:election_strategy: 3 2024-03-25T18:32:00.410 INFO:teuthology.misc.health.smithi001.stderr:0: v1:172.21.15.1:6789/0 mon.a 2024-03-25T18:32:00.410 INFO:teuthology.misc.health.smithi001.stderr:1: v1:172.21.15.115:6789/0 mon.b 2024-03-25T18:32:00.411 INFO:teuthology.misc.health.smithi001.stderr:2: v1:172.21.15.1:6790/0 mon.c 2024-03-25T18:32:00.411 INFO:teuthology.misc.health.smithi001.stderr: 2024-03-25T18:32:00.411 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:32:00.409+0000 7f6282ffd640 10 monclient(hunting): _finish_auth 0 2024-03-25T18:32:00.411 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:32:00.410+0000 7f6282ffd640 1 -- 172.21.15.1:0/3640402332 <== mon.2 v1:172.21.15.1:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1022004203 0 0) 0x7f628c0b0a80 con 0x7f628c13c480 2024-03-25T18:32:00.411 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:32:00.410+0000 7f6282ffd640 10 monclient(hunting): _negotiate 2024-03-25T18:32:00.411 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:32:00.410+0000 7f6282ffd640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:32:00.411 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:32:00.410+0000 7f6282ffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:32:00.411 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:32:00.410+0000 7f6282ffd640 10 monclient(hunting): my global_id is 4460 2024-03-25T18:32:00.412 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:32:00.410+0000 7f6282ffd640 1 -- 172.21.15.1:0/3640402332 --> v1:172.21.15.1:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f6270004eb0 con 0x7f628c13c480 2024-03-25T18:32:00.416 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:32:00.415+0000 7f6282ffd640 1 -- 172.21.15.1:0/3640402332 <== mon.0 v1:172.21.15.1:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3754670062 0 0) 0x7f628c0b07a0 con 0x7f628c0b4670 2024-03-25T18:32:00.416 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:32:00.415+0000 7f6282ffd640 1 -- 172.21.15.1:0/3640402332 --> v1:172.21.15.1:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f628c0b0a80 con 0x7f628c0b4670 2024-03-25T18:32:00.416 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:32:00.415+0000 7f6282ffd640 1 -- 172.21.15.1:0/3640402332 <== mon.0 v1:172.21.15.1:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1586965025 0 0) 0x7f62880018a0 con 0x7f628c0b4670 2024-03-25T18:32:00.417 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:32:00.415+0000 7f6282ffd640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:32:00.417 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:32:00.415+0000 7f6282ffd640 10 monclient(hunting): got monmap 1 from mon.a (according to old e1) 2024-03-25T18:32:00.417 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:32:00.415+0000 7f6282ffd640 10 monclient(hunting): dump: 2024-03-25T18:32:00.417 INFO:teuthology.misc.health.smithi001.stderr:epoch 1 2024-03-25T18:32:00.417 INFO:teuthology.misc.health.smithi001.stderr:fsid c8ea72cf-9e99-4383-9b52-6c23d365fdfc 2024-03-25T18:32:00.417 INFO:teuthology.misc.health.smithi001.stderr:last_changed 2024-03-25T18:25:50.510519+0000 2024-03-25T18:32:00.417 INFO:teuthology.misc.health.smithi001.stderr:created 2024-03-25T18:25:50.510519+0000 2024-03-25T18:32:00.417 INFO:teuthology.misc.health.smithi001.stderr:min_mon_release 19 (squid) 2024-03-25T18:32:00.417 INFO:teuthology.misc.health.smithi001.stderr:election_strategy: 3 2024-03-25T18:32:00.417 INFO:teuthology.misc.health.smithi001.stderr:0: v1:172.21.15.1:6789/0 mon.a 2024-03-25T18:32:00.417 INFO:teuthology.misc.health.smithi001.stderr:1: v1:172.21.15.115:6789/0 mon.b 2024-03-25T18:32:00.417 INFO:teuthology.misc.health.smithi001.stderr:2: v1:172.21.15.1:6790/0 mon.c 2024-03-25T18:32:00.417 INFO:teuthology.misc.health.smithi001.stderr: 2024-03-25T18:32:00.423 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:32:00.421+0000 7f6282ffd640 1 -- 172.21.15.1:0/3640402332 <== mon.1 v1:172.21.15.115:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 615331397 0 0) 0x7f628c0aec70 con 0x7f628c0b4ff0 2024-03-25T18:32:00.423 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:32:00.421+0000 7f6282ffd640 1 -- 172.21.15.1:0/3640402332 >> v1:172.21.15.1:6790/0 conn(0x7f628c13c480 legacy=0x7f628c13a9d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:32:00.423 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:32:00.421+0000 7f6282ffd640 1 -- 172.21.15.1:0/3640402332 >> v1:172.21.15.1:6789/0 conn(0x7f628c0b4670 legacy=0x7f628c0af100 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:32:00.423 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:32:00.421+0000 7f6282ffd640 10 monclient: _finish_hunting 0 2024-03-25T18:32:00.423 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:32:00.421+0000 7f6282ffd640 1 monclient: found mon.b 2024-03-25T18:32:00.423 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:32:00.422+0000 7f6282ffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:32:00.423 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:32:00.422+0000 7f6282ffd640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.115:6789/0 2024-03-25T18:32:00.423 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:32:00.422+0000 7f6282ffd640 1 -- 172.21.15.1:0/3640402332 --> v1:172.21.15.115:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f628c0b0d60 con 0x7f628c0b4ff0 2024-03-25T18:32:00.423 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:32:00.422+0000 7f6282ffd640 10 monclient: _finish_auth 0 2024-03-25T18:32:00.423 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:32:00.422+0000 7f6282ffd640 10 monclient: _check_auth_tickets 2024-03-25T18:32:00.423 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:32:00.422+0000 7f6282ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:32:00.423 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:32:00.422+0000 7f629457b640 5 monclient: authenticate success, global_id 4459 2024-03-25T18:32:00.424 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:32:00.422+0000 7f629457b640 10 monclient: _renew_subs 2024-03-25T18:32:00.424 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:32:00.422+0000 7f629457b640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.115:6789/0 2024-03-25T18:32:00.424 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:32:00.422+0000 7f629457b640 1 -- 172.21.15.1:0/3640402332 --> v1:172.21.15.115:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f628c13b0e0 con 0x7f628c0b4ff0 2024-03-25T18:32:00.424 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:32:00.422+0000 7f629457b640 10 client.4459.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T18:32:00.424 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:32:00.422+0000 7f629457b640 10 monclient: _renew_subs 2024-03-25T18:32:00.424 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:32:00.422+0000 7f629457b640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.115:6789/0 2024-03-25T18:32:00.424 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:32:00.422+0000 7f629457b640 1 -- 172.21.15.1:0/3640402332 --> v1:172.21.15.115:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f628c0ae8e0 con 0x7f628c0b4ff0 2024-03-25T18:32:00.424 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:32:00.422+0000 7f629457b640 1 librados: init done 2024-03-25T18:32:00.424 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:32:00.423+0000 7f629457b640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T18:32:00.425 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:32:00.423+0000 7f629457b640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:32:00.425 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:32:00.423+0000 7f629457b640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.115:6789/0 2024-03-25T18:32:00.425 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:32:00.423+0000 7f629457b640 1 -- 172.21.15.1:0/3640402332 --> v1:172.21.15.115:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f628c06dbd0 con 0x7f628c0b4ff0 2024-03-25T18:32:00.701 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:32:00.700+0000 7f6282ffd640 1 -- 172.21.15.1:0/3640402332 <== mon.1 v1:172.21.15.115:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f627c0047e0 con 0x7f628c0b4ff0 2024-03-25T18:32:00.701 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:32:00.700+0000 7f6282ffd640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:32:00.701 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:32:00.700+0000 7f6282ffd640 1 -- 172.21.15.1:0/3640402332 <== mon.1 v1:172.21.15.115:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1586965025 0 0) 0x7f627c004af0 con 0x7f628c0b4ff0 2024-03-25T18:32:00.701 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:32:00.700+0000 7f6282ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:32:00.701 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:32:00.700+0000 7f6282ffd640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-25T18:32:00.701 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:32:00.700+0000 7f6282ffd640 10 monclient: dump: 2024-03-25T18:32:00.701 INFO:teuthology.misc.health.smithi001.stderr:epoch 1 2024-03-25T18:32:00.701 INFO:teuthology.misc.health.smithi001.stderr:fsid c8ea72cf-9e99-4383-9b52-6c23d365fdfc 2024-03-25T18:32:00.701 INFO:teuthology.misc.health.smithi001.stderr:last_changed 2024-03-25T18:25:50.510519+0000 2024-03-25T18:32:00.702 INFO:teuthology.misc.health.smithi001.stderr:created 2024-03-25T18:25:50.510519+0000 2024-03-25T18:32:00.702 INFO:teuthology.misc.health.smithi001.stderr:min_mon_release 19 (squid) 2024-03-25T18:32:00.702 INFO:teuthology.misc.health.smithi001.stderr:election_strategy: 3 2024-03-25T18:32:00.702 INFO:teuthology.misc.health.smithi001.stderr:0: v1:172.21.15.1:6789/0 mon.a 2024-03-25T18:32:00.702 INFO:teuthology.misc.health.smithi001.stderr:1: v1:172.21.15.115:6789/0 mon.b 2024-03-25T18:32:00.702 INFO:teuthology.misc.health.smithi001.stderr:2: v1:172.21.15.1:6790/0 mon.c 2024-03-25T18:32:00.702 INFO:teuthology.misc.health.smithi001.stderr: 2024-03-25T18:32:00.702 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:32:00.700+0000 7f6290b14640 4 set_mon_vals no callback set 2024-03-25T18:32:00.703 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:32:00.700+0000 7f6282ffd640 1 -- 172.21.15.1:0/3640402332 <== mon.1 v1:172.21.15.115:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 21787975 0 0) 0x7f627c01aac0 con 0x7f628c0b4ff0 2024-03-25T18:32:00.703 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:32:00.701+0000 7f6292317640 10 monclient: get_auth_request con 0x7f627006df70 auth_method 0 2024-03-25T18:32:00.703 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:32:00.702+0000 7f6282ffd640 10 client.4459.objecter ms_handle_connect 0x7f627006df70 2024-03-25T18:32:00.710 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:32:00.709+0000 7f6282ffd640 1 -- 172.21.15.1:0/3640402332 <== mon.1 v1:172.21.15.115:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 996254230 0 0) 0x7f627c054870 con 0x7f628c0b4ff0 2024-03-25T18:32:00.710 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:32:00.709+0000 7f6282ffd640 10 client.4459.objecter ms_dispatch 0x7f628c000b90 osd_map(10..10 src has 1..10) v4 2024-03-25T18:32:00.711 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:32:00.709+0000 7f6282ffd640 3 client.4459.objecter handle_osd_map got epochs [10,10] > 0 2024-03-25T18:32:00.711 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:32:00.709+0000 7f6282ffd640 3 client.4459.objecter handle_osd_map decoding full epoch 10 2024-03-25T18:32:00.711 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:32:00.709+0000 7f6282ffd640 20 client.4459.objecter dump_active .. 0 homeless 2024-03-25T18:32:00.711 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:32:00.710+0000 7f6282ffd640 1 -- 172.21.15.1:0/3640402332 <== mon.1 v1:172.21.15.115:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f627c0501b0 con 0x7f628c0b4ff0 2024-03-25T18:32:00.711 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:32:00.710+0000 7f6282ffd640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:32:00.711 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:32:00.710+0000 7f6282ffd640 10 monclient: _finish_command 1 = system:0 2024-03-25T18:32:00.940 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:32:00.939+0000 7f629457b640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:32:00.940 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:32:00.939+0000 7f629457b640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:32:00.940 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:32:00.939+0000 7f629457b640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.115:6789/0 2024-03-25T18:32:00.941 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:32:00.939+0000 7f629457b640 1 -- 172.21.15.1:0/3640402332 --> v1:172.21.15.115:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f628c0aee20 con 0x7f628c0b4ff0 2024-03-25T18:32:01.118 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:32:01.117+0000 7f6282ffd640 1 -- 172.21.15.1:0/3640402332 <== mon.1 v1:172.21.15.115:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 4092213348) 0x7f628c0aee20 con 0x7f628c0b4ff0 2024-03-25T18:32:01.118 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:32:01.117+0000 7f6282ffd640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:32:01.118 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:32:01.117+0000 7f6282ffd640 10 monclient: _finish_command 2 = system:0 2024-03-25T18:32:01.119 INFO:teuthology.misc.health.smithi001.stdout: 2024-03-25T18:32:01.119 INFO:teuthology.misc.health.smithi001.stdout:{"epoch":10,"fsid":"c8ea72cf-9e99-4383-9b52-6c23d365fdfc","created":"2024-03-25T18:26:38.856638+0000","modified":"2024-03-25T18:26:44.224208+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T18:26:44.224208+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"b9f34922-3a84-49be-8426-807c1eceaefc","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"42bf306f-55e3-4634-9874-8b4b73ac6644","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"292cb23a-18a9-4207-a7cc-4d5727feaec0","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"45e0e9ab-898d-4673-9590-b4117bc5e84a","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"831a5a09-73ab-4a61-9872-088e76bdbc66","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"88cd903d-3b55-4840-9396-6e3cfe005282","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"89db7355-bc9d-46f4-8ab0-e8d9e744e026","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"cac47bee-0914-4f08-896f-348838882866","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T18:32:01.121 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:32:01.120+0000 7f629457b640 10 librados: watch_flush enter 2024-03-25T18:32:01.121 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:32:01.120+0000 7f629457b640 10 librados: watch_flush exit 2024-03-25T18:32:01.121 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:32:01.120+0000 7f629457b640 20 client.4459.objecter shutdown clearing up homeless session... 2024-03-25T18:32:01.121 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:32:01.120+0000 7f629457b640 10 client.4459.objecter successfully canceled tick 2024-03-25T18:32:01.121 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:32:01.120+0000 7f629457b640 1 -- 172.21.15.1:0/3640402332 >> v1:172.21.15.115:6800/4237379176 conn(0x7f627006df70 legacy=0x7f6270070430 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:32:01.121 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:32:01.120+0000 7f629457b640 10 monclient: shutdown 2024-03-25T18:32:01.122 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:32:01.120+0000 7f629457b640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:32:01.122 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:32:01.120+0000 7f629457b640 1 -- 172.21.15.1:0/3640402332 >> v1:172.21.15.115:6789/0 conn(0x7f628c0b4ff0 legacy=0x7f628c13a2c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:32:01.122 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:32:01.120+0000 7f629457b640 1 -- 172.21.15.1:0/3640402332 shutdown_connections 2024-03-25T18:32:01.122 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:32:01.120+0000 7f629457b640 1 -- 172.21.15.1:0/3640402332 >> 172.21.15.1:0/3640402332 conn(0x7f628c0617a0 msgr2=0x7f628c140760 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:32:01.122 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:32:01.120+0000 7f629457b640 1 -- 172.21.15.1:0/3640402332 shutdown_connections 2024-03-25T18:32:01.122 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:32:01.120+0000 7f629457b640 1 -- 172.21.15.1:0/3640402332 wait complete. 2024-03-25T18:32:01.122 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:32:01.120+0000 7f629457b640 1 librados: shutdown 2024-03-25T18:32:01.132 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T18:32:07.939 DEBUG:teuthology.orchestra.run.smithi001:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T18:32:08.057 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:32:08.056+0000 7fc33f923640 10 monclient: get_monmap_and_config 2024-03-25T18:32:08.057 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:32:08.056+0000 7fc33f923640 10 monclient: build_initial_monmap 2024-03-25T18:32:08.057 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:32:08.056+0000 7fc33f923640 10 monclient: monmap: 2024-03-25T18:32:08.057 INFO:teuthology.misc.health.smithi001.stderr:epoch 0 2024-03-25T18:32:08.058 INFO:teuthology.misc.health.smithi001.stderr:fsid c8ea72cf-9e99-4383-9b52-6c23d365fdfc 2024-03-25T18:32:08.058 INFO:teuthology.misc.health.smithi001.stderr:last_changed 2024-03-25T18:32:08.057239+0000 2024-03-25T18:32:08.058 INFO:teuthology.misc.health.smithi001.stderr:created 2024-03-25T18:32:08.057239+0000 2024-03-25T18:32:08.058 INFO:teuthology.misc.health.smithi001.stderr:min_mon_release 0 (unknown) 2024-03-25T18:32:08.058 INFO:teuthology.misc.health.smithi001.stderr:election_strategy: 3 2024-03-25T18:32:08.058 INFO:teuthology.misc.health.smithi001.stderr:0: v1:172.21.15.1:6789/0 mon.noname-a 2024-03-25T18:32:08.058 INFO:teuthology.misc.health.smithi001.stderr:1: v1:172.21.15.115:6789/0 mon.noname-c 2024-03-25T18:32:08.058 INFO:teuthology.misc.health.smithi001.stderr:2: v1:172.21.15.1:6790/0 mon.noname-b 2024-03-25T18:32:08.058 INFO:teuthology.misc.health.smithi001.stderr: 2024-03-25T18:32:08.059 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:32:08.057+0000 7fc33f923640 1 Processor -- start 2024-03-25T18:32:08.059 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:32:08.057+0000 7fc33f923640 1 -- start start 2024-03-25T18:32:08.059 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:32:08.057+0000 7fc33f923640 10 monclient: init 2024-03-25T18:32:08.059 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:32:08.058+0000 7fc33f923640 10 monclient: _reopen_session rank -1 2024-03-25T18:32:08.059 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:32:08.058+0000 7fc33f923640 10 monclient: _add_conns ranks=[0,1,2] 2024-03-25T18:32:08.059 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:32:08.058+0000 7fc33f923640 10 monclient(hunting): picked mon.noname-a con 0x7fc3380b4ff0 addr v1:172.21.15.1:6789/0 2024-03-25T18:32:08.059 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:32:08.058+0000 7fc33f923640 10 monclient(hunting): picked mon.noname-c con 0x7fc3380b4670 addr v1:172.21.15.115:6789/0 2024-03-25T18:32:08.059 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:32:08.058+0000 7fc33f923640 10 monclient(hunting): picked mon.noname-b con 0x7fc33813c480 addr v1:172.21.15.1:6790/0 2024-03-25T18:32:08.060 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:32:08.058+0000 7fc33f923640 1 -- --> v1:172.21.15.1:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc3380ab0e0 con 0x7fc3380b4ff0 2024-03-25T18:32:08.060 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:32:08.058+0000 7fc33f923640 1 -- --> v1:172.21.15.115:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc3380ab3c0 con 0x7fc3380b4670 2024-03-25T18:32:08.060 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:32:08.058+0000 7fc33f923640 1 -- --> v1:172.21.15.1:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc3380a9f30 con 0x7fc33813c480 2024-03-25T18:32:08.060 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:32:08.058+0000 7fc33f923640 10 monclient(hunting): _renew_subs 2024-03-25T18:32:08.060 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:32:08.058+0000 7fc33f923640 10 monclient(hunting): authenticate will time out at 7808.528061s 2024-03-25T18:32:08.064 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:32:08.063+0000 7fc33dec0640 1 --1- >> v1:172.21.15.1:6790/0 conn(0x7fc33813c480 0x7fc33814dc00 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.1:6790/0 says I am v1:172.21.15.1:41218/0 (socket says 172.21.15.1:41218) 2024-03-25T18:32:08.064 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:32:08.063+0000 7fc33dec0640 1 -- 172.21.15.1:0/3900877892 learned_addr learned my addr 172.21.15.1:0/3900877892 (peer_addr_for_me v1:172.21.15.1:0/0) 2024-03-25T18:32:08.076 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:32:08.075+0000 7fc32ffff640 1 -- 172.21.15.1:0/3900877892 <== mon.2 v1:172.21.15.1:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3088082390 0 0) 0x7fc3380a9f30 con 0x7fc33813c480 2024-03-25T18:32:08.076 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:32:08.075+0000 7fc32ffff640 10 monclient(hunting): _negotiate 2024-03-25T18:32:08.076 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:32:08.075+0000 7fc32ffff640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:32:08.077 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:32:08.075+0000 7fc32ffff640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:32:08.077 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:32:08.075+0000 7fc32ffff640 10 monclient(hunting): my global_id is 4463 2024-03-25T18:32:08.077 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:32:08.075+0000 7fc32ffff640 1 -- 172.21.15.1:0/3900877892 --> v1:172.21.15.1:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fc324003520 con 0x7fc33813c480 2024-03-25T18:32:08.084 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:32:08.083+0000 7fc32ffff640 1 -- 172.21.15.1:0/3900877892 <== mon.1 v1:172.21.15.115:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 231789831 0 0) 0x7fc3380ab3c0 con 0x7fc3380b4670 2024-03-25T18:32:08.084 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:32:08.083+0000 7fc32ffff640 10 monclient(hunting): _negotiate 2024-03-25T18:32:08.084 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:32:08.083+0000 7fc32ffff640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:32:08.084 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:32:08.083+0000 7fc32ffff640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:32:08.084 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:32:08.083+0000 7fc32ffff640 10 monclient(hunting): my global_id is 4462 2024-03-25T18:32:08.084 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:32:08.083+0000 7fc32ffff640 1 -- 172.21.15.1:0/3900877892 --> v1:172.21.15.115:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fc3380a9f30 con 0x7fc3380b4670 2024-03-25T18:32:08.090 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:32:08.088+0000 7fc32ffff640 1 -- 172.21.15.1:0/3900877892 <== mon.2 v1:172.21.15.1:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1936627915 0 0) 0x7fc324003520 con 0x7fc33813c480 2024-03-25T18:32:08.090 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:32:08.088+0000 7fc32ffff640 1 -- 172.21.15.1:0/3900877892 >> v1:172.21.15.115:6789/0 conn(0x7fc3380b4670 legacy=0x7fc3381425c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:32:08.090 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:32:08.088+0000 7fc32ffff640 1 -- 172.21.15.1:0/3900877892 >> v1:172.21.15.1:6789/0 conn(0x7fc3380b4ff0 legacy=0x7fc33813bd70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-25T18:32:08.090 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:32:08.088+0000 7fc32ffff640 10 monclient: _finish_hunting 0 2024-03-25T18:32:08.090 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:32:08.088+0000 7fc32ffff640 1 monclient: found mon.noname-b 2024-03-25T18:32:08.090 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:32:08.089+0000 7fc32ffff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:32:08.090 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:32:08.089+0000 7fc32ffff640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.1:6790/0 2024-03-25T18:32:08.090 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:32:08.089+0000 7fc32ffff640 1 -- 172.21.15.1:0/3900877892 --> v1:172.21.15.1:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc3380a9b70 con 0x7fc33813c480 2024-03-25T18:32:08.090 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:32:08.089+0000 7fc32ffff640 10 monclient: _finish_auth 0 2024-03-25T18:32:08.090 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:32:08.089+0000 7fc32ffff640 10 monclient: _check_auth_tickets 2024-03-25T18:32:08.090 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:32:08.089+0000 7fc32ffff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:32:08.091 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:32:08.089+0000 7fc32ffff640 1 -- 172.21.15.1:0/3900877892 <== mon.2 v1:172.21.15.1:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1586965025 0 0) 0x7fc330001a10 con 0x7fc33813c480 2024-03-25T18:32:08.091 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:32:08.089+0000 7fc32ffff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:32:08.091 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:32:08.089+0000 7fc32ffff640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-03-25T18:32:08.091 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:32:08.089+0000 7fc32ffff640 10 monclient: dump: 2024-03-25T18:32:08.091 INFO:teuthology.misc.health.smithi001.stderr:epoch 1 2024-03-25T18:32:08.091 INFO:teuthology.misc.health.smithi001.stderr:fsid c8ea72cf-9e99-4383-9b52-6c23d365fdfc 2024-03-25T18:32:08.091 INFO:teuthology.misc.health.smithi001.stderr:last_changed 2024-03-25T18:25:50.510519+0000 2024-03-25T18:32:08.091 INFO:teuthology.misc.health.smithi001.stderr:created 2024-03-25T18:25:50.510519+0000 2024-03-25T18:32:08.091 INFO:teuthology.misc.health.smithi001.stderr:min_mon_release 19 (squid) 2024-03-25T18:32:08.091 INFO:teuthology.misc.health.smithi001.stderr:election_strategy: 3 2024-03-25T18:32:08.091 INFO:teuthology.misc.health.smithi001.stderr:0: v1:172.21.15.1:6789/0 mon.a 2024-03-25T18:32:08.091 INFO:teuthology.misc.health.smithi001.stderr:1: v1:172.21.15.115:6789/0 mon.b 2024-03-25T18:32:08.091 INFO:teuthology.misc.health.smithi001.stderr:2: v1:172.21.15.1:6790/0 mon.c 2024-03-25T18:32:08.091 INFO:teuthology.misc.health.smithi001.stderr: 2024-03-25T18:32:08.091 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:32:08.089+0000 7fc33f923640 5 monclient: authenticate success, global_id 4463 2024-03-25T18:32:08.091 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:32:08.089+0000 7fc33f923640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T18:32:08.095 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:32:08.094+0000 7fc32ffff640 1 -- 172.21.15.1:0/3900877892 <== mon.2 v1:172.21.15.1:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fc330001b70 con 0x7fc33813c480 2024-03-25T18:32:08.095 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:32:08.094+0000 7fc32ffff640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:32:08.095 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:32:08.094+0000 7fc32ffff640 1 -- 172.21.15.1:0/3900877892 <== mon.2 v1:172.21.15.1:6790/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1586965025 0 0) 0x7fc330001f80 con 0x7fc33813c480 2024-03-25T18:32:08.095 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:32:08.094+0000 7fc33f923640 10 monclient: get_monmap_and_config success 2024-03-25T18:32:08.095 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:32:08.094+0000 7fc33f923640 4 set_mon_vals no callback set 2024-03-25T18:32:08.095 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:32:08.094+0000 7fc32ffff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:32:08.095 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:32:08.094+0000 7fc32ffff640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-25T18:32:08.095 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:32:08.094+0000 7fc32ffff640 10 monclient: dump: 2024-03-25T18:32:08.095 INFO:teuthology.misc.health.smithi001.stderr:epoch 1 2024-03-25T18:32:08.095 INFO:teuthology.misc.health.smithi001.stderr:fsid c8ea72cf-9e99-4383-9b52-6c23d365fdfc 2024-03-25T18:32:08.095 INFO:teuthology.misc.health.smithi001.stderr:last_changed 2024-03-25T18:25:50.510519+0000 2024-03-25T18:32:08.095 INFO:teuthology.misc.health.smithi001.stderr:created 2024-03-25T18:25:50.510519+0000 2024-03-25T18:32:08.095 INFO:teuthology.misc.health.smithi001.stderr:min_mon_release 19 (squid) 2024-03-25T18:32:08.095 INFO:teuthology.misc.health.smithi001.stderr:election_strategy: 3 2024-03-25T18:32:08.096 INFO:teuthology.misc.health.smithi001.stderr:0: v1:172.21.15.1:6789/0 mon.a 2024-03-25T18:32:08.096 INFO:teuthology.misc.health.smithi001.stderr:1: v1:172.21.15.115:6789/0 mon.b 2024-03-25T18:32:08.096 INFO:teuthology.misc.health.smithi001.stderr:2: v1:172.21.15.1:6790/0 mon.c 2024-03-25T18:32:08.096 INFO:teuthology.misc.health.smithi001.stderr: 2024-03-25T18:32:08.098 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:32:08.094+0000 7fc33f923640 10 monclient: shutdown 2024-03-25T18:32:08.098 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:32:08.094+0000 7fc33f923640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:32:08.098 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:32:08.094+0000 7fc33f923640 1 -- 172.21.15.1:0/3900877892 >> v1:172.21.15.1:6790/0 conn(0x7fc33813c480 legacy=0x7fc33814dc00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:32:08.098 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:32:08.094+0000 7fc33f923640 1 -- 172.21.15.1:0/3900877892 shutdown_connections 2024-03-25T18:32:08.098 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:32:08.094+0000 7fc33f923640 1 -- 172.21.15.1:0/3900877892 >> 172.21.15.1:0/3900877892 conn(0x7fc3380617a0 msgr2=0x7fc338061bb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:32:08.098 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:32:08.095+0000 7fc33f923640 1 -- 172.21.15.1:0/3900877892 shutdown_connections 2024-03-25T18:32:08.098 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:32:08.095+0000 7fc33f923640 1 -- 172.21.15.1:0/3900877892 wait complete. 2024-03-25T18:32:08.098 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:32:08.095+0000 7fc33f923640 10 monclient: build_initial_monmap 2024-03-25T18:32:08.098 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:32:08.095+0000 7fc33f923640 10 monclient: monmap: 2024-03-25T18:32:08.098 INFO:teuthology.misc.health.smithi001.stderr:epoch 0 2024-03-25T18:32:08.099 INFO:teuthology.misc.health.smithi001.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T18:32:08.099 INFO:teuthology.misc.health.smithi001.stderr:last_changed 0.000000 2024-03-25T18:32:08.099 INFO:teuthology.misc.health.smithi001.stderr:created 0.000000 2024-03-25T18:32:08.099 INFO:teuthology.misc.health.smithi001.stderr:min_mon_release 0 (unknown) 2024-03-25T18:32:08.099 INFO:teuthology.misc.health.smithi001.stderr:election_strategy: 1 2024-03-25T18:32:08.099 INFO:teuthology.misc.health.smithi001.stderr:0: v1:172.21.15.1:6789/0 mon.noname-a 2024-03-25T18:32:08.099 INFO:teuthology.misc.health.smithi001.stderr:1: v1:172.21.15.115:6789/0 mon.noname-b 2024-03-25T18:32:08.099 INFO:teuthology.misc.health.smithi001.stderr:2: v1:172.21.15.1:6790/0 mon.noname-c 2024-03-25T18:32:08.099 INFO:teuthology.misc.health.smithi001.stderr: 2024-03-25T18:32:08.099 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:32:08.096+0000 7fc33f923640 1 librados: starting msgr at 2024-03-25T18:32:08.099 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:32:08.096+0000 7fc33f923640 1 librados: starting objecter 2024-03-25T18:32:08.099 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:32:08.096+0000 7fc33f923640 1 Processor -- start 2024-03-25T18:32:08.099 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:32:08.096+0000 7fc33f923640 1 -- start start 2024-03-25T18:32:08.099 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:32:08.096+0000 7fc33f923640 1 librados: setting wanted keys 2024-03-25T18:32:08.099 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:32:08.096+0000 7fc33f923640 1 librados: calling monclient init 2024-03-25T18:32:08.100 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:32:08.096+0000 7fc33f923640 10 monclient: init 2024-03-25T18:32:08.100 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:32:08.096+0000 7fc33f923640 10 monclient: _reopen_session rank -1 2024-03-25T18:32:08.100 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:32:08.096+0000 7fc33f923640 10 monclient: _add_conns ranks=[2,1,0] 2024-03-25T18:32:08.100 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:32:08.096+0000 7fc33f923640 10 monclient(hunting): picked mon.noname-c con 0x7fc3380b4670 addr v1:172.21.15.1:6790/0 2024-03-25T18:32:08.100 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:32:08.096+0000 7fc33f923640 10 monclient(hunting): picked mon.noname-b con 0x7fc3380b4ff0 addr v1:172.21.15.115:6789/0 2024-03-25T18:32:08.100 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:32:08.096+0000 7fc33f923640 10 monclient(hunting): picked mon.noname-a con 0x7fc33813c480 addr v1:172.21.15.1:6789/0 2024-03-25T18:32:08.100 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:32:08.096+0000 7fc33f923640 1 -- --> v1:172.21.15.1:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc338134420 con 0x7fc33813c480 2024-03-25T18:32:08.100 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:32:08.096+0000 7fc33f923640 1 -- --> v1:172.21.15.115:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc338131b20 con 0x7fc3380b4ff0 2024-03-25T18:32:08.100 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:32:08.096+0000 7fc33f923640 1 -- --> v1:172.21.15.1:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc338143b60 con 0x7fc3380b4670 2024-03-25T18:32:08.100 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:32:08.096+0000 7fc33f923640 10 monclient(hunting): _renew_subs 2024-03-25T18:32:08.100 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:32:08.096+0000 7fc33f923640 10 monclient(hunting): authenticate will time out at 7808.566605s 2024-03-25T18:32:08.109 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:32:08.108+0000 7fc33cebe640 1 --1- >> v1:172.21.15.115:6789/0 conn(0x7fc3380b4ff0 0x7fc338132040 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.115:6789/0 says I am v1:172.21.15.1:48430/0 (socket says 172.21.15.1:48430) 2024-03-25T18:32:08.109 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:32:08.108+0000 7fc33cebe640 1 -- 172.21.15.1:0/346445085 learned_addr learned my addr 172.21.15.1:0/346445085 (peer_addr_for_me v1:172.21.15.1:0/0) 2024-03-25T18:32:08.110 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:32:08.109+0000 7fc32dffb640 10 client.?.objecter ms_handle_connect 0x7fc3380b4ff0 2024-03-25T18:32:08.110 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:32:08.109+0000 7fc32dffb640 10 client.?.objecter resend_mon_ops 2024-03-25T18:32:08.111 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:32:08.111+0000 7fc32dffb640 10 client.?.objecter ms_handle_connect 0x7fc33813c480 2024-03-25T18:32:08.111 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:32:08.111+0000 7fc32dffb640 10 client.?.objecter resend_mon_ops 2024-03-25T18:32:08.122 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:32:08.121+0000 7fc32dffb640 1 -- 172.21.15.1:0/346445085 <== mon.0 v1:172.21.15.1:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1574184134 0 0) 0x7fc338134420 con 0x7fc33813c480 2024-03-25T18:32:08.122 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:32:08.121+0000 7fc32dffb640 10 monclient(hunting): _negotiate 2024-03-25T18:32:08.122 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:32:08.121+0000 7fc32dffb640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:32:08.122 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:32:08.121+0000 7fc32dffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:32:08.122 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:32:08.121+0000 7fc32dffb640 10 monclient(hunting): my global_id is 4452 2024-03-25T18:32:08.122 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:32:08.121+0000 7fc32dffb640 1 -- 172.21.15.1:0/346445085 --> v1:172.21.15.1:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fc31c004eb0 con 0x7fc33813c480 2024-03-25T18:32:08.124 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:32:08.123+0000 7fc32dffb640 10 client.?.objecter ms_handle_connect 0x7fc3380b4670 2024-03-25T18:32:08.124 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:32:08.123+0000 7fc32dffb640 10 client.?.objecter resend_mon_ops 2024-03-25T18:32:08.128 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:32:08.127+0000 7fc32dffb640 1 -- 172.21.15.1:0/346445085 <== mon.1 v1:172.21.15.115:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 901359758 0 0) 0x7fc338131b20 con 0x7fc3380b4ff0 2024-03-25T18:32:08.128 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:32:08.127+0000 7fc32dffb640 10 monclient(hunting): _negotiate 2024-03-25T18:32:08.128 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:32:08.127+0000 7fc32dffb640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:32:08.128 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:32:08.127+0000 7fc32dffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:32:08.128 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:32:08.127+0000 7fc32dffb640 10 monclient(hunting): my global_id is 4465 2024-03-25T18:32:08.128 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:32:08.127+0000 7fc32dffb640 1 -- 172.21.15.1:0/346445085 --> v1:172.21.15.115:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fc338134420 con 0x7fc3380b4ff0 2024-03-25T18:32:08.143 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:32:08.142+0000 7fc32dffb640 1 -- 172.21.15.1:0/346445085 <== mon.2 v1:172.21.15.1:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3846135395 0 0) 0x7fc338143b60 con 0x7fc3380b4670 2024-03-25T18:32:08.143 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:32:08.142+0000 7fc32dffb640 10 monclient(hunting): _negotiate 2024-03-25T18:32:08.143 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:32:08.142+0000 7fc32dffb640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:32:08.143 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:32:08.142+0000 7fc32dffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:32:08.143 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:32:08.142+0000 7fc32dffb640 10 monclient(hunting): my global_id is 4466 2024-03-25T18:32:08.143 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:32:08.142+0000 7fc32dffb640 1 -- 172.21.15.1:0/346445085 --> v1:172.21.15.1:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fc338131b20 con 0x7fc3380b4670 2024-03-25T18:32:08.151 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:32:08.149+0000 7fc32dffb640 1 -- 172.21.15.1:0/346445085 <== mon.1 v1:172.21.15.115:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2260360165 0 0) 0x7fc338134420 con 0x7fc3380b4ff0 2024-03-25T18:32:08.151 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:32:08.150+0000 7fc32dffb640 1 -- 172.21.15.1:0/346445085 --> v1:172.21.15.115:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fc338143b60 con 0x7fc3380b4ff0 2024-03-25T18:32:08.151 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:32:08.150+0000 7fc32dffb640 1 -- 172.21.15.1:0/346445085 <== mon.1 v1:172.21.15.115:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1586965025 0 0) 0x7fc3280019f0 con 0x7fc3380b4ff0 2024-03-25T18:32:08.151 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:32:08.150+0000 7fc32dffb640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:32:08.151 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:32:08.150+0000 7fc32dffb640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-03-25T18:32:08.151 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:32:08.150+0000 7fc32dffb640 10 monclient(hunting): dump: 2024-03-25T18:32:08.151 INFO:teuthology.misc.health.smithi001.stderr:epoch 1 2024-03-25T18:32:08.151 INFO:teuthology.misc.health.smithi001.stderr:fsid c8ea72cf-9e99-4383-9b52-6c23d365fdfc 2024-03-25T18:32:08.151 INFO:teuthology.misc.health.smithi001.stderr:last_changed 2024-03-25T18:25:50.510519+0000 2024-03-25T18:32:08.151 INFO:teuthology.misc.health.smithi001.stderr:created 2024-03-25T18:25:50.510519+0000 2024-03-25T18:32:08.151 INFO:teuthology.misc.health.smithi001.stderr:min_mon_release 19 (squid) 2024-03-25T18:32:08.151 INFO:teuthology.misc.health.smithi001.stderr:election_strategy: 3 2024-03-25T18:32:08.151 INFO:teuthology.misc.health.smithi001.stderr:0: v1:172.21.15.1:6789/0 mon.a 2024-03-25T18:32:08.151 INFO:teuthology.misc.health.smithi001.stderr:1: v1:172.21.15.115:6789/0 mon.b 2024-03-25T18:32:08.151 INFO:teuthology.misc.health.smithi001.stderr:2: v1:172.21.15.1:6790/0 mon.c 2024-03-25T18:32:08.152 INFO:teuthology.misc.health.smithi001.stderr: 2024-03-25T18:32:08.152 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:32:08.150+0000 7fc32dffb640 10 monclient(hunting): _finish_auth 0 2024-03-25T18:32:08.157 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:32:08.156+0000 7fc32dffb640 1 -- 172.21.15.1:0/346445085 <== mon.0 v1:172.21.15.1:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1356281059 0 0) 0x7fc31c004eb0 con 0x7fc33813c480 2024-03-25T18:32:08.157 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:32:08.156+0000 7fc32dffb640 1 -- 172.21.15.1:0/346445085 --> v1:172.21.15.1:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fc338134420 con 0x7fc33813c480 2024-03-25T18:32:08.157 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:32:08.156+0000 7fc32dffb640 1 -- 172.21.15.1:0/346445085 <== mon.0 v1:172.21.15.1:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1586965025 0 0) 0x7fc330001ec0 con 0x7fc33813c480 2024-03-25T18:32:08.158 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:32:08.156+0000 7fc32dffb640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:32:08.158 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:32:08.156+0000 7fc32dffb640 10 monclient(hunting): got monmap 1 from mon.a (according to old e1) 2024-03-25T18:32:08.158 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:32:08.156+0000 7fc32dffb640 10 monclient(hunting): dump: 2024-03-25T18:32:08.158 INFO:teuthology.misc.health.smithi001.stderr:epoch 1 2024-03-25T18:32:08.158 INFO:teuthology.misc.health.smithi001.stderr:fsid c8ea72cf-9e99-4383-9b52-6c23d365fdfc 2024-03-25T18:32:08.158 INFO:teuthology.misc.health.smithi001.stderr:last_changed 2024-03-25T18:25:50.510519+0000 2024-03-25T18:32:08.158 INFO:teuthology.misc.health.smithi001.stderr:created 2024-03-25T18:25:50.510519+0000 2024-03-25T18:32:08.158 INFO:teuthology.misc.health.smithi001.stderr:min_mon_release 19 (squid) 2024-03-25T18:32:08.158 INFO:teuthology.misc.health.smithi001.stderr:election_strategy: 3 2024-03-25T18:32:08.158 INFO:teuthology.misc.health.smithi001.stderr:0: v1:172.21.15.1:6789/0 mon.a 2024-03-25T18:32:08.158 INFO:teuthology.misc.health.smithi001.stderr:1: v1:172.21.15.115:6789/0 mon.b 2024-03-25T18:32:08.158 INFO:teuthology.misc.health.smithi001.stderr:2: v1:172.21.15.1:6790/0 mon.c 2024-03-25T18:32:08.158 INFO:teuthology.misc.health.smithi001.stderr: 2024-03-25T18:32:08.174 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:32:08.173+0000 7fc32dffb640 1 -- 172.21.15.1:0/346445085 <== mon.2 v1:172.21.15.1:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1013266788 0 0) 0x7fc338131b20 con 0x7fc3380b4670 2024-03-25T18:32:08.174 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:32:08.173+0000 7fc32dffb640 1 -- 172.21.15.1:0/346445085 --> v1:172.21.15.1:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fc31c004eb0 con 0x7fc3380b4670 2024-03-25T18:32:08.174 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:32:08.173+0000 7fc32dffb640 1 -- 172.21.15.1:0/346445085 <== mon.2 v1:172.21.15.1:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1586965025 0 0) 0x7fc334001510 con 0x7fc3380b4670 2024-03-25T18:32:08.174 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:32:08.173+0000 7fc32dffb640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:32:08.174 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:32:08.173+0000 7fc32dffb640 10 monclient(hunting): got monmap 1 from mon.c (according to old e1) 2024-03-25T18:32:08.175 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:32:08.173+0000 7fc32dffb640 10 monclient(hunting): dump: 2024-03-25T18:32:08.175 INFO:teuthology.misc.health.smithi001.stderr:epoch 1 2024-03-25T18:32:08.175 INFO:teuthology.misc.health.smithi001.stderr:fsid c8ea72cf-9e99-4383-9b52-6c23d365fdfc 2024-03-25T18:32:08.175 INFO:teuthology.misc.health.smithi001.stderr:last_changed 2024-03-25T18:25:50.510519+0000 2024-03-25T18:32:08.175 INFO:teuthology.misc.health.smithi001.stderr:created 2024-03-25T18:25:50.510519+0000 2024-03-25T18:32:08.175 INFO:teuthology.misc.health.smithi001.stderr:min_mon_release 19 (squid) 2024-03-25T18:32:08.175 INFO:teuthology.misc.health.smithi001.stderr:election_strategy: 3 2024-03-25T18:32:08.175 INFO:teuthology.misc.health.smithi001.stderr:0: v1:172.21.15.1:6789/0 mon.a 2024-03-25T18:32:08.175 INFO:teuthology.misc.health.smithi001.stderr:1: v1:172.21.15.115:6789/0 mon.b 2024-03-25T18:32:08.175 INFO:teuthology.misc.health.smithi001.stderr:2: v1:172.21.15.1:6790/0 mon.c 2024-03-25T18:32:08.175 INFO:teuthology.misc.health.smithi001.stderr: 2024-03-25T18:32:08.186 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:32:08.184+0000 7fc32dffb640 1 -- 172.21.15.1:0/346445085 <== mon.1 v1:172.21.15.115:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 2280638920 0 0) 0x7fc338143b60 con 0x7fc3380b4ff0 2024-03-25T18:32:08.186 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:32:08.184+0000 7fc32dffb640 1 -- 172.21.15.1:0/346445085 >> v1:172.21.15.1:6790/0 conn(0x7fc3380b4670 legacy=0x7fc338133d10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:32:08.186 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:32:08.184+0000 7fc32dffb640 1 -- 172.21.15.1:0/346445085 >> v1:172.21.15.1:6789/0 conn(0x7fc33813c480 legacy=0x7fc338132750 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:32:08.186 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:32:08.184+0000 7fc32dffb640 10 monclient: _finish_hunting 0 2024-03-25T18:32:08.186 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:32:08.184+0000 7fc32dffb640 1 monclient: found mon.b 2024-03-25T18:32:08.186 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:32:08.184+0000 7fc32dffb640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:32:08.186 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:32:08.184+0000 7fc32dffb640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.115:6789/0 2024-03-25T18:32:08.186 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:32:08.184+0000 7fc32dffb640 1 -- 172.21.15.1:0/346445085 --> v1:172.21.15.115:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc338133480 con 0x7fc3380b4ff0 2024-03-25T18:32:08.186 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:32:08.184+0000 7fc32dffb640 10 monclient: _finish_auth 0 2024-03-25T18:32:08.186 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:32:08.184+0000 7fc32dffb640 10 monclient: _check_auth_tickets 2024-03-25T18:32:08.186 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:32:08.184+0000 7fc32dffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:32:08.186 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:32:08.184+0000 7fc33f923640 5 monclient: authenticate success, global_id 4465 2024-03-25T18:32:08.187 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:32:08.184+0000 7fc33f923640 10 monclient: _renew_subs 2024-03-25T18:32:08.187 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:32:08.184+0000 7fc33f923640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.115:6789/0 2024-03-25T18:32:08.187 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:32:08.184+0000 7fc33f923640 1 -- 172.21.15.1:0/346445085 --> v1:172.21.15.115:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc338133730 con 0x7fc3380b4ff0 2024-03-25T18:32:08.187 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:32:08.184+0000 7fc33f923640 10 client.4465.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T18:32:08.187 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:32:08.184+0000 7fc33f923640 10 monclient: _renew_subs 2024-03-25T18:32:08.187 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:32:08.184+0000 7fc33f923640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.115:6789/0 2024-03-25T18:32:08.187 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:32:08.184+0000 7fc33f923640 1 -- 172.21.15.1:0/346445085 --> v1:172.21.15.115:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7fc3381e3860 con 0x7fc3380b4ff0 2024-03-25T18:32:08.187 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:32:08.184+0000 7fc33f923640 1 librados: init done 2024-03-25T18:32:08.187 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:32:08.185+0000 7fc33f923640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T18:32:08.188 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:32:08.185+0000 7fc33f923640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:32:08.188 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:32:08.185+0000 7fc33f923640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.115:6789/0 2024-03-25T18:32:08.188 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:32:08.185+0000 7fc33f923640 1 -- 172.21.15.1:0/346445085 --> v1:172.21.15.115:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc33806dbd0 con 0x7fc3380b4ff0 2024-03-25T18:32:08.408 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:32:08.407+0000 7fc32dffb640 1 -- 172.21.15.1:0/346445085 <== mon.1 v1:172.21.15.115:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fc328001c90 con 0x7fc3380b4ff0 2024-03-25T18:32:08.408 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:32:08.407+0000 7fc32dffb640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:32:08.408 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:32:08.407+0000 7fc32dffb640 1 -- 172.21.15.1:0/346445085 <== mon.1 v1:172.21.15.115:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1586965025 0 0) 0x7fc328003a60 con 0x7fc3380b4ff0 2024-03-25T18:32:08.408 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:32:08.407+0000 7fc32dffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:32:08.408 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:32:08.407+0000 7fc32dffb640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-25T18:32:08.408 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:32:08.407+0000 7fc32dffb640 10 monclient: dump: 2024-03-25T18:32:08.408 INFO:teuthology.misc.health.smithi001.stderr:epoch 1 2024-03-25T18:32:08.408 INFO:teuthology.misc.health.smithi001.stderr:fsid c8ea72cf-9e99-4383-9b52-6c23d365fdfc 2024-03-25T18:32:08.408 INFO:teuthology.misc.health.smithi001.stderr:last_changed 2024-03-25T18:25:50.510519+0000 2024-03-25T18:32:08.408 INFO:teuthology.misc.health.smithi001.stderr:created 2024-03-25T18:25:50.510519+0000 2024-03-25T18:32:08.408 INFO:teuthology.misc.health.smithi001.stderr:min_mon_release 19 (squid) 2024-03-25T18:32:08.409 INFO:teuthology.misc.health.smithi001.stderr:election_strategy: 3 2024-03-25T18:32:08.409 INFO:teuthology.misc.health.smithi001.stderr:0: v1:172.21.15.1:6789/0 mon.a 2024-03-25T18:32:08.409 INFO:teuthology.misc.health.smithi001.stderr:1: v1:172.21.15.115:6789/0 mon.b 2024-03-25T18:32:08.409 INFO:teuthology.misc.health.smithi001.stderr:2: v1:172.21.15.1:6790/0 mon.c 2024-03-25T18:32:08.409 INFO:teuthology.misc.health.smithi001.stderr: 2024-03-25T18:32:08.409 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:32:08.407+0000 7fc32f7fe640 4 set_mon_vals no callback set 2024-03-25T18:32:08.471 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:32:08.470+0000 7fc32dffb640 1 -- 172.21.15.1:0/346445085 <== mon.1 v1:172.21.15.115:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 21787975 0 0) 0x7fc328003bc0 con 0x7fc3380b4ff0 2024-03-25T18:32:08.471 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:32:08.471+0000 7fc33d6bf640 10 monclient: get_auth_request con 0x7fc31c06e0b0 auth_method 0 2024-03-25T18:32:08.472 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:32:08.471+0000 7fc32dffb640 10 client.4465.objecter ms_handle_connect 0x7fc31c06e0b0 2024-03-25T18:32:08.474 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:32:08.473+0000 7fc32dffb640 1 -- 172.21.15.1:0/346445085 <== mon.1 v1:172.21.15.115:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 996254230 0 0) 0x7fc328054830 con 0x7fc3380b4ff0 2024-03-25T18:32:08.474 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:32:08.473+0000 7fc32dffb640 10 client.4465.objecter ms_dispatch 0x7fc338000b90 osd_map(10..10 src has 1..10) v4 2024-03-25T18:32:08.474 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:32:08.473+0000 7fc32dffb640 3 client.4465.objecter handle_osd_map got epochs [10,10] > 0 2024-03-25T18:32:08.474 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:32:08.473+0000 7fc32dffb640 3 client.4465.objecter handle_osd_map decoding full epoch 10 2024-03-25T18:32:08.474 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:32:08.473+0000 7fc32dffb640 20 client.4465.objecter dump_active .. 0 homeless 2024-03-25T18:32:08.476 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:32:08.476+0000 7fc32dffb640 1 -- 172.21.15.1:0/346445085 <== mon.1 v1:172.21.15.115:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7fc328050340 con 0x7fc3380b4ff0 2024-03-25T18:32:08.477 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:32:08.476+0000 7fc32dffb640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:32:08.477 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:32:08.476+0000 7fc32dffb640 10 monclient: _finish_command 1 = system:0 2024-03-25T18:32:08.748 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:32:08.747+0000 7fc33f923640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:32:08.748 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:32:08.747+0000 7fc33f923640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:32:08.748 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:32:08.747+0000 7fc33f923640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.115:6789/0 2024-03-25T18:32:08.748 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:32:08.747+0000 7fc33f923640 1 -- 172.21.15.1:0/346445085 --> v1:172.21.15.115:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fc3380688c0 con 0x7fc3380b4ff0 2024-03-25T18:32:08.783 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:32:08.782+0000 7fc32dffb640 1 -- 172.21.15.1:0/346445085 <== mon.1 v1:172.21.15.115:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 4092213348) 0x7fc328052270 con 0x7fc3380b4ff0 2024-03-25T18:32:08.783 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:32:08.782+0000 7fc32dffb640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:32:08.783 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:32:08.782+0000 7fc32dffb640 10 monclient: _finish_command 2 = system:0 2024-03-25T18:32:08.783 INFO:teuthology.misc.health.smithi001.stdout: 2024-03-25T18:32:08.783 INFO:teuthology.misc.health.smithi001.stdout:{"epoch":10,"fsid":"c8ea72cf-9e99-4383-9b52-6c23d365fdfc","created":"2024-03-25T18:26:38.856638+0000","modified":"2024-03-25T18:26:44.224208+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T18:26:44.224208+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"b9f34922-3a84-49be-8426-807c1eceaefc","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"42bf306f-55e3-4634-9874-8b4b73ac6644","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"292cb23a-18a9-4207-a7cc-4d5727feaec0","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"45e0e9ab-898d-4673-9590-b4117bc5e84a","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"831a5a09-73ab-4a61-9872-088e76bdbc66","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"88cd903d-3b55-4840-9396-6e3cfe005282","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"89db7355-bc9d-46f4-8ab0-e8d9e744e026","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"cac47bee-0914-4f08-896f-348838882866","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T18:32:08.786 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:32:08.784+0000 7fc33f923640 10 librados: watch_flush enter 2024-03-25T18:32:08.786 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:32:08.784+0000 7fc33f923640 10 librados: watch_flush exit 2024-03-25T18:32:08.786 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:32:08.784+0000 7fc33f923640 20 client.4465.objecter shutdown clearing up homeless session... 2024-03-25T18:32:08.786 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:32:08.784+0000 7fc33f923640 10 client.4465.objecter successfully canceled tick 2024-03-25T18:32:08.786 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:32:08.784+0000 7fc33f923640 1 -- 172.21.15.1:0/346445085 >> v1:172.21.15.115:6800/4237379176 conn(0x7fc31c06e0b0 legacy=0x7fc31c070570 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:32:08.786 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:32:08.784+0000 7fc33f923640 10 monclient: shutdown 2024-03-25T18:32:08.786 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:32:08.784+0000 7fc33f923640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:32:08.786 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:32:08.784+0000 7fc33f923640 1 -- 172.21.15.1:0/346445085 >> v1:172.21.15.115:6789/0 conn(0x7fc3380b4ff0 legacy=0x7fc338132040 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:32:08.786 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:32:08.784+0000 7fc33f923640 1 -- 172.21.15.1:0/346445085 shutdown_connections 2024-03-25T18:32:08.787 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:32:08.784+0000 7fc33f923640 1 -- 172.21.15.1:0/346445085 >> 172.21.15.1:0/346445085 conn(0x7fc3380617a0 msgr2=0x7fc338061b80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:32:08.787 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:32:08.785+0000 7fc33f923640 1 -- 172.21.15.1:0/346445085 shutdown_connections 2024-03-25T18:32:08.787 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:32:08.785+0000 7fc33f923640 1 -- 172.21.15.1:0/346445085 wait complete. 2024-03-25T18:32:08.787 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:32:08.785+0000 7fc33f923640 1 librados: shutdown 2024-03-25T18:32:08.796 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T18:32:15.603 DEBUG:teuthology.orchestra.run.smithi001:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T18:32:15.719 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:32:15.717+0000 7f52ea34b640 10 monclient: get_monmap_and_config 2024-03-25T18:32:15.719 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:32:15.717+0000 7f52ea34b640 10 monclient: build_initial_monmap 2024-03-25T18:32:15.719 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:32:15.717+0000 7f52ea34b640 10 monclient: monmap: 2024-03-25T18:32:15.719 INFO:teuthology.misc.health.smithi001.stderr:epoch 0 2024-03-25T18:32:15.719 INFO:teuthology.misc.health.smithi001.stderr:fsid c8ea72cf-9e99-4383-9b52-6c23d365fdfc 2024-03-25T18:32:15.719 INFO:teuthology.misc.health.smithi001.stderr:last_changed 2024-03-25T18:32:15.718256+0000 2024-03-25T18:32:15.719 INFO:teuthology.misc.health.smithi001.stderr:created 2024-03-25T18:32:15.718256+0000 2024-03-25T18:32:15.720 INFO:teuthology.misc.health.smithi001.stderr:min_mon_release 0 (unknown) 2024-03-25T18:32:15.720 INFO:teuthology.misc.health.smithi001.stderr:election_strategy: 3 2024-03-25T18:32:15.720 INFO:teuthology.misc.health.smithi001.stderr:0: v1:172.21.15.1:6789/0 mon.noname-a 2024-03-25T18:32:15.720 INFO:teuthology.misc.health.smithi001.stderr:1: v1:172.21.15.115:6789/0 mon.noname-c 2024-03-25T18:32:15.720 INFO:teuthology.misc.health.smithi001.stderr:2: v1:172.21.15.1:6790/0 mon.noname-b 2024-03-25T18:32:15.720 INFO:teuthology.misc.health.smithi001.stderr: 2024-03-25T18:32:15.720 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:32:15.718+0000 7f52ea34b640 1 Processor -- start 2024-03-25T18:32:15.720 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:32:15.718+0000 7f52ea34b640 1 -- start start 2024-03-25T18:32:15.720 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:32:15.718+0000 7f52ea34b640 10 monclient: init 2024-03-25T18:32:15.721 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:32:15.719+0000 7f52ea34b640 10 monclient: _reopen_session rank -1 2024-03-25T18:32:15.721 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:32:15.719+0000 7f52ea34b640 10 monclient: _add_conns ranks=[1,2,0] 2024-03-25T18:32:15.721 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:32:15.719+0000 7f52ea34b640 10 monclient(hunting): picked mon.noname-c con 0x7f52e40b4ff0 addr v1:172.21.15.115:6789/0 2024-03-25T18:32:15.721 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:32:15.719+0000 7f52ea34b640 10 monclient(hunting): picked mon.noname-b con 0x7f52e40b4670 addr v1:172.21.15.1:6790/0 2024-03-25T18:32:15.721 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:32:15.719+0000 7f52ea34b640 10 monclient(hunting): picked mon.noname-a con 0x7f52e413c480 addr v1:172.21.15.1:6789/0 2024-03-25T18:32:15.721 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:32:15.719+0000 7f52ea34b640 1 -- --> v1:172.21.15.1:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f52e40ab0e0 con 0x7f52e413c480 2024-03-25T18:32:15.721 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:32:15.719+0000 7f52ea34b640 1 -- --> v1:172.21.15.115:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f52e40ab3c0 con 0x7f52e40b4ff0 2024-03-25T18:32:15.722 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:32:15.719+0000 7f52ea34b640 1 -- --> v1:172.21.15.1:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f52e40a9f30 con 0x7f52e40b4670 2024-03-25T18:32:15.722 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:32:15.719+0000 7f52ea34b640 10 monclient(hunting): _renew_subs 2024-03-25T18:32:15.722 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:32:15.719+0000 7f52ea34b640 10 monclient(hunting): authenticate will time out at 7816.188970s 2024-03-25T18:32:15.728 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:32:15.727+0000 7f52e37fe640 1 --1- >> v1:172.21.15.1:6790/0 conn(0x7f52e40b4670 0x7f52e41425c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.1:6790/0 says I am v1:172.21.15.1:41066/0 (socket says 172.21.15.1:41066) 2024-03-25T18:32:15.728 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:32:15.727+0000 7f52e37fe640 1 -- 172.21.15.1:0/1169649849 learned_addr learned my addr 172.21.15.1:0/1169649849 (peer_addr_for_me v1:172.21.15.1:0/0) 2024-03-25T18:32:15.740 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:32:15.739+0000 7f52e2ffd640 1 -- 172.21.15.1:0/1169649849 <== mon.0 v1:172.21.15.1:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3660646360 0 0) 0x7f52e40ab0e0 con 0x7f52e413c480 2024-03-25T18:32:15.740 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:32:15.739+0000 7f52e2ffd640 10 monclient(hunting): _negotiate 2024-03-25T18:32:15.740 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:32:15.739+0000 7f52e2ffd640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:32:15.740 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:32:15.739+0000 7f52e2ffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:32:15.740 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:32:15.739+0000 7f52e2ffd640 10 monclient(hunting): my global_id is 4455 2024-03-25T18:32:15.740 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:32:15.739+0000 7f52e2ffd640 1 -- 172.21.15.1:0/1169649849 --> v1:172.21.15.1:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f52cc003520 con 0x7f52e413c480 2024-03-25T18:32:15.744 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:32:15.743+0000 7f52e2ffd640 1 -- 172.21.15.1:0/1169649849 <== mon.1 v1:172.21.15.115:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1536048814 0 0) 0x7f52e40ab3c0 con 0x7f52e40b4ff0 2024-03-25T18:32:15.744 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:32:15.743+0000 7f52e2ffd640 10 monclient(hunting): _negotiate 2024-03-25T18:32:15.744 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:32:15.743+0000 7f52e2ffd640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:32:15.744 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:32:15.743+0000 7f52e2ffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:32:15.744 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:32:15.743+0000 7f52e2ffd640 10 monclient(hunting): my global_id is 4468 2024-03-25T18:32:15.745 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:32:15.743+0000 7f52e2ffd640 1 -- 172.21.15.1:0/1169649849 --> v1:172.21.15.115:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f52e40ab0e0 con 0x7f52e40b4ff0 2024-03-25T18:32:15.746 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:32:15.745+0000 7f52e2ffd640 1 -- 172.21.15.1:0/1169649849 <== mon.2 v1:172.21.15.1:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2055619682 0 0) 0x7f52e40a9f30 con 0x7f52e40b4670 2024-03-25T18:32:15.746 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:32:15.745+0000 7f52e2ffd640 10 monclient(hunting): _negotiate 2024-03-25T18:32:15.746 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:32:15.745+0000 7f52e2ffd640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:32:15.746 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:32:15.745+0000 7f52e2ffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:32:15.746 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:32:15.745+0000 7f52e2ffd640 10 monclient(hunting): my global_id is 4469 2024-03-25T18:32:15.746 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:32:15.745+0000 7f52e2ffd640 1 -- 172.21.15.1:0/1169649849 --> v1:172.21.15.1:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f52e40ab3c0 con 0x7f52e40b4670 2024-03-25T18:32:15.764 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:32:15.762+0000 7f52e2ffd640 1 -- 172.21.15.1:0/1169649849 <== mon.2 v1:172.21.15.1:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1153092295 0 0) 0x7f52e40ab3c0 con 0x7f52e40b4670 2024-03-25T18:32:15.764 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:32:15.763+0000 7f52e2ffd640 1 -- 172.21.15.1:0/1169649849 >> v1:172.21.15.115:6789/0 conn(0x7f52e40b4ff0 legacy=0x7f52e413bd70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:32:15.764 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:32:15.763+0000 7f52e2ffd640 1 -- 172.21.15.1:0/1169649849 >> v1:172.21.15.1:6789/0 conn(0x7f52e413c480 legacy=0x7f52e414dc00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:32:15.764 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:32:15.763+0000 7f52e2ffd640 10 monclient: _finish_hunting 0 2024-03-25T18:32:15.764 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:32:15.763+0000 7f52e2ffd640 1 monclient: found mon.noname-b 2024-03-25T18:32:15.764 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:32:15.763+0000 7f52e2ffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:32:15.764 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:32:15.763+0000 7f52e2ffd640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.1:6790/0 2024-03-25T18:32:15.764 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:32:15.763+0000 7f52e2ffd640 1 -- 172.21.15.1:0/1169649849 --> v1:172.21.15.1:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f52e40a9b70 con 0x7f52e40b4670 2024-03-25T18:32:15.764 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:32:15.763+0000 7f52e2ffd640 10 monclient: _finish_auth 0 2024-03-25T18:32:15.764 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:32:15.763+0000 7f52e2ffd640 10 monclient: _check_auth_tickets 2024-03-25T18:32:15.764 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:32:15.763+0000 7f52e2ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:32:15.765 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:32:15.763+0000 7f52e2ffd640 1 -- 172.21.15.1:0/1169649849 <== mon.2 v1:172.21.15.1:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1586965025 0 0) 0x7f52d80015b0 con 0x7f52e40b4670 2024-03-25T18:32:15.765 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:32:15.763+0000 7f52e2ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:32:15.765 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:32:15.763+0000 7f52e2ffd640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-03-25T18:32:15.765 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:32:15.763+0000 7f52e2ffd640 10 monclient: dump: 2024-03-25T18:32:15.765 INFO:teuthology.misc.health.smithi001.stderr:epoch 1 2024-03-25T18:32:15.765 INFO:teuthology.misc.health.smithi001.stderr:fsid c8ea72cf-9e99-4383-9b52-6c23d365fdfc 2024-03-25T18:32:15.765 INFO:teuthology.misc.health.smithi001.stderr:last_changed 2024-03-25T18:25:50.510519+0000 2024-03-25T18:32:15.765 INFO:teuthology.misc.health.smithi001.stderr:created 2024-03-25T18:25:50.510519+0000 2024-03-25T18:32:15.765 INFO:teuthology.misc.health.smithi001.stderr:min_mon_release 19 (squid) 2024-03-25T18:32:15.765 INFO:teuthology.misc.health.smithi001.stderr:election_strategy: 3 2024-03-25T18:32:15.765 INFO:teuthology.misc.health.smithi001.stderr:0: v1:172.21.15.1:6789/0 mon.a 2024-03-25T18:32:15.766 INFO:teuthology.misc.health.smithi001.stderr:1: v1:172.21.15.115:6789/0 mon.b 2024-03-25T18:32:15.766 INFO:teuthology.misc.health.smithi001.stderr:2: v1:172.21.15.1:6790/0 mon.c 2024-03-25T18:32:15.766 INFO:teuthology.misc.health.smithi001.stderr: 2024-03-25T18:32:15.766 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:32:15.763+0000 7f52ea34b640 5 monclient: authenticate success, global_id 4469 2024-03-25T18:32:15.766 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:32:15.763+0000 7f52ea34b640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T18:32:15.788 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:32:15.787+0000 7f52e2ffd640 1 -- 172.21.15.1:0/1169649849 <== mon.2 v1:172.21.15.1:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f52d8001710 con 0x7f52e40b4670 2024-03-25T18:32:15.788 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:32:15.787+0000 7f52e2ffd640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:32:15.788 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:32:15.787+0000 7f52e2ffd640 1 -- 172.21.15.1:0/1169649849 <== mon.2 v1:172.21.15.1:6790/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1586965025 0 0) 0x7f52d8001b60 con 0x7f52e40b4670 2024-03-25T18:32:15.788 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:32:15.787+0000 7f52ea34b640 10 monclient: get_monmap_and_config success 2024-03-25T18:32:15.788 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:32:15.787+0000 7f52ea34b640 4 set_mon_vals no callback set 2024-03-25T18:32:15.788 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:32:15.787+0000 7f52e2ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:32:15.788 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:32:15.787+0000 7f52e2ffd640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-25T18:32:15.788 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:32:15.787+0000 7f52e2ffd640 10 monclient: dump: 2024-03-25T18:32:15.788 INFO:teuthology.misc.health.smithi001.stderr:epoch 1 2024-03-25T18:32:15.788 INFO:teuthology.misc.health.smithi001.stderr:fsid c8ea72cf-9e99-4383-9b52-6c23d365fdfc 2024-03-25T18:32:15.788 INFO:teuthology.misc.health.smithi001.stderr:last_changed 2024-03-25T18:25:50.510519+0000 2024-03-25T18:32:15.789 INFO:teuthology.misc.health.smithi001.stderr:created 2024-03-25T18:25:50.510519+0000 2024-03-25T18:32:15.789 INFO:teuthology.misc.health.smithi001.stderr:min_mon_release 19 (squid) 2024-03-25T18:32:15.789 INFO:teuthology.misc.health.smithi001.stderr:election_strategy: 3 2024-03-25T18:32:15.789 INFO:teuthology.misc.health.smithi001.stderr:0: v1:172.21.15.1:6789/0 mon.a 2024-03-25T18:32:15.789 INFO:teuthology.misc.health.smithi001.stderr:1: v1:172.21.15.115:6789/0 mon.b 2024-03-25T18:32:15.789 INFO:teuthology.misc.health.smithi001.stderr:2: v1:172.21.15.1:6790/0 mon.c 2024-03-25T18:32:15.789 INFO:teuthology.misc.health.smithi001.stderr: 2024-03-25T18:32:15.790 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:32:15.788+0000 7f52ea34b640 10 monclient: shutdown 2024-03-25T18:32:15.790 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:32:15.788+0000 7f52ea34b640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:32:15.790 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:32:15.788+0000 7f52ea34b640 1 -- 172.21.15.1:0/1169649849 >> v1:172.21.15.1:6790/0 conn(0x7f52e40b4670 legacy=0x7f52e41425c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:32:15.790 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:32:15.788+0000 7f52ea34b640 1 -- 172.21.15.1:0/1169649849 shutdown_connections 2024-03-25T18:32:15.791 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:32:15.788+0000 7f52ea34b640 1 -- 172.21.15.1:0/1169649849 >> 172.21.15.1:0/1169649849 conn(0x7f52e40617a0 msgr2=0x7f52e4061bb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:32:15.791 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:32:15.788+0000 7f52ea34b640 1 -- 172.21.15.1:0/1169649849 shutdown_connections 2024-03-25T18:32:15.791 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:32:15.788+0000 7f52ea34b640 1 -- 172.21.15.1:0/1169649849 wait complete. 2024-03-25T18:32:15.791 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:32:15.789+0000 7f52ea34b640 10 monclient: build_initial_monmap 2024-03-25T18:32:15.791 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:32:15.789+0000 7f52ea34b640 10 monclient: monmap: 2024-03-25T18:32:15.791 INFO:teuthology.misc.health.smithi001.stderr:epoch 0 2024-03-25T18:32:15.791 INFO:teuthology.misc.health.smithi001.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T18:32:15.791 INFO:teuthology.misc.health.smithi001.stderr:last_changed 0.000000 2024-03-25T18:32:15.791 INFO:teuthology.misc.health.smithi001.stderr:created 0.000000 2024-03-25T18:32:15.791 INFO:teuthology.misc.health.smithi001.stderr:min_mon_release 0 (unknown) 2024-03-25T18:32:15.791 INFO:teuthology.misc.health.smithi001.stderr:election_strategy: 1 2024-03-25T18:32:15.791 INFO:teuthology.misc.health.smithi001.stderr:0: v1:172.21.15.1:6789/0 mon.noname-a 2024-03-25T18:32:15.791 INFO:teuthology.misc.health.smithi001.stderr:1: v1:172.21.15.115:6789/0 mon.noname-b 2024-03-25T18:32:15.791 INFO:teuthology.misc.health.smithi001.stderr:2: v1:172.21.15.1:6790/0 mon.noname-c 2024-03-25T18:32:15.792 INFO:teuthology.misc.health.smithi001.stderr: 2024-03-25T18:32:15.792 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:32:15.789+0000 7f52ea34b640 1 librados: starting msgr at 2024-03-25T18:32:15.792 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:32:15.789+0000 7f52ea34b640 1 librados: starting objecter 2024-03-25T18:32:15.792 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:32:15.789+0000 7f52ea34b640 1 Processor -- start 2024-03-25T18:32:15.792 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:32:15.790+0000 7f52ea34b640 1 -- start start 2024-03-25T18:32:15.792 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:32:15.790+0000 7f52ea34b640 1 librados: setting wanted keys 2024-03-25T18:32:15.792 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:32:15.790+0000 7f52ea34b640 1 librados: calling monclient init 2024-03-25T18:32:15.792 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:32:15.790+0000 7f52ea34b640 10 monclient: init 2024-03-25T18:32:15.793 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:32:15.790+0000 7f52ea34b640 10 monclient: _reopen_session rank -1 2024-03-25T18:32:15.793 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:32:15.790+0000 7f52ea34b640 10 monclient: _add_conns ranks=[1,0,2] 2024-03-25T18:32:15.793 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:32:15.790+0000 7f52ea34b640 10 monclient(hunting): picked mon.noname-b con 0x7f52e40b4670 addr v1:172.21.15.115:6789/0 2024-03-25T18:32:15.793 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:32:15.790+0000 7f52ea34b640 10 monclient(hunting): picked mon.noname-a con 0x7f52e40b4ff0 addr v1:172.21.15.1:6789/0 2024-03-25T18:32:15.794 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:32:15.790+0000 7f52ea34b640 10 monclient(hunting): picked mon.noname-c con 0x7f52e413c480 addr v1:172.21.15.1:6790/0 2024-03-25T18:32:15.794 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:32:15.790+0000 7f52ea34b640 1 -- --> v1:172.21.15.1:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f52e40b03d0 con 0x7f52e40b4ff0 2024-03-25T18:32:15.794 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:32:15.790+0000 7f52ea34b640 1 -- --> v1:172.21.15.115:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f52e40ae360 con 0x7f52e40b4670 2024-03-25T18:32:15.794 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:32:15.790+0000 7f52ea34b640 1 -- --> v1:172.21.15.1:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f52e40b0640 con 0x7f52e413c480 2024-03-25T18:32:15.794 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:32:15.790+0000 7f52ea34b640 10 monclient(hunting): _renew_subs 2024-03-25T18:32:15.794 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:32:15.790+0000 7f52ea34b640 10 monclient(hunting): authenticate will time out at 7816.260630s 2024-03-25T18:32:15.794 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:32:15.792+0000 7f52e3fff640 1 --1- >> v1:172.21.15.115:6789/0 conn(0x7f52e40b4670 0x7f52e40b10b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.115:6789/0 says I am v1:172.21.15.1:36904/0 (socket says 172.21.15.1:36904) 2024-03-25T18:32:15.794 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:32:15.792+0000 7f52e3fff640 1 -- 172.21.15.1:0/3072066538 learned_addr learned my addr 172.21.15.1:0/3072066538 (peer_addr_for_me v1:172.21.15.1:0/0) 2024-03-25T18:32:15.794 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:32:15.793+0000 7f52e0ff9640 10 client.?.objecter ms_handle_connect 0x7f52e40b4670 2024-03-25T18:32:15.794 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:32:15.793+0000 7f52e0ff9640 10 client.?.objecter resend_mon_ops 2024-03-25T18:32:15.795 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:32:15.793+0000 7f52e0ff9640 10 client.?.objecter ms_handle_connect 0x7f52e413c480 2024-03-25T18:32:15.795 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:32:15.793+0000 7f52e0ff9640 10 client.?.objecter resend_mon_ops 2024-03-25T18:32:15.829 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:32:15.828+0000 7f52e0ff9640 10 client.?.objecter ms_handle_connect 0x7f52e40b4ff0 2024-03-25T18:32:15.829 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:32:15.828+0000 7f52e0ff9640 10 client.?.objecter resend_mon_ops 2024-03-25T18:32:15.887 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:32:15.885+0000 7f52e0ff9640 1 -- 172.21.15.1:0/3072066538 <== mon.1 v1:172.21.15.115:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2172830304 0 0) 0x7f52e40ae360 con 0x7f52e40b4670 2024-03-25T18:32:15.887 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:32:15.885+0000 7f52e0ff9640 10 monclient(hunting): _negotiate 2024-03-25T18:32:15.887 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:32:15.885+0000 7f52e0ff9640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:32:15.887 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:32:15.885+0000 7f52e0ff9640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:32:15.887 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:32:15.885+0000 7f52e0ff9640 10 monclient(hunting): my global_id is 4471 2024-03-25T18:32:15.887 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:32:15.886+0000 7f52e0ff9640 1 -- 172.21.15.1:0/3072066538 --> v1:172.21.15.115:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f52d0004eb0 con 0x7f52e40b4670 2024-03-25T18:32:15.893 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:32:15.892+0000 7f52e0ff9640 1 -- 172.21.15.1:0/3072066538 <== mon.2 v1:172.21.15.1:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1403277592 0 0) 0x7f52e40b0640 con 0x7f52e413c480 2024-03-25T18:32:15.893 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:32:15.892+0000 7f52e0ff9640 10 monclient(hunting): _negotiate 2024-03-25T18:32:15.893 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:32:15.892+0000 7f52e0ff9640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:32:15.893 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:32:15.892+0000 7f52e0ff9640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:32:15.893 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:32:15.892+0000 7f52e0ff9640 10 monclient(hunting): my global_id is 4472 2024-03-25T18:32:15.893 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:32:15.892+0000 7f52e0ff9640 1 -- 172.21.15.1:0/3072066538 --> v1:172.21.15.1:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f52e40ae360 con 0x7f52e413c480 2024-03-25T18:32:15.897 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:32:15.896+0000 7f52e0ff9640 1 -- 172.21.15.1:0/3072066538 <== mon.0 v1:172.21.15.1:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1113770493 0 0) 0x7f52e40b03d0 con 0x7f52e40b4ff0 2024-03-25T18:32:15.897 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:32:15.896+0000 7f52e0ff9640 10 monclient(hunting): _negotiate 2024-03-25T18:32:15.897 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:32:15.896+0000 7f52e0ff9640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:32:15.897 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:32:15.896+0000 7f52e0ff9640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:32:15.897 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:32:15.896+0000 7f52e0ff9640 10 monclient(hunting): my global_id is 4458 2024-03-25T18:32:15.897 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:32:15.896+0000 7f52e0ff9640 1 -- 172.21.15.1:0/3072066538 --> v1:172.21.15.1:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f52e40b0640 con 0x7f52e40b4ff0 2024-03-25T18:32:15.907 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:32:15.906+0000 7f52e0ff9640 1 -- 172.21.15.1:0/3072066538 <== mon.2 v1:172.21.15.1:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3274311948 0 0) 0x7f52e40ae360 con 0x7f52e413c480 2024-03-25T18:32:15.907 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:32:15.906+0000 7f52e0ff9640 1 -- 172.21.15.1:0/3072066538 --> v1:172.21.15.1:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f52e40b03d0 con 0x7f52e413c480 2024-03-25T18:32:15.907 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:32:15.906+0000 7f52e0ff9640 1 -- 172.21.15.1:0/3072066538 <== mon.2 v1:172.21.15.1:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1586965025 0 0) 0x7f52dc001ab0 con 0x7f52e413c480 2024-03-25T18:32:15.907 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:32:15.906+0000 7f52e0ff9640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:32:15.907 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:32:15.906+0000 7f52e0ff9640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-03-25T18:32:15.907 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:32:15.906+0000 7f52e0ff9640 10 monclient(hunting): dump: 2024-03-25T18:32:15.907 INFO:teuthology.misc.health.smithi001.stderr:epoch 1 2024-03-25T18:32:15.907 INFO:teuthology.misc.health.smithi001.stderr:fsid c8ea72cf-9e99-4383-9b52-6c23d365fdfc 2024-03-25T18:32:15.908 INFO:teuthology.misc.health.smithi001.stderr:last_changed 2024-03-25T18:25:50.510519+0000 2024-03-25T18:32:15.908 INFO:teuthology.misc.health.smithi001.stderr:created 2024-03-25T18:25:50.510519+0000 2024-03-25T18:32:15.908 INFO:teuthology.misc.health.smithi001.stderr:min_mon_release 19 (squid) 2024-03-25T18:32:15.908 INFO:teuthology.misc.health.smithi001.stderr:election_strategy: 3 2024-03-25T18:32:15.908 INFO:teuthology.misc.health.smithi001.stderr:0: v1:172.21.15.1:6789/0 mon.a 2024-03-25T18:32:15.908 INFO:teuthology.misc.health.smithi001.stderr:1: v1:172.21.15.115:6789/0 mon.b 2024-03-25T18:32:15.908 INFO:teuthology.misc.health.smithi001.stderr:2: v1:172.21.15.1:6790/0 mon.c 2024-03-25T18:32:15.908 INFO:teuthology.misc.health.smithi001.stderr: 2024-03-25T18:32:15.908 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:32:15.906+0000 7f52e0ff9640 10 monclient(hunting): _finish_auth 0 2024-03-25T18:32:15.909 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:32:15.908+0000 7f52e0ff9640 1 -- 172.21.15.1:0/3072066538 <== mon.0 v1:172.21.15.1:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1742767525 0 0) 0x7f52e40b0640 con 0x7f52e40b4ff0 2024-03-25T18:32:15.909 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:32:15.908+0000 7f52e0ff9640 1 -- 172.21.15.1:0/3072066538 --> v1:172.21.15.1:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f52e40ae360 con 0x7f52e40b4ff0 2024-03-25T18:32:15.909 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:32:15.908+0000 7f52e0ff9640 1 -- 172.21.15.1:0/3072066538 <== mon.0 v1:172.21.15.1:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1586965025 0 0) 0x7f52d8001a20 con 0x7f52e40b4ff0 2024-03-25T18:32:15.909 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:32:15.908+0000 7f52e0ff9640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:32:15.909 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:32:15.908+0000 7f52e0ff9640 10 monclient(hunting): got monmap 1 from mon.a (according to old e1) 2024-03-25T18:32:15.909 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:32:15.908+0000 7f52e0ff9640 10 monclient(hunting): dump: 2024-03-25T18:32:15.910 INFO:teuthology.misc.health.smithi001.stderr:epoch 1 2024-03-25T18:32:15.910 INFO:teuthology.misc.health.smithi001.stderr:fsid c8ea72cf-9e99-4383-9b52-6c23d365fdfc 2024-03-25T18:32:15.910 INFO:teuthology.misc.health.smithi001.stderr:last_changed 2024-03-25T18:25:50.510519+0000 2024-03-25T18:32:15.910 INFO:teuthology.misc.health.smithi001.stderr:created 2024-03-25T18:25:50.510519+0000 2024-03-25T18:32:15.910 INFO:teuthology.misc.health.smithi001.stderr:min_mon_release 19 (squid) 2024-03-25T18:32:15.910 INFO:teuthology.misc.health.smithi001.stderr:election_strategy: 3 2024-03-25T18:32:15.910 INFO:teuthology.misc.health.smithi001.stderr:0: v1:172.21.15.1:6789/0 mon.a 2024-03-25T18:32:15.910 INFO:teuthology.misc.health.smithi001.stderr:1: v1:172.21.15.115:6789/0 mon.b 2024-03-25T18:32:15.910 INFO:teuthology.misc.health.smithi001.stderr:2: v1:172.21.15.1:6790/0 mon.c 2024-03-25T18:32:15.910 INFO:teuthology.misc.health.smithi001.stderr: 2024-03-25T18:32:15.917 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:32:15.916+0000 7f52e0ff9640 1 -- 172.21.15.1:0/3072066538 <== mon.1 v1:172.21.15.115:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1654179333 0 0) 0x7f52d0004eb0 con 0x7f52e40b4670 2024-03-25T18:32:15.917 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:32:15.916+0000 7f52e0ff9640 1 -- 172.21.15.1:0/3072066538 --> v1:172.21.15.115:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f52e40b0640 con 0x7f52e40b4670 2024-03-25T18:32:15.917 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:32:15.916+0000 7f52e0ff9640 1 -- 172.21.15.1:0/3072066538 <== mon.1 v1:172.21.15.115:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1586965025 0 0) 0x7f52d4001a10 con 0x7f52e40b4670 2024-03-25T18:32:15.918 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:32:15.916+0000 7f52e0ff9640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:32:15.918 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:32:15.916+0000 7f52e0ff9640 10 monclient(hunting): got monmap 1 from mon.b (according to old e1) 2024-03-25T18:32:15.918 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:32:15.916+0000 7f52e0ff9640 10 monclient(hunting): dump: 2024-03-25T18:32:15.918 INFO:teuthology.misc.health.smithi001.stderr:epoch 1 2024-03-25T18:32:15.918 INFO:teuthology.misc.health.smithi001.stderr:fsid c8ea72cf-9e99-4383-9b52-6c23d365fdfc 2024-03-25T18:32:15.918 INFO:teuthology.misc.health.smithi001.stderr:last_changed 2024-03-25T18:25:50.510519+0000 2024-03-25T18:32:15.918 INFO:teuthology.misc.health.smithi001.stderr:created 2024-03-25T18:25:50.510519+0000 2024-03-25T18:32:15.918 INFO:teuthology.misc.health.smithi001.stderr:min_mon_release 19 (squid) 2024-03-25T18:32:15.918 INFO:teuthology.misc.health.smithi001.stderr:election_strategy: 3 2024-03-25T18:32:15.918 INFO:teuthology.misc.health.smithi001.stderr:0: v1:172.21.15.1:6789/0 mon.a 2024-03-25T18:32:15.919 INFO:teuthology.misc.health.smithi001.stderr:1: v1:172.21.15.115:6789/0 mon.b 2024-03-25T18:32:15.919 INFO:teuthology.misc.health.smithi001.stderr:2: v1:172.21.15.1:6790/0 mon.c 2024-03-25T18:32:15.919 INFO:teuthology.misc.health.smithi001.stderr: 2024-03-25T18:32:15.924 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:32:15.922+0000 7f52e0ff9640 1 -- 172.21.15.1:0/3072066538 <== mon.0 v1:172.21.15.1:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 1640161193 0 0) 0x7f52e40ae360 con 0x7f52e40b4ff0 2024-03-25T18:32:15.924 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:32:15.923+0000 7f52e0ff9640 1 -- 172.21.15.1:0/3072066538 >> v1:172.21.15.1:6790/0 conn(0x7f52e413c480 legacy=0x7f52e40aef50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:32:15.924 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:32:15.923+0000 7f52e0ff9640 1 -- 172.21.15.1:0/3072066538 >> v1:172.21.15.115:6789/0 conn(0x7f52e40b4670 legacy=0x7f52e40b10b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:32:15.925 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:32:15.923+0000 7f52e0ff9640 10 monclient: _finish_hunting 0 2024-03-25T18:32:15.925 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:32:15.923+0000 7f52e0ff9640 1 monclient: found mon.a 2024-03-25T18:32:15.925 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:32:15.923+0000 7f52e0ff9640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:32:15.925 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:32:15.923+0000 7f52e0ff9640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.1:6789/0 2024-03-25T18:32:15.925 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:32:15.923+0000 7f52e0ff9640 1 -- 172.21.15.1:0/3072066538 --> v1:172.21.15.1:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f52e40b0920 con 0x7f52e40b4ff0 2024-03-25T18:32:15.925 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:32:15.923+0000 7f52e0ff9640 10 monclient: _finish_auth 0 2024-03-25T18:32:15.925 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:32:15.923+0000 7f52e0ff9640 10 monclient: _check_auth_tickets 2024-03-25T18:32:15.925 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:32:15.923+0000 7f52e0ff9640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:32:15.925 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:32:15.923+0000 7f52ea34b640 5 monclient: authenticate success, global_id 4458 2024-03-25T18:32:15.925 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:32:15.923+0000 7f52ea34b640 10 monclient: _renew_subs 2024-03-25T18:32:15.925 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:32:15.923+0000 7f52ea34b640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.1:6789/0 2024-03-25T18:32:15.925 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:32:15.923+0000 7f52ea34b640 1 -- 172.21.15.1:0/3072066538 --> v1:172.21.15.1:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f52e40b17c0 con 0x7f52e40b4ff0 2024-03-25T18:32:15.925 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:32:15.923+0000 7f52ea34b640 10 client.4458.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T18:32:15.925 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:32:15.923+0000 7f52ea34b640 10 monclient: _renew_subs 2024-03-25T18:32:15.926 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:32:15.923+0000 7f52ea34b640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.1:6789/0 2024-03-25T18:32:15.926 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:32:15.923+0000 7f52ea34b640 1 -- 172.21.15.1:0/3072066538 --> v1:172.21.15.1:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f52e41e3780 con 0x7f52e40b4ff0 2024-03-25T18:32:15.926 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:32:15.923+0000 7f52ea34b640 1 librados: init done 2024-03-25T18:32:15.926 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:32:15.924+0000 7f52ea34b640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T18:32:15.926 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:32:15.924+0000 7f52ea34b640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:32:15.926 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:32:15.924+0000 7f52ea34b640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.1:6789/0 2024-03-25T18:32:15.926 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:32:15.924+0000 7f52ea34b640 1 -- 172.21.15.1:0/3072066538 --> v1:172.21.15.1:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f52e406dbd0 con 0x7f52e40b4ff0 2024-03-25T18:32:16.244 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:32:16.243+0000 7f52e0ff9640 1 -- 172.21.15.1:0/3072066538 <== mon.0 v1:172.21.15.1:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f52d8001ba0 con 0x7f52e40b4ff0 2024-03-25T18:32:16.244 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:32:16.243+0000 7f52e0ff9640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:32:16.245 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:32:16.243+0000 7f52e0ff9640 1 -- 172.21.15.1:0/3072066538 <== mon.0 v1:172.21.15.1:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1586965025 0 0) 0x7f52d8004c30 con 0x7f52e40b4ff0 2024-03-25T18:32:16.245 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:32:16.243+0000 7f52e0ff9640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:32:16.245 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:32:16.243+0000 7f52e0ff9640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-25T18:32:16.245 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:32:16.243+0000 7f52e0ff9640 10 monclient: dump: 2024-03-25T18:32:16.245 INFO:teuthology.misc.health.smithi001.stderr:epoch 1 2024-03-25T18:32:16.245 INFO:teuthology.misc.health.smithi001.stderr:fsid c8ea72cf-9e99-4383-9b52-6c23d365fdfc 2024-03-25T18:32:16.245 INFO:teuthology.misc.health.smithi001.stderr:last_changed 2024-03-25T18:25:50.510519+0000 2024-03-25T18:32:16.245 INFO:teuthology.misc.health.smithi001.stderr:created 2024-03-25T18:25:50.510519+0000 2024-03-25T18:32:16.245 INFO:teuthology.misc.health.smithi001.stderr:min_mon_release 19 (squid) 2024-03-25T18:32:16.245 INFO:teuthology.misc.health.smithi001.stderr:election_strategy: 3 2024-03-25T18:32:16.245 INFO:teuthology.misc.health.smithi001.stderr:0: v1:172.21.15.1:6789/0 mon.a 2024-03-25T18:32:16.245 INFO:teuthology.misc.health.smithi001.stderr:1: v1:172.21.15.115:6789/0 mon.b 2024-03-25T18:32:16.245 INFO:teuthology.misc.health.smithi001.stderr:2: v1:172.21.15.1:6790/0 mon.c 2024-03-25T18:32:16.245 INFO:teuthology.misc.health.smithi001.stderr: 2024-03-25T18:32:16.246 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:32:16.243+0000 7f52e27fc640 4 set_mon_vals no callback set 2024-03-25T18:32:16.246 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:32:16.244+0000 7f52e0ff9640 1 -- 172.21.15.1:0/3072066538 <== mon.0 v1:172.21.15.1:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 21787975 0 0) 0x7f52d801ac00 con 0x7f52e40b4ff0 2024-03-25T18:32:16.246 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:32:16.245+0000 7f52e3fff640 10 monclient: get_auth_request con 0x7f52d006e160 auth_method 0 2024-03-25T18:32:16.246 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:32:16.245+0000 7f52e0ff9640 10 client.4458.objecter ms_handle_connect 0x7f52d006e160 2024-03-25T18:32:16.252 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:32:16.251+0000 7f52e0ff9640 1 -- 172.21.15.1:0/3072066538 <== mon.0 v1:172.21.15.1:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 996254230 0 0) 0x7f52d8054720 con 0x7f52e40b4ff0 2024-03-25T18:32:16.252 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:32:16.251+0000 7f52e0ff9640 10 client.4458.objecter ms_dispatch 0x7f52e4000b90 osd_map(10..10 src has 1..10) v4 2024-03-25T18:32:16.252 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:32:16.251+0000 7f52e0ff9640 3 client.4458.objecter handle_osd_map got epochs [10,10] > 0 2024-03-25T18:32:16.252 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:32:16.251+0000 7f52e0ff9640 3 client.4458.objecter handle_osd_map decoding full epoch 10 2024-03-25T18:32:16.252 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:32:16.251+0000 7f52e0ff9640 20 client.4458.objecter dump_active .. 0 homeless 2024-03-25T18:32:16.252 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:32:16.251+0000 7f52e0ff9640 1 -- 172.21.15.1:0/3072066538 <== mon.0 v1:172.21.15.1:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f52d8050060 con 0x7f52e40b4ff0 2024-03-25T18:32:16.253 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:32:16.251+0000 7f52e0ff9640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:32:16.253 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:32:16.251+0000 7f52e0ff9640 10 monclient: _finish_command 1 = system:0 2024-03-25T18:32:16.506 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:32:16.505+0000 7f52ea34b640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:32:16.506 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:32:16.505+0000 7f52ea34b640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:32:16.506 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:32:16.505+0000 7f52ea34b640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.1:6789/0 2024-03-25T18:32:16.506 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:32:16.505+0000 7f52ea34b640 1 -- 172.21.15.1:0/3072066538 --> v1:172.21.15.1:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f52e40ae4f0 con 0x7f52e40b4ff0 2024-03-25T18:32:16.529 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:32:16.528+0000 7f52e0ff9640 1 -- 172.21.15.1:0/3072066538 <== mon.0 v1:172.21.15.1:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 4092213348) 0x7f52d8052160 con 0x7f52e40b4ff0 2024-03-25T18:32:16.529 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:32:16.528+0000 7f52e0ff9640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:32:16.529 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:32:16.528+0000 7f52e0ff9640 10 monclient: _finish_command 2 = system:0 2024-03-25T18:32:16.530 INFO:teuthology.misc.health.smithi001.stdout: 2024-03-25T18:32:16.530 INFO:teuthology.misc.health.smithi001.stdout:{"epoch":10,"fsid":"c8ea72cf-9e99-4383-9b52-6c23d365fdfc","created":"2024-03-25T18:26:38.856638+0000","modified":"2024-03-25T18:26:44.224208+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T18:26:44.224208+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"b9f34922-3a84-49be-8426-807c1eceaefc","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"42bf306f-55e3-4634-9874-8b4b73ac6644","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"292cb23a-18a9-4207-a7cc-4d5727feaec0","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"45e0e9ab-898d-4673-9590-b4117bc5e84a","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"831a5a09-73ab-4a61-9872-088e76bdbc66","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"88cd903d-3b55-4840-9396-6e3cfe005282","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"89db7355-bc9d-46f4-8ab0-e8d9e744e026","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"cac47bee-0914-4f08-896f-348838882866","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T18:32:16.532 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:32:16.531+0000 7f52ea34b640 10 librados: watch_flush enter 2024-03-25T18:32:16.532 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:32:16.531+0000 7f52ea34b640 10 librados: watch_flush exit 2024-03-25T18:32:16.532 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:32:16.531+0000 7f52ea34b640 20 client.4458.objecter shutdown clearing up homeless session... 2024-03-25T18:32:16.532 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:32:16.531+0000 7f52ea34b640 10 client.4458.objecter successfully canceled tick 2024-03-25T18:32:16.532 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:32:16.531+0000 7f52ea34b640 1 -- 172.21.15.1:0/3072066538 >> v1:172.21.15.115:6800/4237379176 conn(0x7f52d006e160 legacy=0x7f52d0070620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:32:16.532 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:32:16.531+0000 7f52ea34b640 10 monclient: shutdown 2024-03-25T18:32:16.532 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:32:16.531+0000 7f52ea34b640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:32:16.532 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:32:16.531+0000 7f52ea34b640 1 -- 172.21.15.1:0/3072066538 >> v1:172.21.15.1:6789/0 conn(0x7f52e40b4ff0 legacy=0x7f52e40ae840 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:32:16.533 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:32:16.531+0000 7f52ea34b640 1 -- 172.21.15.1:0/3072066538 shutdown_connections 2024-03-25T18:32:16.533 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:32:16.531+0000 7f52ea34b640 1 -- 172.21.15.1:0/3072066538 >> 172.21.15.1:0/3072066538 conn(0x7f52e40617a0 msgr2=0x7f52e4061b80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:32:16.533 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:32:16.531+0000 7f52ea34b640 1 -- 172.21.15.1:0/3072066538 shutdown_connections 2024-03-25T18:32:16.533 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:32:16.531+0000 7f52ea34b640 1 -- 172.21.15.1:0/3072066538 wait complete. 2024-03-25T18:32:16.533 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:32:16.531+0000 7f52ea34b640 1 librados: shutdown 2024-03-25T18:32:16.546 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T18:32:22.071 INFO:tasks.ceph.mon.a.smithi001.stderr:==00:00:06:00.630 34876== Warning: unimplemented fcntl command: 1036 2024-03-25T18:32:22.078 INFO:tasks.ceph.mon.a.smithi001.stderr:==00:00:06:00.636 34876== Warning: unimplemented fcntl command: 1036 2024-03-25T18:32:22.103 INFO:tasks.ceph.mon.a.smithi001.stderr:==00:00:06:00.662 34876== Warning: unimplemented fcntl command: 1036 2024-03-25T18:32:22.127 INFO:tasks.ceph.mon.b.smithi115.stderr:==00:00:06:00.610 34836== Warning: unimplemented fcntl command: 1036 2024-03-25T18:32:22.131 INFO:tasks.ceph.mon.b.smithi115.stderr:==00:00:06:00.615 34836== Warning: unimplemented fcntl command: 1036 2024-03-25T18:32:22.132 INFO:tasks.ceph.mon.c.smithi001.stderr:==00:00:06:00.690 34877== Warning: unimplemented fcntl command: 1036 2024-03-25T18:32:22.143 INFO:tasks.ceph.mon.c.smithi001.stderr:==00:00:06:00.701 34877== Warning: unimplemented fcntl command: 1036 2024-03-25T18:32:22.168 INFO:tasks.ceph.mon.b.smithi115.stderr:==00:00:06:00.652 34836== Warning: unimplemented fcntl command: 1036 2024-03-25T18:32:22.187 INFO:tasks.ceph.mon.c.smithi001.stderr:==00:00:06:00.745 34877== Warning: unimplemented fcntl command: 1036 2024-03-25T18:32:23.352 DEBUG:teuthology.orchestra.run.smithi001:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T18:32:23.466 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:32:23.465+0000 7fe9c6cfb640 10 monclient: get_monmap_and_config 2024-03-25T18:32:23.466 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:32:23.465+0000 7fe9c6cfb640 10 monclient: build_initial_monmap 2024-03-25T18:32:23.466 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:32:23.465+0000 7fe9c6cfb640 10 monclient: monmap: 2024-03-25T18:32:23.466 INFO:teuthology.misc.health.smithi001.stderr:epoch 0 2024-03-25T18:32:23.466 INFO:teuthology.misc.health.smithi001.stderr:fsid c8ea72cf-9e99-4383-9b52-6c23d365fdfc 2024-03-25T18:32:23.466 INFO:teuthology.misc.health.smithi001.stderr:last_changed 2024-03-25T18:32:23.465720+0000 2024-03-25T18:32:23.466 INFO:teuthology.misc.health.smithi001.stderr:created 2024-03-25T18:32:23.465720+0000 2024-03-25T18:32:23.466 INFO:teuthology.misc.health.smithi001.stderr:min_mon_release 0 (unknown) 2024-03-25T18:32:23.467 INFO:teuthology.misc.health.smithi001.stderr:election_strategy: 3 2024-03-25T18:32:23.467 INFO:teuthology.misc.health.smithi001.stderr:0: v1:172.21.15.1:6789/0 mon.noname-a 2024-03-25T18:32:23.467 INFO:teuthology.misc.health.smithi001.stderr:1: v1:172.21.15.115:6789/0 mon.noname-c 2024-03-25T18:32:23.467 INFO:teuthology.misc.health.smithi001.stderr:2: v1:172.21.15.1:6790/0 mon.noname-b 2024-03-25T18:32:23.467 INFO:teuthology.misc.health.smithi001.stderr: 2024-03-25T18:32:23.468 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:32:23.466+0000 7fe9c6cfb640 1 Processor -- start 2024-03-25T18:32:23.468 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:32:23.466+0000 7fe9c6cfb640 1 -- start start 2024-03-25T18:32:23.468 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:32:23.466+0000 7fe9c6cfb640 10 monclient: init 2024-03-25T18:32:23.468 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:32:23.466+0000 7fe9c6cfb640 10 monclient: _reopen_session rank -1 2024-03-25T18:32:23.468 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:32:23.466+0000 7fe9c6cfb640 10 monclient: _add_conns ranks=[2,0,1] 2024-03-25T18:32:23.468 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:32:23.466+0000 7fe9c6cfb640 10 monclient(hunting): picked mon.noname-b con 0x7fe9c00b1a70 addr v1:172.21.15.1:6790/0 2024-03-25T18:32:23.468 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:32:23.466+0000 7fe9c6cfb640 10 monclient(hunting): picked mon.noname-a con 0x7fe9c0146490 addr v1:172.21.15.1:6789/0 2024-03-25T18:32:23.468 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:32:23.466+0000 7fe9c6cfb640 10 monclient(hunting): picked mon.noname-c con 0x7fe9c01458a0 addr v1:172.21.15.115:6789/0 2024-03-25T18:32:23.468 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:32:23.467+0000 7fe9c6cfb640 1 -- --> v1:172.21.15.1:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe9c0145d30 con 0x7fe9c0146490 2024-03-25T18:32:23.469 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:32:23.467+0000 7fe9c6cfb640 1 -- --> v1:172.21.15.115:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe9c013ffb0 con 0x7fe9c01458a0 2024-03-25T18:32:23.469 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:32:23.467+0000 7fe9c6cfb640 1 -- --> v1:172.21.15.1:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe9c00b26a0 con 0x7fe9c00b1a70 2024-03-25T18:32:23.469 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:32:23.467+0000 7fe9c6cfb640 10 monclient(hunting): _renew_subs 2024-03-25T18:32:23.469 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:32:23.467+0000 7fe9c6cfb640 10 monclient(hunting): authenticate will time out at 7823.936708s 2024-03-25T18:32:23.482 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:32:23.481+0000 7fe9c5298640 1 --1- >> v1:172.21.15.115:6789/0 conn(0x7fe9c01458a0 0x7fe9c014dc40 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.115:6789/0 says I am v1:172.21.15.1:43922/0 (socket says 172.21.15.1:43922) 2024-03-25T18:32:23.482 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:32:23.481+0000 7fe9c5298640 1 -- 172.21.15.1:0/293962622 learned_addr learned my addr 172.21.15.1:0/293962622 (peer_addr_for_me v1:172.21.15.1:0/0) 2024-03-25T18:32:23.501 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:32:23.500+0000 7fe9bf7fe640 1 -- 172.21.15.1:0/293962622 <== mon.0 v1:172.21.15.1:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4120366337 0 0) 0x7fe9ac001110 con 0x7fe9c0146490 2024-03-25T18:32:23.501 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:32:23.500+0000 7fe9bf7fe640 10 monclient(hunting): _negotiate 2024-03-25T18:32:23.501 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:32:23.500+0000 7fe9bf7fe640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:32:23.501 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:32:23.500+0000 7fe9bf7fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:32:23.501 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:32:23.500+0000 7fe9bf7fe640 10 monclient(hunting): my global_id is 4461 2024-03-25T18:32:23.502 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:32:23.500+0000 7fe9bf7fe640 1 -- 172.21.15.1:0/293962622 --> v1:172.21.15.1:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fe9b0003520 con 0x7fe9c0146490 2024-03-25T18:32:23.514 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:32:23.513+0000 7fe9bf7fe640 1 -- 172.21.15.1:0/293962622 <== mon.1 v1:172.21.15.115:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3429500610 0 0) 0x7fe9c013ffb0 con 0x7fe9c01458a0 2024-03-25T18:32:23.514 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:32:23.513+0000 7fe9bf7fe640 10 monclient(hunting): _negotiate 2024-03-25T18:32:23.514 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:32:23.513+0000 7fe9bf7fe640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:32:23.514 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:32:23.513+0000 7fe9bf7fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:32:23.514 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:32:23.513+0000 7fe9bf7fe640 10 monclient(hunting): my global_id is 4474 2024-03-25T18:32:23.514 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:32:23.513+0000 7fe9bf7fe640 1 -- 172.21.15.1:0/293962622 --> v1:172.21.15.115:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fe9ac001110 con 0x7fe9c01458a0 2024-03-25T18:32:23.523 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:32:23.522+0000 7fe9bf7fe640 1 -- 172.21.15.1:0/293962622 <== mon.2 v1:172.21.15.1:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3543751853 0 0) 0x7fe9c00b26a0 con 0x7fe9c00b1a70 2024-03-25T18:32:23.523 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:32:23.522+0000 7fe9bf7fe640 10 monclient(hunting): _negotiate 2024-03-25T18:32:23.523 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:32:23.522+0000 7fe9bf7fe640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:32:23.523 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:32:23.522+0000 7fe9bf7fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:32:23.523 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:32:23.522+0000 7fe9bf7fe640 10 monclient(hunting): my global_id is 4475 2024-03-25T18:32:23.523 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:32:23.522+0000 7fe9bf7fe640 1 -- 172.21.15.1:0/293962622 --> v1:172.21.15.1:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fe9c013ffb0 con 0x7fe9c00b1a70 2024-03-25T18:32:23.531 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:32:23.529+0000 7fe9bf7fe640 1 -- 172.21.15.1:0/293962622 <== mon.1 v1:172.21.15.115:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1850893081 0 0) 0x7fe9ac001110 con 0x7fe9c01458a0 2024-03-25T18:32:23.531 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:32:23.529+0000 7fe9bf7fe640 1 -- 172.21.15.1:0/293962622 >> v1:172.21.15.1:6790/0 conn(0x7fe9c00b1a70 legacy=0x7fe9c006bcb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:32:23.531 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:32:23.529+0000 7fe9bf7fe640 1 -- 172.21.15.1:0/293962622 >> v1:172.21.15.1:6789/0 conn(0x7fe9c0146490 legacy=0x7fe9c00afd20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:32:23.531 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:32:23.529+0000 7fe9bf7fe640 10 monclient: _finish_hunting 0 2024-03-25T18:32:23.531 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:32:23.529+0000 7fe9bf7fe640 1 monclient: found mon.noname-c 2024-03-25T18:32:23.531 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:32:23.529+0000 7fe9bf7fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:32:23.531 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:32:23.529+0000 7fe9bf7fe640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.115:6789/0 2024-03-25T18:32:23.531 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:32:23.529+0000 7fe9bf7fe640 1 -- 172.21.15.1:0/293962622 --> v1:172.21.15.115:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe9c00ae310 con 0x7fe9c01458a0 2024-03-25T18:32:23.531 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:32:23.529+0000 7fe9bf7fe640 10 monclient: _finish_auth 0 2024-03-25T18:32:23.531 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:32:23.529+0000 7fe9bf7fe640 10 monclient: _check_auth_tickets 2024-03-25T18:32:23.531 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:32:23.529+0000 7fe9bf7fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:32:23.531 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:32:23.529+0000 7fe9bf7fe640 1 -- 172.21.15.1:0/293962622 <== mon.1 v1:172.21.15.115:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1586965025 0 0) 0x7fe9b80019f0 con 0x7fe9c01458a0 2024-03-25T18:32:23.531 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:32:23.529+0000 7fe9bf7fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:32:23.531 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:32:23.529+0000 7fe9bf7fe640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-03-25T18:32:23.532 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:32:23.529+0000 7fe9bf7fe640 10 monclient: dump: 2024-03-25T18:32:23.532 INFO:teuthology.misc.health.smithi001.stderr:epoch 1 2024-03-25T18:32:23.532 INFO:teuthology.misc.health.smithi001.stderr:fsid c8ea72cf-9e99-4383-9b52-6c23d365fdfc 2024-03-25T18:32:23.532 INFO:teuthology.misc.health.smithi001.stderr:last_changed 2024-03-25T18:25:50.510519+0000 2024-03-25T18:32:23.532 INFO:teuthology.misc.health.smithi001.stderr:created 2024-03-25T18:25:50.510519+0000 2024-03-25T18:32:23.532 INFO:teuthology.misc.health.smithi001.stderr:min_mon_release 19 (squid) 2024-03-25T18:32:23.532 INFO:teuthology.misc.health.smithi001.stderr:election_strategy: 3 2024-03-25T18:32:23.532 INFO:teuthology.misc.health.smithi001.stderr:0: v1:172.21.15.1:6789/0 mon.a 2024-03-25T18:32:23.532 INFO:teuthology.misc.health.smithi001.stderr:1: v1:172.21.15.115:6789/0 mon.b 2024-03-25T18:32:23.532 INFO:teuthology.misc.health.smithi001.stderr:2: v1:172.21.15.1:6790/0 mon.c 2024-03-25T18:32:23.532 INFO:teuthology.misc.health.smithi001.stderr: 2024-03-25T18:32:23.532 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:32:23.530+0000 7fe9c6cfb640 5 monclient: authenticate success, global_id 4474 2024-03-25T18:32:23.532 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:32:23.530+0000 7fe9c6cfb640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T18:32:23.549 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:32:23.548+0000 7fe9bf7fe640 1 -- 172.21.15.1:0/293962622 <== mon.1 v1:172.21.15.115:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fe9b8001b50 con 0x7fe9c01458a0 2024-03-25T18:32:23.549 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:32:23.548+0000 7fe9bf7fe640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:32:23.549 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:32:23.548+0000 7fe9bf7fe640 1 -- 172.21.15.1:0/293962622 <== mon.1 v1:172.21.15.115:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1586965025 0 0) 0x7fe9b8001f60 con 0x7fe9c01458a0 2024-03-25T18:32:23.549 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:32:23.548+0000 7fe9c6cfb640 10 monclient: get_monmap_and_config success 2024-03-25T18:32:23.549 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:32:23.548+0000 7fe9c6cfb640 4 set_mon_vals no callback set 2024-03-25T18:32:23.549 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:32:23.548+0000 7fe9bf7fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:32:23.549 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:32:23.548+0000 7fe9bf7fe640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-25T18:32:23.549 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:32:23.548+0000 7fe9bf7fe640 10 monclient: dump: 2024-03-25T18:32:23.549 INFO:teuthology.misc.health.smithi001.stderr:epoch 1 2024-03-25T18:32:23.549 INFO:teuthology.misc.health.smithi001.stderr:fsid c8ea72cf-9e99-4383-9b52-6c23d365fdfc 2024-03-25T18:32:23.550 INFO:teuthology.misc.health.smithi001.stderr:last_changed 2024-03-25T18:25:50.510519+0000 2024-03-25T18:32:23.550 INFO:teuthology.misc.health.smithi001.stderr:created 2024-03-25T18:25:50.510519+0000 2024-03-25T18:32:23.550 INFO:teuthology.misc.health.smithi001.stderr:min_mon_release 19 (squid) 2024-03-25T18:32:23.550 INFO:teuthology.misc.health.smithi001.stderr:election_strategy: 3 2024-03-25T18:32:23.550 INFO:teuthology.misc.health.smithi001.stderr:0: v1:172.21.15.1:6789/0 mon.a 2024-03-25T18:32:23.550 INFO:teuthology.misc.health.smithi001.stderr:1: v1:172.21.15.115:6789/0 mon.b 2024-03-25T18:32:23.550 INFO:teuthology.misc.health.smithi001.stderr:2: v1:172.21.15.1:6790/0 mon.c 2024-03-25T18:32:23.550 INFO:teuthology.misc.health.smithi001.stderr: 2024-03-25T18:32:23.551 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:32:23.549+0000 7fe9c6cfb640 10 monclient: shutdown 2024-03-25T18:32:23.552 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:32:23.549+0000 7fe9c6cfb640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:32:23.552 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:32:23.549+0000 7fe9c6cfb640 1 -- 172.21.15.1:0/293962622 >> v1:172.21.15.115:6789/0 conn(0x7fe9c01458a0 legacy=0x7fe9c014dc40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:32:23.552 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:32:23.549+0000 7fe9c6cfb640 1 -- 172.21.15.1:0/293962622 shutdown_connections 2024-03-25T18:32:23.552 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:32:23.549+0000 7fe9c6cfb640 1 -- 172.21.15.1:0/293962622 >> 172.21.15.1:0/293962622 conn(0x7fe9c00b28f0 msgr2=0x7fe9c00b2d00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:32:23.552 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:32:23.549+0000 7fe9c6cfb640 1 -- 172.21.15.1:0/293962622 shutdown_connections 2024-03-25T18:32:23.552 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:32:23.549+0000 7fe9c6cfb640 1 -- 172.21.15.1:0/293962622 wait complete. 2024-03-25T18:32:23.552 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:32:23.550+0000 7fe9c6cfb640 10 monclient: build_initial_monmap 2024-03-25T18:32:23.552 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:32:23.550+0000 7fe9c6cfb640 10 monclient: monmap: 2024-03-25T18:32:23.552 INFO:teuthology.misc.health.smithi001.stderr:epoch 0 2024-03-25T18:32:23.552 INFO:teuthology.misc.health.smithi001.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T18:32:23.552 INFO:teuthology.misc.health.smithi001.stderr:last_changed 0.000000 2024-03-25T18:32:23.552 INFO:teuthology.misc.health.smithi001.stderr:created 0.000000 2024-03-25T18:32:23.552 INFO:teuthology.misc.health.smithi001.stderr:min_mon_release 0 (unknown) 2024-03-25T18:32:23.552 INFO:teuthology.misc.health.smithi001.stderr:election_strategy: 1 2024-03-25T18:32:23.553 INFO:teuthology.misc.health.smithi001.stderr:0: v1:172.21.15.1:6789/0 mon.noname-a 2024-03-25T18:32:23.553 INFO:teuthology.misc.health.smithi001.stderr:1: v1:172.21.15.115:6789/0 mon.noname-b 2024-03-25T18:32:23.553 INFO:teuthology.misc.health.smithi001.stderr:2: v1:172.21.15.1:6790/0 mon.noname-c 2024-03-25T18:32:23.553 INFO:teuthology.misc.health.smithi001.stderr: 2024-03-25T18:32:23.553 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:32:23.550+0000 7fe9c6cfb640 1 librados: starting msgr at 2024-03-25T18:32:23.553 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:32:23.550+0000 7fe9c6cfb640 1 librados: starting objecter 2024-03-25T18:32:23.553 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:32:23.551+0000 7fe9c6cfb640 1 Processor -- start 2024-03-25T18:32:23.553 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:32:23.551+0000 7fe9c6cfb640 1 -- start start 2024-03-25T18:32:23.553 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:32:23.551+0000 7fe9c6cfb640 1 librados: setting wanted keys 2024-03-25T18:32:23.553 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:32:23.551+0000 7fe9c6cfb640 1 librados: calling monclient init 2024-03-25T18:32:23.553 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:32:23.551+0000 7fe9c6cfb640 10 monclient: init 2024-03-25T18:32:23.554 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:32:23.551+0000 7fe9c6cfb640 10 monclient: _reopen_session rank -1 2024-03-25T18:32:23.554 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:32:23.551+0000 7fe9c6cfb640 10 monclient: _add_conns ranks=[0,1,2] 2024-03-25T18:32:23.555 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:32:23.551+0000 7fe9c6cfb640 10 monclient(hunting): picked mon.noname-a con 0x7fe9c00b1a70 addr v1:172.21.15.1:6789/0 2024-03-25T18:32:23.555 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:32:23.551+0000 7fe9c6cfb640 10 monclient(hunting): picked mon.noname-b con 0x7fe9c01458a0 addr v1:172.21.15.115:6789/0 2024-03-25T18:32:23.555 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:32:23.551+0000 7fe9c6cfb640 10 monclient(hunting): picked mon.noname-c con 0x7fe9c0146490 addr v1:172.21.15.1:6790/0 2024-03-25T18:32:23.555 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:32:23.551+0000 7fe9c6cfb640 1 -- --> v1:172.21.15.1:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe9c011dc90 con 0x7fe9c00b1a70 2024-03-25T18:32:23.555 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:32:23.551+0000 7fe9c6cfb640 1 -- --> v1:172.21.15.115:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe9c011df70 con 0x7fe9c01458a0 2024-03-25T18:32:23.555 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:32:23.551+0000 7fe9c6cfb640 1 -- --> v1:172.21.15.1:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe9c0142610 con 0x7fe9c0146490 2024-03-25T18:32:23.555 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:32:23.551+0000 7fe9c6cfb640 10 monclient(hunting): _renew_subs 2024-03-25T18:32:23.555 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:32:23.551+0000 7fe9c6cfb640 10 monclient(hunting): authenticate will time out at 7824.021638s 2024-03-25T18:32:23.562 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:32:23.561+0000 7fe9bffff640 1 --1- >> v1:172.21.15.115:6789/0 conn(0x7fe9c01458a0 0x7fe9c013abf0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.115:6789/0 says I am v1:172.21.15.1:44008/0 (socket says 172.21.15.1:44008) 2024-03-25T18:32:23.562 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:32:23.561+0000 7fe9bffff640 1 -- 172.21.15.1:0/377903075 learned_addr learned my addr 172.21.15.1:0/377903075 (peer_addr_for_me v1:172.21.15.1:0/0) 2024-03-25T18:32:23.564 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:32:23.563+0000 7fe9bd7fa640 10 client.?.objecter ms_handle_connect 0x7fe9c01458a0 2024-03-25T18:32:23.564 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:32:23.563+0000 7fe9bd7fa640 10 client.?.objecter resend_mon_ops 2024-03-25T18:32:23.572 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:32:23.571+0000 7fe9bd7fa640 10 client.?.objecter ms_handle_connect 0x7fe9c0146490 2024-03-25T18:32:23.572 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:32:23.571+0000 7fe9bd7fa640 10 client.?.objecter resend_mon_ops 2024-03-25T18:32:23.580 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:32:23.579+0000 7fe9bd7fa640 1 -- 172.21.15.1:0/377903075 <== mon.1 v1:172.21.15.115:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1113172250 0 0) 0x7fe9c011df70 con 0x7fe9c01458a0 2024-03-25T18:32:23.580 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:32:23.579+0000 7fe9bd7fa640 10 monclient(hunting): _negotiate 2024-03-25T18:32:23.580 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:32:23.579+0000 7fe9bd7fa640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:32:23.580 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:32:23.579+0000 7fe9bd7fa640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:32:23.580 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:32:23.579+0000 7fe9bd7fa640 10 monclient(hunting): my global_id is 4477 2024-03-25T18:32:23.580 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:32:23.579+0000 7fe9bd7fa640 1 -- 172.21.15.1:0/377903075 --> v1:172.21.15.115:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fe9a4004eb0 con 0x7fe9c01458a0 2024-03-25T18:32:23.581 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:32:23.581+0000 7fe9bd7fa640 10 client.?.objecter ms_handle_connect 0x7fe9c00b1a70 2024-03-25T18:32:23.581 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:32:23.581+0000 7fe9bd7fa640 10 client.?.objecter resend_mon_ops 2024-03-25T18:32:23.590 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:32:23.590+0000 7fe9bd7fa640 1 -- 172.21.15.1:0/377903075 <== mon.2 v1:172.21.15.1:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3400471967 0 0) 0x7fe9c0142610 con 0x7fe9c0146490 2024-03-25T18:32:23.590 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:32:23.590+0000 7fe9bd7fa640 10 monclient(hunting): _negotiate 2024-03-25T18:32:23.591 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:32:23.590+0000 7fe9bd7fa640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:32:23.591 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:32:23.590+0000 7fe9bd7fa640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:32:23.591 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:32:23.590+0000 7fe9bd7fa640 10 monclient(hunting): my global_id is 4478 2024-03-25T18:32:23.591 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:32:23.590+0000 7fe9bd7fa640 1 -- 172.21.15.1:0/377903075 --> v1:172.21.15.1:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fe9c011df70 con 0x7fe9c0146490 2024-03-25T18:32:23.595 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:32:23.594+0000 7fe9bd7fa640 1 -- 172.21.15.1:0/377903075 <== mon.0 v1:172.21.15.1:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 35120619 0 0) 0x7fe9c011dc90 con 0x7fe9c00b1a70 2024-03-25T18:32:23.595 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:32:23.594+0000 7fe9bd7fa640 10 monclient(hunting): _negotiate 2024-03-25T18:32:23.595 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:32:23.594+0000 7fe9bd7fa640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:32:23.595 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:32:23.594+0000 7fe9bd7fa640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:32:23.595 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:32:23.594+0000 7fe9bd7fa640 10 monclient(hunting): my global_id is 4464 2024-03-25T18:32:23.595 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:32:23.594+0000 7fe9bd7fa640 1 -- 172.21.15.1:0/377903075 --> v1:172.21.15.1:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fe9c0142610 con 0x7fe9c00b1a70 2024-03-25T18:32:23.605 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:32:23.605+0000 7fe9bd7fa640 1 -- 172.21.15.1:0/377903075 <== mon.1 v1:172.21.15.115:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2324431765 0 0) 0x7fe9a4004eb0 con 0x7fe9c01458a0 2024-03-25T18:32:23.605 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:32:23.605+0000 7fe9bd7fa640 1 -- 172.21.15.1:0/377903075 --> v1:172.21.15.115:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fe9c011dc90 con 0x7fe9c01458a0 2024-03-25T18:32:23.606 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:32:23.605+0000 7fe9bd7fa640 1 -- 172.21.15.1:0/377903075 <== mon.1 v1:172.21.15.115:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1586965025 0 0) 0x7fe9ac001280 con 0x7fe9c01458a0 2024-03-25T18:32:23.606 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:32:23.605+0000 7fe9bd7fa640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:32:23.606 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:32:23.605+0000 7fe9bd7fa640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-03-25T18:32:23.606 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:32:23.605+0000 7fe9bd7fa640 10 monclient(hunting): dump: 2024-03-25T18:32:23.606 INFO:teuthology.misc.health.smithi001.stderr:epoch 1 2024-03-25T18:32:23.606 INFO:teuthology.misc.health.smithi001.stderr:fsid c8ea72cf-9e99-4383-9b52-6c23d365fdfc 2024-03-25T18:32:23.606 INFO:teuthology.misc.health.smithi001.stderr:last_changed 2024-03-25T18:25:50.510519+0000 2024-03-25T18:32:23.606 INFO:teuthology.misc.health.smithi001.stderr:created 2024-03-25T18:25:50.510519+0000 2024-03-25T18:32:23.606 INFO:teuthology.misc.health.smithi001.stderr:min_mon_release 19 (squid) 2024-03-25T18:32:23.606 INFO:teuthology.misc.health.smithi001.stderr:election_strategy: 3 2024-03-25T18:32:23.606 INFO:teuthology.misc.health.smithi001.stderr:0: v1:172.21.15.1:6789/0 mon.a 2024-03-25T18:32:23.606 INFO:teuthology.misc.health.smithi001.stderr:1: v1:172.21.15.115:6789/0 mon.b 2024-03-25T18:32:23.606 INFO:teuthology.misc.health.smithi001.stderr:2: v1:172.21.15.1:6790/0 mon.c 2024-03-25T18:32:23.606 INFO:teuthology.misc.health.smithi001.stderr: 2024-03-25T18:32:23.606 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:32:23.605+0000 7fe9bd7fa640 10 monclient(hunting): _finish_auth 0 2024-03-25T18:32:23.618 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:32:23.617+0000 7fe9bd7fa640 1 -- 172.21.15.1:0/377903075 <== mon.2 v1:172.21.15.1:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1593879281 0 0) 0x7fe9c011df70 con 0x7fe9c0146490 2024-03-25T18:32:23.618 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:32:23.617+0000 7fe9bd7fa640 1 -- 172.21.15.1:0/377903075 --> v1:172.21.15.1:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fe9a4004eb0 con 0x7fe9c0146490 2024-03-25T18:32:23.618 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:32:23.617+0000 7fe9bd7fa640 1 -- 172.21.15.1:0/377903075 <== mon.2 v1:172.21.15.1:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1586965025 0 0) 0x7fe9b8001e60 con 0x7fe9c0146490 2024-03-25T18:32:23.619 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:32:23.617+0000 7fe9bd7fa640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:32:23.619 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:32:23.617+0000 7fe9bd7fa640 10 monclient(hunting): got monmap 1 from mon.c (according to old e1) 2024-03-25T18:32:23.619 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:32:23.617+0000 7fe9bd7fa640 10 monclient(hunting): dump: 2024-03-25T18:32:23.619 INFO:teuthology.misc.health.smithi001.stderr:epoch 1 2024-03-25T18:32:23.619 INFO:teuthology.misc.health.smithi001.stderr:fsid c8ea72cf-9e99-4383-9b52-6c23d365fdfc 2024-03-25T18:32:23.619 INFO:teuthology.misc.health.smithi001.stderr:last_changed 2024-03-25T18:25:50.510519+0000 2024-03-25T18:32:23.619 INFO:teuthology.misc.health.smithi001.stderr:created 2024-03-25T18:25:50.510519+0000 2024-03-25T18:32:23.619 INFO:teuthology.misc.health.smithi001.stderr:min_mon_release 19 (squid) 2024-03-25T18:32:23.619 INFO:teuthology.misc.health.smithi001.stderr:election_strategy: 3 2024-03-25T18:32:23.619 INFO:teuthology.misc.health.smithi001.stderr:0: v1:172.21.15.1:6789/0 mon.a 2024-03-25T18:32:23.619 INFO:teuthology.misc.health.smithi001.stderr:1: v1:172.21.15.115:6789/0 mon.b 2024-03-25T18:32:23.619 INFO:teuthology.misc.health.smithi001.stderr:2: v1:172.21.15.1:6790/0 mon.c 2024-03-25T18:32:23.619 INFO:teuthology.misc.health.smithi001.stderr: 2024-03-25T18:32:23.624 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:32:23.623+0000 7fe9bd7fa640 1 -- 172.21.15.1:0/377903075 <== mon.1 v1:172.21.15.115:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 95145297 0 0) 0x7fe9c011dc90 con 0x7fe9c01458a0 2024-03-25T18:32:23.624 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:32:23.623+0000 7fe9bd7fa640 1 -- 172.21.15.1:0/377903075 >> v1:172.21.15.1:6790/0 conn(0x7fe9c0146490 legacy=0x7fe9c01e2ec0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:32:23.624 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:32:23.623+0000 7fe9bd7fa640 1 -- 172.21.15.1:0/377903075 >> v1:172.21.15.1:6789/0 conn(0x7fe9c00b1a70 legacy=0x7fe9c01d75a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:32:23.624 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:32:23.623+0000 7fe9bd7fa640 10 monclient: _finish_hunting 0 2024-03-25T18:32:23.624 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:32:23.623+0000 7fe9bd7fa640 1 monclient: found mon.b 2024-03-25T18:32:23.624 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:32:23.623+0000 7fe9bd7fa640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:32:23.624 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:32:23.623+0000 7fe9bd7fa640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.115:6789/0 2024-03-25T18:32:23.624 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:32:23.623+0000 7fe9bd7fa640 1 -- 172.21.15.1:0/377903075 --> v1:172.21.15.115:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe9c0142b30 con 0x7fe9c01458a0 2024-03-25T18:32:23.624 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:32:23.623+0000 7fe9bd7fa640 10 monclient: _finish_auth 0 2024-03-25T18:32:23.624 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:32:23.623+0000 7fe9bd7fa640 10 monclient: _check_auth_tickets 2024-03-25T18:32:23.624 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:32:23.623+0000 7fe9bd7fa640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:32:23.624 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:32:23.623+0000 7fe9c6cfb640 5 monclient: authenticate success, global_id 4477 2024-03-25T18:32:23.624 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:32:23.623+0000 7fe9c6cfb640 10 monclient: _renew_subs 2024-03-25T18:32:23.625 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:32:23.623+0000 7fe9c6cfb640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.115:6789/0 2024-03-25T18:32:23.625 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:32:23.623+0000 7fe9c6cfb640 1 -- 172.21.15.1:0/377903075 --> v1:172.21.15.115:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe9c01d7cb0 con 0x7fe9c01458a0 2024-03-25T18:32:23.625 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:32:23.623+0000 7fe9c6cfb640 10 client.4477.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T18:32:23.625 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:32:23.623+0000 7fe9c6cfb640 10 monclient: _renew_subs 2024-03-25T18:32:23.625 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:32:23.623+0000 7fe9c6cfb640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.115:6789/0 2024-03-25T18:32:23.625 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:32:23.623+0000 7fe9c6cfb640 1 -- 172.21.15.1:0/377903075 --> v1:172.21.15.115:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7fe9c013b300 con 0x7fe9c01458a0 2024-03-25T18:32:23.625 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:32:23.623+0000 7fe9c6cfb640 1 librados: init done 2024-03-25T18:32:23.625 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:32:23.624+0000 7fe9c6cfb640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T18:32:23.625 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:32:23.624+0000 7fe9c6cfb640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:32:23.626 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:32:23.624+0000 7fe9c6cfb640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.115:6789/0 2024-03-25T18:32:23.626 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:32:23.624+0000 7fe9c6cfb640 1 -- 172.21.15.1:0/377903075 --> v1:172.21.15.115:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe9c01427a0 con 0x7fe9c01458a0 2024-03-25T18:32:23.911 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:32:23.910+0000 7fe9bd7fa640 1 -- 172.21.15.1:0/377903075 <== mon.1 v1:172.21.15.115:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fe9ac001c30 con 0x7fe9c01458a0 2024-03-25T18:32:23.912 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:32:23.910+0000 7fe9bd7fa640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:32:23.912 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:32:23.910+0000 7fe9bd7fa640 1 -- 172.21.15.1:0/377903075 <== mon.1 v1:172.21.15.115:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1586965025 0 0) 0x7fe9ac0037b0 con 0x7fe9c01458a0 2024-03-25T18:32:23.912 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:32:23.910+0000 7fe9bd7fa640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:32:23.912 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:32:23.910+0000 7fe9bd7fa640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-25T18:32:23.912 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:32:23.910+0000 7fe9bd7fa640 10 monclient: dump: 2024-03-25T18:32:23.912 INFO:teuthology.misc.health.smithi001.stderr:epoch 1 2024-03-25T18:32:23.912 INFO:teuthology.misc.health.smithi001.stderr:fsid c8ea72cf-9e99-4383-9b52-6c23d365fdfc 2024-03-25T18:32:23.912 INFO:teuthology.misc.health.smithi001.stderr:last_changed 2024-03-25T18:25:50.510519+0000 2024-03-25T18:32:23.912 INFO:teuthology.misc.health.smithi001.stderr:created 2024-03-25T18:25:50.510519+0000 2024-03-25T18:32:23.912 INFO:teuthology.misc.health.smithi001.stderr:min_mon_release 19 (squid) 2024-03-25T18:32:23.912 INFO:teuthology.misc.health.smithi001.stderr:election_strategy: 3 2024-03-25T18:32:23.913 INFO:teuthology.misc.health.smithi001.stderr:0: v1:172.21.15.1:6789/0 mon.a 2024-03-25T18:32:23.913 INFO:teuthology.misc.health.smithi001.stderr:1: v1:172.21.15.115:6789/0 mon.b 2024-03-25T18:32:23.913 INFO:teuthology.misc.health.smithi001.stderr:2: v1:172.21.15.1:6790/0 mon.c 2024-03-25T18:32:23.913 INFO:teuthology.misc.health.smithi001.stderr: 2024-03-25T18:32:23.913 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:32:23.910+0000 7fe9beffd640 4 set_mon_vals no callback set 2024-03-25T18:32:23.913 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:32:23.911+0000 7fe9bd7fa640 1 -- 172.21.15.1:0/377903075 <== mon.1 v1:172.21.15.115:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 21787975 0 0) 0x7fe9ac0039d0 con 0x7fe9c01458a0 2024-03-25T18:32:23.913 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:32:23.912+0000 7fe9c4a97640 10 monclient: get_auth_request con 0x7fe9a406df70 auth_method 0 2024-03-25T18:32:23.914 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:32:23.912+0000 7fe9bd7fa640 10 client.4477.objecter ms_handle_connect 0x7fe9a406df70 2024-03-25T18:32:23.916 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:32:23.916+0000 7fe9bd7fa640 1 -- 172.21.15.1:0/377903075 <== mon.1 v1:172.21.15.115:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 996254230 0 0) 0x7fe9ac0545a0 con 0x7fe9c01458a0 2024-03-25T18:32:23.917 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:32:23.916+0000 7fe9bd7fa640 10 client.4477.objecter ms_dispatch 0x7fe9c0000b90 osd_map(10..10 src has 1..10) v4 2024-03-25T18:32:23.917 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:32:23.916+0000 7fe9bd7fa640 3 client.4477.objecter handle_osd_map got epochs [10,10] > 0 2024-03-25T18:32:23.917 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:32:23.916+0000 7fe9bd7fa640 3 client.4477.objecter handle_osd_map decoding full epoch 10 2024-03-25T18:32:23.917 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:32:23.916+0000 7fe9bd7fa640 20 client.4477.objecter dump_active .. 0 homeless 2024-03-25T18:32:23.917 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:32:23.916+0000 7fe9bd7fa640 1 -- 172.21.15.1:0/377903075 <== mon.1 v1:172.21.15.115:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7fe9ac0500b0 con 0x7fe9c01458a0 2024-03-25T18:32:23.917 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:32:23.916+0000 7fe9bd7fa640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:32:23.917 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:32:23.916+0000 7fe9bd7fa640 10 monclient: _finish_command 1 = system:0 2024-03-25T18:32:24.173 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:32:24.172+0000 7fe9c6cfb640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:32:24.173 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:32:24.172+0000 7fe9c6cfb640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:32:24.173 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:32:24.172+0000 7fe9c6cfb640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.115:6789/0 2024-03-25T18:32:24.173 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:32:24.172+0000 7fe9c6cfb640 1 -- 172.21.15.1:0/377903075 --> v1:172.21.15.115:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fe9c00688c0 con 0x7fe9c01458a0 2024-03-25T18:32:24.206 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:32:24.205+0000 7fe9bd7fa640 1 -- 172.21.15.1:0/377903075 <== mon.1 v1:172.21.15.115:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 4092213348) 0x7fe9ac051fe0 con 0x7fe9c01458a0 2024-03-25T18:32:24.207 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:32:24.205+0000 7fe9bd7fa640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:32:24.207 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:32:24.205+0000 7fe9bd7fa640 10 monclient: _finish_command 2 = system:0 2024-03-25T18:32:24.207 INFO:teuthology.misc.health.smithi001.stdout: 2024-03-25T18:32:24.207 INFO:teuthology.misc.health.smithi001.stdout:{"epoch":10,"fsid":"c8ea72cf-9e99-4383-9b52-6c23d365fdfc","created":"2024-03-25T18:26:38.856638+0000","modified":"2024-03-25T18:26:44.224208+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T18:26:44.224208+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"b9f34922-3a84-49be-8426-807c1eceaefc","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"42bf306f-55e3-4634-9874-8b4b73ac6644","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"292cb23a-18a9-4207-a7cc-4d5727feaec0","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"45e0e9ab-898d-4673-9590-b4117bc5e84a","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"831a5a09-73ab-4a61-9872-088e76bdbc66","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"88cd903d-3b55-4840-9396-6e3cfe005282","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"89db7355-bc9d-46f4-8ab0-e8d9e744e026","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"cac47bee-0914-4f08-896f-348838882866","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T18:32:24.209 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:32:24.208+0000 7fe9c6cfb640 10 librados: watch_flush enter 2024-03-25T18:32:24.209 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:32:24.208+0000 7fe9c6cfb640 10 librados: watch_flush exit 2024-03-25T18:32:24.210 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:32:24.208+0000 7fe9c6cfb640 20 client.4477.objecter shutdown clearing up homeless session... 2024-03-25T18:32:24.210 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:32:24.208+0000 7fe9c6cfb640 10 client.4477.objecter successfully canceled tick 2024-03-25T18:32:24.210 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:32:24.208+0000 7fe9c6cfb640 1 -- 172.21.15.1:0/377903075 >> v1:172.21.15.115:6800/4237379176 conn(0x7fe9a406df70 legacy=0x7fe9a4070430 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:32:24.210 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:32:24.208+0000 7fe9c6cfb640 10 monclient: shutdown 2024-03-25T18:32:24.210 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:32:24.208+0000 7fe9c6cfb640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:32:24.210 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:32:24.208+0000 7fe9c6cfb640 1 -- 172.21.15.1:0/377903075 >> v1:172.21.15.115:6789/0 conn(0x7fe9c01458a0 legacy=0x7fe9c013abf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:32:24.210 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:32:24.208+0000 7fe9c6cfb640 1 -- 172.21.15.1:0/377903075 shutdown_connections 2024-03-25T18:32:24.210 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:32:24.208+0000 7fe9c6cfb640 1 -- 172.21.15.1:0/377903075 >> 172.21.15.1:0/377903075 conn(0x7fe9c00b28f0 msgr2=0x7fe9c0074490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:32:24.210 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:32:24.208+0000 7fe9c6cfb640 1 -- 172.21.15.1:0/377903075 shutdown_connections 2024-03-25T18:32:24.210 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:32:24.208+0000 7fe9c6cfb640 1 -- 172.21.15.1:0/377903075 wait complete. 2024-03-25T18:32:24.210 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:32:24.208+0000 7fe9c6cfb640 1 librados: shutdown 2024-03-25T18:32:24.221 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T18:32:31.028 DEBUG:teuthology.orchestra.run.smithi001:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T18:32:31.135 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:32:31.134+0000 7f9a6e148640 10 monclient: get_monmap_and_config 2024-03-25T18:32:31.136 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:32:31.134+0000 7f9a6e148640 10 monclient: build_initial_monmap 2024-03-25T18:32:31.136 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:32:31.134+0000 7f9a6e148640 10 monclient: monmap: 2024-03-25T18:32:31.136 INFO:teuthology.misc.health.smithi001.stderr:epoch 0 2024-03-25T18:32:31.136 INFO:teuthology.misc.health.smithi001.stderr:fsid c8ea72cf-9e99-4383-9b52-6c23d365fdfc 2024-03-25T18:32:31.136 INFO:teuthology.misc.health.smithi001.stderr:last_changed 2024-03-25T18:32:31.135221+0000 2024-03-25T18:32:31.136 INFO:teuthology.misc.health.smithi001.stderr:created 2024-03-25T18:32:31.135221+0000 2024-03-25T18:32:31.136 INFO:teuthology.misc.health.smithi001.stderr:min_mon_release 0 (unknown) 2024-03-25T18:32:31.136 INFO:teuthology.misc.health.smithi001.stderr:election_strategy: 3 2024-03-25T18:32:31.136 INFO:teuthology.misc.health.smithi001.stderr:0: v1:172.21.15.1:6789/0 mon.noname-a 2024-03-25T18:32:31.136 INFO:teuthology.misc.health.smithi001.stderr:1: v1:172.21.15.115:6789/0 mon.noname-c 2024-03-25T18:32:31.136 INFO:teuthology.misc.health.smithi001.stderr:2: v1:172.21.15.1:6790/0 mon.noname-b 2024-03-25T18:32:31.136 INFO:teuthology.misc.health.smithi001.stderr: 2024-03-25T18:32:31.138 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:32:31.135+0000 7f9a6e148640 1 Processor -- start 2024-03-25T18:32:31.138 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:32:31.135+0000 7f9a6e148640 1 -- start start 2024-03-25T18:32:31.138 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:32:31.135+0000 7f9a6e148640 10 monclient: init 2024-03-25T18:32:31.138 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:32:31.136+0000 7f9a6e148640 10 monclient: _reopen_session rank -1 2024-03-25T18:32:31.138 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:32:31.136+0000 7f9a6e148640 10 monclient: _add_conns ranks=[2,1,0] 2024-03-25T18:32:31.138 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:32:31.136+0000 7f9a6e148640 10 monclient(hunting): picked mon.noname-b con 0x7f9a680615c0 addr v1:172.21.15.1:6790/0 2024-03-25T18:32:31.138 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:32:31.136+0000 7f9a6e148640 10 monclient(hunting): picked mon.noname-c con 0x7f9a680619a0 addr v1:172.21.15.115:6789/0 2024-03-25T18:32:31.138 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:32:31.136+0000 7f9a6e148640 10 monclient(hunting): picked mon.noname-a con 0x7f9a680aff00 addr v1:172.21.15.1:6789/0 2024-03-25T18:32:31.138 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:32:31.136+0000 7f9a6e148640 1 -- --> v1:172.21.15.1:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9a6806ef20 con 0x7f9a680aff00 2024-03-25T18:32:31.139 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:32:31.136+0000 7f9a6e148640 1 -- --> v1:172.21.15.115:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9a6806f200 con 0x7f9a680619a0 2024-03-25T18:32:31.139 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:32:31.136+0000 7f9a6e148640 1 -- --> v1:172.21.15.1:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9a6806f4e0 con 0x7f9a680615c0 2024-03-25T18:32:31.139 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:32:31.136+0000 7f9a6e148640 10 monclient(hunting): _renew_subs 2024-03-25T18:32:31.139 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:32:31.136+0000 7f9a6e148640 10 monclient(hunting): authenticate will time out at 7831.606466s 2024-03-25T18:32:31.144 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:32:31.143+0000 7f9a677fe640 1 --1- >> v1:172.21.15.1:6790/0 conn(0x7f9a680615c0 0x7f9a68071f50 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.1:6790/0 says I am v1:172.21.15.1:38416/0 (socket says 172.21.15.1:38416) 2024-03-25T18:32:31.144 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:32:31.143+0000 7f9a677fe640 1 -- 172.21.15.1:0/2996572945 learned_addr learned my addr 172.21.15.1:0/2996572945 (peer_addr_for_me v1:172.21.15.1:0/0) 2024-03-25T18:32:31.155 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:32:31.154+0000 7f9a667fc640 1 -- 172.21.15.1:0/2996572945 <== mon.2 v1:172.21.15.1:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2547131162 0 0) 0x7f9a6806f4e0 con 0x7f9a680615c0 2024-03-25T18:32:31.155 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:32:31.154+0000 7f9a667fc640 10 monclient(hunting): _negotiate 2024-03-25T18:32:31.155 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:32:31.154+0000 7f9a667fc640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:32:31.155 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:32:31.154+0000 7f9a667fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:32:31.155 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:32:31.154+0000 7f9a667fc640 10 monclient(hunting): my global_id is 4481 2024-03-25T18:32:31.155 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:32:31.154+0000 7f9a667fc640 1 -- 172.21.15.1:0/2996572945 --> v1:172.21.15.1:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f9a54003520 con 0x7f9a680615c0 2024-03-25T18:32:31.155 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:32:31.154+0000 7f9a667fc640 1 -- 172.21.15.1:0/2996572945 <== mon.1 v1:172.21.15.115:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 746962471 0 0) 0x7f9a6806f200 con 0x7f9a680619a0 2024-03-25T18:32:31.155 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:32:31.154+0000 7f9a667fc640 10 monclient(hunting): _negotiate 2024-03-25T18:32:31.155 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:32:31.154+0000 7f9a667fc640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:32:31.156 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:32:31.154+0000 7f9a667fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:32:31.156 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:32:31.154+0000 7f9a667fc640 10 monclient(hunting): my global_id is 4480 2024-03-25T18:32:31.156 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:32:31.154+0000 7f9a667fc640 1 -- 172.21.15.1:0/2996572945 --> v1:172.21.15.115:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f9a6806f4e0 con 0x7f9a680619a0 2024-03-25T18:32:31.163 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:32:31.162+0000 7f9a667fc640 1 -- 172.21.15.1:0/2996572945 <== mon.1 v1:172.21.15.115:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 324794997 0 0) 0x7f9a6806f4e0 con 0x7f9a680619a0 2024-03-25T18:32:31.164 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:32:31.162+0000 7f9a667fc640 1 -- 172.21.15.1:0/2996572945 >> v1:172.21.15.1:6790/0 conn(0x7f9a680615c0 legacy=0x7f9a68071f50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:32:31.164 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:32:31.162+0000 7f9a667fc640 1 -- 172.21.15.1:0/2996572945 >> v1:172.21.15.1:6789/0 conn(0x7f9a680aff00 legacy=0x7f9a6814dc30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:32:31.164 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:32:31.162+0000 7f9a667fc640 10 monclient: _finish_hunting 0 2024-03-25T18:32:31.164 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:32:31.162+0000 7f9a667fc640 1 monclient: found mon.noname-c 2024-03-25T18:32:31.164 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:32:31.162+0000 7f9a667fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:32:31.164 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:32:31.162+0000 7f9a667fc640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.115:6789/0 2024-03-25T18:32:31.164 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:32:31.162+0000 7f9a667fc640 1 -- 172.21.15.1:0/2996572945 --> v1:172.21.15.115:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9a6813fe50 con 0x7f9a680619a0 2024-03-25T18:32:31.164 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:32:31.162+0000 7f9a667fc640 10 monclient: _finish_auth 0 2024-03-25T18:32:31.164 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:32:31.162+0000 7f9a667fc640 10 monclient: _check_auth_tickets 2024-03-25T18:32:31.164 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:32:31.162+0000 7f9a667fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:32:31.164 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:32:31.162+0000 7f9a667fc640 1 -- 172.21.15.1:0/2996572945 <== mon.1 v1:172.21.15.115:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1586965025 0 0) 0x7f9a58001670 con 0x7f9a680619a0 2024-03-25T18:32:31.164 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:32:31.162+0000 7f9a667fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:32:31.164 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:32:31.162+0000 7f9a667fc640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-03-25T18:32:31.165 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:32:31.162+0000 7f9a667fc640 10 monclient: dump: 2024-03-25T18:32:31.165 INFO:teuthology.misc.health.smithi001.stderr:epoch 1 2024-03-25T18:32:31.165 INFO:teuthology.misc.health.smithi001.stderr:fsid c8ea72cf-9e99-4383-9b52-6c23d365fdfc 2024-03-25T18:32:31.165 INFO:teuthology.misc.health.smithi001.stderr:last_changed 2024-03-25T18:25:50.510519+0000 2024-03-25T18:32:31.165 INFO:teuthology.misc.health.smithi001.stderr:created 2024-03-25T18:25:50.510519+0000 2024-03-25T18:32:31.165 INFO:teuthology.misc.health.smithi001.stderr:min_mon_release 19 (squid) 2024-03-25T18:32:31.165 INFO:teuthology.misc.health.smithi001.stderr:election_strategy: 3 2024-03-25T18:32:31.165 INFO:teuthology.misc.health.smithi001.stderr:0: v1:172.21.15.1:6789/0 mon.a 2024-03-25T18:32:31.165 INFO:teuthology.misc.health.smithi001.stderr:1: v1:172.21.15.115:6789/0 mon.b 2024-03-25T18:32:31.165 INFO:teuthology.misc.health.smithi001.stderr:2: v1:172.21.15.1:6790/0 mon.c 2024-03-25T18:32:31.165 INFO:teuthology.misc.health.smithi001.stderr: 2024-03-25T18:32:31.165 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:32:31.162+0000 7f9a6e148640 5 monclient: authenticate success, global_id 4480 2024-03-25T18:32:31.165 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:32:31.162+0000 7f9a6e148640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T18:32:31.166 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:32:31.165+0000 7f9a667fc640 1 -- 172.21.15.1:0/2996572945 <== mon.1 v1:172.21.15.115:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f9a580017d0 con 0x7f9a680619a0 2024-03-25T18:32:31.166 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:32:31.165+0000 7f9a667fc640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:32:31.166 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:32:31.165+0000 7f9a667fc640 1 -- 172.21.15.1:0/2996572945 <== mon.1 v1:172.21.15.115:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1586965025 0 0) 0x7f9a58001c20 con 0x7f9a680619a0 2024-03-25T18:32:31.167 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:32:31.165+0000 7f9a667fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:32:31.167 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:32:31.165+0000 7f9a667fc640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-25T18:32:31.167 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:32:31.165+0000 7f9a667fc640 10 monclient: dump: 2024-03-25T18:32:31.167 INFO:teuthology.misc.health.smithi001.stderr:epoch 1 2024-03-25T18:32:31.167 INFO:teuthology.misc.health.smithi001.stderr:fsid c8ea72cf-9e99-4383-9b52-6c23d365fdfc 2024-03-25T18:32:31.167 INFO:teuthology.misc.health.smithi001.stderr:last_changed 2024-03-25T18:25:50.510519+0000 2024-03-25T18:32:31.167 INFO:teuthology.misc.health.smithi001.stderr:created 2024-03-25T18:25:50.510519+0000 2024-03-25T18:32:31.167 INFO:teuthology.misc.health.smithi001.stderr:min_mon_release 19 (squid) 2024-03-25T18:32:31.167 INFO:teuthology.misc.health.smithi001.stderr:election_strategy: 3 2024-03-25T18:32:31.167 INFO:teuthology.misc.health.smithi001.stderr:0: v1:172.21.15.1:6789/0 mon.a 2024-03-25T18:32:31.167 INFO:teuthology.misc.health.smithi001.stderr:1: v1:172.21.15.115:6789/0 mon.b 2024-03-25T18:32:31.167 INFO:teuthology.misc.health.smithi001.stderr:2: v1:172.21.15.1:6790/0 mon.c 2024-03-25T18:32:31.167 INFO:teuthology.misc.health.smithi001.stderr: 2024-03-25T18:32:31.167 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:32:31.165+0000 7f9a6e148640 10 monclient: get_monmap_and_config success 2024-03-25T18:32:31.167 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:32:31.165+0000 7f9a6e148640 4 set_mon_vals no callback set 2024-03-25T18:32:31.170 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:32:31.166+0000 7f9a6e148640 10 monclient: shutdown 2024-03-25T18:32:31.170 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:32:31.166+0000 7f9a6e148640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:32:31.170 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:32:31.166+0000 7f9a6e148640 1 -- 172.21.15.1:0/2996572945 >> v1:172.21.15.115:6789/0 conn(0x7f9a680619a0 legacy=0x7f9a68145890 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:32:31.170 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:32:31.166+0000 7f9a6e148640 1 -- 172.21.15.1:0/2996572945 shutdown_connections 2024-03-25T18:32:31.170 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:32:31.166+0000 7f9a6e148640 1 -- 172.21.15.1:0/2996572945 >> 172.21.15.1:0/2996572945 conn(0x7f9a680b1020 msgr2=0x7f9a680b1430 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:32:31.170 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:32:31.166+0000 7f9a6e148640 1 -- 172.21.15.1:0/2996572945 shutdown_connections 2024-03-25T18:32:31.170 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:32:31.166+0000 7f9a6e148640 1 -- 172.21.15.1:0/2996572945 wait complete. 2024-03-25T18:32:31.170 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:32:31.167+0000 7f9a6e148640 10 monclient: build_initial_monmap 2024-03-25T18:32:31.170 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:32:31.167+0000 7f9a6e148640 10 monclient: monmap: 2024-03-25T18:32:31.170 INFO:teuthology.misc.health.smithi001.stderr:epoch 0 2024-03-25T18:32:31.171 INFO:teuthology.misc.health.smithi001.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T18:32:31.171 INFO:teuthology.misc.health.smithi001.stderr:last_changed 0.000000 2024-03-25T18:32:31.171 INFO:teuthology.misc.health.smithi001.stderr:created 0.000000 2024-03-25T18:32:31.171 INFO:teuthology.misc.health.smithi001.stderr:min_mon_release 0 (unknown) 2024-03-25T18:32:31.171 INFO:teuthology.misc.health.smithi001.stderr:election_strategy: 1 2024-03-25T18:32:31.171 INFO:teuthology.misc.health.smithi001.stderr:0: v1:172.21.15.1:6789/0 mon.noname-a 2024-03-25T18:32:31.171 INFO:teuthology.misc.health.smithi001.stderr:1: v1:172.21.15.115:6789/0 mon.noname-b 2024-03-25T18:32:31.171 INFO:teuthology.misc.health.smithi001.stderr:2: v1:172.21.15.1:6790/0 mon.noname-c 2024-03-25T18:32:31.171 INFO:teuthology.misc.health.smithi001.stderr: 2024-03-25T18:32:31.171 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:32:31.167+0000 7f9a6e148640 1 librados: starting msgr at 2024-03-25T18:32:31.171 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:32:31.167+0000 7f9a6e148640 1 librados: starting objecter 2024-03-25T18:32:31.171 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:32:31.167+0000 7f9a6e148640 1 Processor -- start 2024-03-25T18:32:31.171 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:32:31.167+0000 7f9a6e148640 1 -- start start 2024-03-25T18:32:31.171 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:32:31.167+0000 7f9a6e148640 1 librados: setting wanted keys 2024-03-25T18:32:31.171 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:32:31.167+0000 7f9a6e148640 1 librados: calling monclient init 2024-03-25T18:32:31.171 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:32:31.168+0000 7f9a6e148640 10 monclient: init 2024-03-25T18:32:31.172 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:32:31.168+0000 7f9a6e148640 10 monclient: _reopen_session rank -1 2024-03-25T18:32:31.172 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:32:31.168+0000 7f9a6e148640 10 monclient: _add_conns ranks=[2,1,0] 2024-03-25T18:32:31.172 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:32:31.168+0000 7f9a6e148640 10 monclient(hunting): picked mon.noname-c con 0x7f9a680615c0 addr v1:172.21.15.1:6790/0 2024-03-25T18:32:31.172 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:32:31.168+0000 7f9a6e148640 10 monclient(hunting): picked mon.noname-b con 0x7f9a680aff00 addr v1:172.21.15.115:6789/0 2024-03-25T18:32:31.172 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:32:31.168+0000 7f9a6e148640 10 monclient(hunting): picked mon.noname-a con 0x7f9a68142650 addr v1:172.21.15.1:6789/0 2024-03-25T18:32:31.172 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:32:31.168+0000 7f9a6e148640 1 -- --> v1:172.21.15.1:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9a6811cbb0 con 0x7f9a68142650 2024-03-25T18:32:31.172 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:32:31.168+0000 7f9a6e148640 1 -- --> v1:172.21.15.115:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9a681d8200 con 0x7f9a680aff00 2024-03-25T18:32:31.172 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:32:31.168+0000 7f9a6e148640 1 -- --> v1:172.21.15.1:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9a681d50d0 con 0x7f9a680615c0 2024-03-25T18:32:31.172 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:32:31.168+0000 7f9a6e148640 10 monclient(hunting): _renew_subs 2024-03-25T18:32:31.172 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:32:31.168+0000 7f9a6e148640 10 monclient(hunting): authenticate will time out at 7831.638435s 2024-03-25T18:32:31.180 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:32:31.179+0000 7f9a66ffd640 1 --1- >> v1:172.21.15.115:6789/0 conn(0x7f9a680aff00 0x7f9a6813d0e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.115:6789/0 says I am v1:172.21.15.1:48428/0 (socket says 172.21.15.1:48428) 2024-03-25T18:32:31.180 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:32:31.179+0000 7f9a66ffd640 1 -- 172.21.15.1:0/2740159156 learned_addr learned my addr 172.21.15.1:0/2740159156 (peer_addr_for_me v1:172.21.15.1:0/0) 2024-03-25T18:32:31.185 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:32:31.182+0000 7f9a5ffff640 10 client.?.objecter ms_handle_connect 0x7f9a680aff00 2024-03-25T18:32:31.185 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:32:31.182+0000 7f9a5ffff640 10 client.?.objecter resend_mon_ops 2024-03-25T18:32:31.186 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:32:31.183+0000 7f9a5ffff640 10 client.?.objecter ms_handle_connect 0x7f9a68142650 2024-03-25T18:32:31.186 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:32:31.183+0000 7f9a5ffff640 10 client.?.objecter resend_mon_ops 2024-03-25T18:32:31.186 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:32:31.183+0000 7f9a5ffff640 10 client.?.objecter ms_handle_connect 0x7f9a680615c0 2024-03-25T18:32:31.186 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:32:31.183+0000 7f9a5ffff640 10 client.?.objecter resend_mon_ops 2024-03-25T18:32:31.191 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:32:31.190+0000 7f9a5ffff640 1 -- 172.21.15.1:0/2740159156 <== mon.0 v1:172.21.15.1:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2185554084 0 0) 0x7f9a6811cbb0 con 0x7f9a68142650 2024-03-25T18:32:31.191 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:32:31.190+0000 7f9a5ffff640 10 monclient(hunting): _negotiate 2024-03-25T18:32:31.191 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:32:31.190+0000 7f9a5ffff640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:32:31.191 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:32:31.190+0000 7f9a5ffff640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:32:31.191 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:32:31.190+0000 7f9a5ffff640 10 monclient(hunting): my global_id is 4470 2024-03-25T18:32:31.191 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:32:31.190+0000 7f9a5ffff640 1 -- 172.21.15.1:0/2740159156 --> v1:172.21.15.1:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f9a48004eb0 con 0x7f9a68142650 2024-03-25T18:32:31.196 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:32:31.195+0000 7f9a5ffff640 1 -- 172.21.15.1:0/2740159156 <== mon.1 v1:172.21.15.115:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3099806417 0 0) 0x7f9a681d8200 con 0x7f9a680aff00 2024-03-25T18:32:31.196 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:32:31.195+0000 7f9a5ffff640 10 monclient(hunting): _negotiate 2024-03-25T18:32:31.196 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:32:31.195+0000 7f9a5ffff640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:32:31.196 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:32:31.195+0000 7f9a5ffff640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:32:31.196 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:32:31.195+0000 7f9a5ffff640 10 monclient(hunting): my global_id is 4483 2024-03-25T18:32:31.196 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:32:31.195+0000 7f9a5ffff640 1 -- 172.21.15.1:0/2740159156 --> v1:172.21.15.115:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f9a6811cbb0 con 0x7f9a680aff00 2024-03-25T18:32:31.196 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:32:31.195+0000 7f9a5ffff640 1 -- 172.21.15.1:0/2740159156 <== mon.2 v1:172.21.15.1:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 370613421 0 0) 0x7f9a681d50d0 con 0x7f9a680615c0 2024-03-25T18:32:31.196 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:32:31.195+0000 7f9a5ffff640 10 monclient(hunting): _negotiate 2024-03-25T18:32:31.196 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:32:31.195+0000 7f9a5ffff640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:32:31.197 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:32:31.195+0000 7f9a5ffff640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:32:31.197 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:32:31.195+0000 7f9a5ffff640 10 monclient(hunting): my global_id is 4484 2024-03-25T18:32:31.197 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:32:31.195+0000 7f9a5ffff640 1 -- 172.21.15.1:0/2740159156 --> v1:172.21.15.1:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f9a681d8200 con 0x7f9a680615c0 2024-03-25T18:32:31.208 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:32:31.207+0000 7f9a5ffff640 1 -- 172.21.15.1:0/2740159156 <== mon.0 v1:172.21.15.1:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3883013417 0 0) 0x7f9a48004eb0 con 0x7f9a68142650 2024-03-25T18:32:31.208 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:32:31.207+0000 7f9a5ffff640 1 -- 172.21.15.1:0/2740159156 --> v1:172.21.15.1:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f9a681d50d0 con 0x7f9a68142650 2024-03-25T18:32:31.208 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:32:31.207+0000 7f9a5ffff640 1 -- 172.21.15.1:0/2740159156 <== mon.0 v1:172.21.15.1:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1586965025 0 0) 0x7f9a60001840 con 0x7f9a68142650 2024-03-25T18:32:31.208 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:32:31.207+0000 7f9a5ffff640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:32:31.208 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:32:31.207+0000 7f9a5ffff640 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2024-03-25T18:32:31.208 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:32:31.207+0000 7f9a5ffff640 10 monclient(hunting): dump: 2024-03-25T18:32:31.208 INFO:teuthology.misc.health.smithi001.stderr:epoch 1 2024-03-25T18:32:31.208 INFO:teuthology.misc.health.smithi001.stderr:fsid c8ea72cf-9e99-4383-9b52-6c23d365fdfc 2024-03-25T18:32:31.208 INFO:teuthology.misc.health.smithi001.stderr:last_changed 2024-03-25T18:25:50.510519+0000 2024-03-25T18:32:31.208 INFO:teuthology.misc.health.smithi001.stderr:created 2024-03-25T18:25:50.510519+0000 2024-03-25T18:32:31.209 INFO:teuthology.misc.health.smithi001.stderr:min_mon_release 19 (squid) 2024-03-25T18:32:31.209 INFO:teuthology.misc.health.smithi001.stderr:election_strategy: 3 2024-03-25T18:32:31.209 INFO:teuthology.misc.health.smithi001.stderr:0: v1:172.21.15.1:6789/0 mon.a 2024-03-25T18:32:31.209 INFO:teuthology.misc.health.smithi001.stderr:1: v1:172.21.15.115:6789/0 mon.b 2024-03-25T18:32:31.209 INFO:teuthology.misc.health.smithi001.stderr:2: v1:172.21.15.1:6790/0 mon.c 2024-03-25T18:32:31.209 INFO:teuthology.misc.health.smithi001.stderr: 2024-03-25T18:32:31.209 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:32:31.207+0000 7f9a5ffff640 10 monclient(hunting): _finish_auth 0 2024-03-25T18:32:31.219 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:32:31.217+0000 7f9a5ffff640 1 -- 172.21.15.1:0/2740159156 <== mon.0 v1:172.21.15.1:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 45727351 0 0) 0x7f9a681d50d0 con 0x7f9a68142650 2024-03-25T18:32:31.219 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:32:31.217+0000 7f9a5ffff640 1 -- 172.21.15.1:0/2740159156 >> v1:172.21.15.1:6790/0 conn(0x7f9a680615c0 legacy=0x7f9a6813c9d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:32:31.219 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:32:31.217+0000 7f9a5ffff640 1 -- 172.21.15.1:0/2740159156 >> v1:172.21.15.115:6789/0 conn(0x7f9a680aff00 legacy=0x7f9a6813d0e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:32:31.219 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:32:31.217+0000 7f9a5ffff640 10 monclient: _finish_hunting 0 2024-03-25T18:32:31.219 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:32:31.217+0000 7f9a5ffff640 1 monclient: found mon.a 2024-03-25T18:32:31.219 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:32:31.217+0000 7f9a5ffff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:32:31.219 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:32:31.217+0000 7f9a5ffff640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.1:6789/0 2024-03-25T18:32:31.219 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:32:31.218+0000 7f9a5ffff640 1 -- 172.21.15.1:0/2740159156 --> v1:172.21.15.1:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9a681d7aa0 con 0x7f9a68142650 2024-03-25T18:32:31.219 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:32:31.218+0000 7f9a5ffff640 10 monclient: _finish_auth 0 2024-03-25T18:32:31.219 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:32:31.218+0000 7f9a5ffff640 10 monclient: _check_auth_tickets 2024-03-25T18:32:31.219 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:32:31.218+0000 7f9a5ffff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:32:31.219 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:32:31.218+0000 7f9a6e148640 5 monclient: authenticate success, global_id 4470 2024-03-25T18:32:31.220 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:32:31.218+0000 7f9a6e148640 10 monclient: _renew_subs 2024-03-25T18:32:31.220 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:32:31.218+0000 7f9a6e148640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.1:6789/0 2024-03-25T18:32:31.220 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:32:31.218+0000 7f9a6e148640 1 -- 172.21.15.1:0/2740159156 --> v1:172.21.15.1:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9a681d7c90 con 0x7f9a68142650 2024-03-25T18:32:31.220 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:32:31.218+0000 7f9a6e148640 10 client.4470.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T18:32:31.220 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:32:31.218+0000 7f9a6e148640 10 monclient: _renew_subs 2024-03-25T18:32:31.220 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:32:31.218+0000 7f9a6e148640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.1:6789/0 2024-03-25T18:32:31.220 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:32:31.218+0000 7f9a6e148640 1 -- 172.21.15.1:0/2740159156 --> v1:172.21.15.1:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f9a681e39b0 con 0x7f9a68142650 2024-03-25T18:32:31.220 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:32:31.218+0000 7f9a6e148640 1 librados: init done 2024-03-25T18:32:31.220 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:32:31.219+0000 7f9a6e148640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T18:32:31.221 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:32:31.219+0000 7f9a6e148640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:32:31.221 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:32:31.219+0000 7f9a6e148640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.1:6789/0 2024-03-25T18:32:31.221 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:32:31.219+0000 7f9a6e148640 1 -- 172.21.15.1:0/2740159156 --> v1:172.21.15.1:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f9a68145f20 con 0x7f9a68142650 2024-03-25T18:32:31.532 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:32:31.531+0000 7f9a5ffff640 1 -- 172.21.15.1:0/2740159156 <== mon.0 v1:172.21.15.1:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f9a60001d50 con 0x7f9a68142650 2024-03-25T18:32:31.532 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:32:31.531+0000 7f9a5ffff640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:32:31.532 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:32:31.531+0000 7f9a5ffff640 1 -- 172.21.15.1:0/2740159156 <== mon.0 v1:172.21.15.1:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1586965025 0 0) 0x7f9a60003790 con 0x7f9a68142650 2024-03-25T18:32:31.533 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:32:31.531+0000 7f9a5ffff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:32:31.533 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:32:31.531+0000 7f9a5ffff640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-25T18:32:31.533 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:32:31.531+0000 7f9a5ffff640 10 monclient: dump: 2024-03-25T18:32:31.533 INFO:teuthology.misc.health.smithi001.stderr:epoch 1 2024-03-25T18:32:31.533 INFO:teuthology.misc.health.smithi001.stderr:fsid c8ea72cf-9e99-4383-9b52-6c23d365fdfc 2024-03-25T18:32:31.533 INFO:teuthology.misc.health.smithi001.stderr:last_changed 2024-03-25T18:25:50.510519+0000 2024-03-25T18:32:31.533 INFO:teuthology.misc.health.smithi001.stderr:created 2024-03-25T18:25:50.510519+0000 2024-03-25T18:32:31.533 INFO:teuthology.misc.health.smithi001.stderr:min_mon_release 19 (squid) 2024-03-25T18:32:31.533 INFO:teuthology.misc.health.smithi001.stderr:election_strategy: 3 2024-03-25T18:32:31.533 INFO:teuthology.misc.health.smithi001.stderr:0: v1:172.21.15.1:6789/0 mon.a 2024-03-25T18:32:31.533 INFO:teuthology.misc.health.smithi001.stderr:1: v1:172.21.15.115:6789/0 mon.b 2024-03-25T18:32:31.533 INFO:teuthology.misc.health.smithi001.stderr:2: v1:172.21.15.1:6790/0 mon.c 2024-03-25T18:32:31.533 INFO:teuthology.misc.health.smithi001.stderr: 2024-03-25T18:32:31.533 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:32:31.531+0000 7f9a65ffb640 4 set_mon_vals no callback set 2024-03-25T18:32:31.534 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:32:31.532+0000 7f9a5ffff640 1 -- 172.21.15.1:0/2740159156 <== mon.0 v1:172.21.15.1:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 21787975 0 0) 0x7f9a600039b0 con 0x7f9a68142650 2024-03-25T18:32:31.534 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:32:31.533+0000 7f9a677fe640 10 monclient: get_auth_request con 0x7f9a4806dda0 auth_method 0 2024-03-25T18:32:31.534 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:32:31.533+0000 7f9a5ffff640 10 client.4470.objecter ms_handle_connect 0x7f9a4806dda0 2024-03-25T18:32:31.540 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:32:31.539+0000 7f9a5ffff640 1 -- 172.21.15.1:0/2740159156 <== mon.0 v1:172.21.15.1:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 996254230 0 0) 0x7f9a60053520 con 0x7f9a68142650 2024-03-25T18:32:31.540 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:32:31.539+0000 7f9a5ffff640 10 client.4470.objecter ms_dispatch 0x7f9a68000b90 osd_map(10..10 src has 1..10) v4 2024-03-25T18:32:31.540 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:32:31.539+0000 7f9a5ffff640 3 client.4470.objecter handle_osd_map got epochs [10,10] > 0 2024-03-25T18:32:31.540 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:32:31.539+0000 7f9a5ffff640 3 client.4470.objecter handle_osd_map decoding full epoch 10 2024-03-25T18:32:31.540 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:32:31.539+0000 7f9a5ffff640 20 client.4470.objecter dump_active .. 0 homeless 2024-03-25T18:32:31.540 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:32:31.539+0000 7f9a5ffff640 1 -- 172.21.15.1:0/2740159156 <== mon.0 v1:172.21.15.1:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f9a6004ee60 con 0x7f9a68142650 2024-03-25T18:32:31.540 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:32:31.539+0000 7f9a5ffff640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:32:31.540 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:32:31.539+0000 7f9a5ffff640 10 monclient: _finish_command 1 = system:0 2024-03-25T18:32:31.789 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:32:31.788+0000 7f9a6e148640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:32:31.790 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:32:31.788+0000 7f9a6e148640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:32:31.790 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:32:31.788+0000 7f9a6e148640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.1:6789/0 2024-03-25T18:32:31.790 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:32:31.788+0000 7f9a6e148640 1 -- 172.21.15.1:0/2740159156 --> v1:172.21.15.1:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f9a6806bd80 con 0x7f9a68142650 2024-03-25T18:32:31.819 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:32:31.818+0000 7f9a5ffff640 1 -- 172.21.15.1:0/2740159156 <== mon.0 v1:172.21.15.1:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 4092213348) 0x7f9a60050f60 con 0x7f9a68142650 2024-03-25T18:32:31.819 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:32:31.818+0000 7f9a5ffff640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:32:31.819 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:32:31.818+0000 7f9a5ffff640 10 monclient: _finish_command 2 = system:0 2024-03-25T18:32:31.820 INFO:teuthology.misc.health.smithi001.stdout: 2024-03-25T18:32:31.820 INFO:teuthology.misc.health.smithi001.stdout:{"epoch":10,"fsid":"c8ea72cf-9e99-4383-9b52-6c23d365fdfc","created":"2024-03-25T18:26:38.856638+0000","modified":"2024-03-25T18:26:44.224208+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T18:26:44.224208+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"b9f34922-3a84-49be-8426-807c1eceaefc","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"42bf306f-55e3-4634-9874-8b4b73ac6644","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"292cb23a-18a9-4207-a7cc-4d5727feaec0","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"45e0e9ab-898d-4673-9590-b4117bc5e84a","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"831a5a09-73ab-4a61-9872-088e76bdbc66","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"88cd903d-3b55-4840-9396-6e3cfe005282","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"89db7355-bc9d-46f4-8ab0-e8d9e744e026","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"cac47bee-0914-4f08-896f-348838882866","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T18:32:31.822 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:32:31.821+0000 7f9a6e148640 10 librados: watch_flush enter 2024-03-25T18:32:31.822 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:32:31.821+0000 7f9a6e148640 10 librados: watch_flush exit 2024-03-25T18:32:31.822 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:32:31.821+0000 7f9a6e148640 20 client.4470.objecter shutdown clearing up homeless session... 2024-03-25T18:32:31.822 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:32:31.821+0000 7f9a6e148640 10 client.4470.objecter successfully canceled tick 2024-03-25T18:32:31.822 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:32:31.821+0000 7f9a6e148640 1 -- 172.21.15.1:0/2740159156 >> v1:172.21.15.115:6800/4237379176 conn(0x7f9a4806dda0 legacy=0x7f9a48070260 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:32:31.823 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:32:31.821+0000 7f9a6e148640 10 monclient: shutdown 2024-03-25T18:32:31.823 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:32:31.821+0000 7f9a6e148640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:32:31.823 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:32:31.821+0000 7f9a6e148640 1 -- 172.21.15.1:0/2740159156 >> v1:172.21.15.1:6789/0 conn(0x7f9a68142650 legacy=0x7f9a681e20b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:32:31.823 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:32:31.821+0000 7f9a6e148640 1 -- 172.21.15.1:0/2740159156 shutdown_connections 2024-03-25T18:32:31.823 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:32:31.821+0000 7f9a6e148640 1 -- 172.21.15.1:0/2740159156 >> 172.21.15.1:0/2740159156 conn(0x7f9a680b1020 msgr2=0x7f9a680ad860 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:32:31.823 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:32:31.821+0000 7f9a6e148640 1 -- 172.21.15.1:0/2740159156 shutdown_connections 2024-03-25T18:32:31.823 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:32:31.821+0000 7f9a6e148640 1 -- 172.21.15.1:0/2740159156 wait complete. 2024-03-25T18:32:31.823 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:32:31.821+0000 7f9a6e148640 1 librados: shutdown 2024-03-25T18:32:31.835 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T18:32:38.641 DEBUG:teuthology.orchestra.run.smithi001:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T18:32:38.759 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:32:38.757+0000 7f398b5d6640 10 monclient: get_monmap_and_config 2024-03-25T18:32:38.760 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:32:38.757+0000 7f398b5d6640 10 monclient: build_initial_monmap 2024-03-25T18:32:38.760 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:32:38.758+0000 7f398b5d6640 10 monclient: monmap: 2024-03-25T18:32:38.760 INFO:teuthology.misc.health.smithi001.stderr:epoch 0 2024-03-25T18:32:38.760 INFO:teuthology.misc.health.smithi001.stderr:fsid c8ea72cf-9e99-4383-9b52-6c23d365fdfc 2024-03-25T18:32:38.760 INFO:teuthology.misc.health.smithi001.stderr:last_changed 2024-03-25T18:32:38.759560+0000 2024-03-25T18:32:38.760 INFO:teuthology.misc.health.smithi001.stderr:created 2024-03-25T18:32:38.759560+0000 2024-03-25T18:32:38.760 INFO:teuthology.misc.health.smithi001.stderr:min_mon_release 0 (unknown) 2024-03-25T18:32:38.760 INFO:teuthology.misc.health.smithi001.stderr:election_strategy: 3 2024-03-25T18:32:38.760 INFO:teuthology.misc.health.smithi001.stderr:0: v1:172.21.15.1:6789/0 mon.noname-a 2024-03-25T18:32:38.760 INFO:teuthology.misc.health.smithi001.stderr:1: v1:172.21.15.115:6789/0 mon.noname-c 2024-03-25T18:32:38.760 INFO:teuthology.misc.health.smithi001.stderr:2: v1:172.21.15.1:6790/0 mon.noname-b 2024-03-25T18:32:38.760 INFO:teuthology.misc.health.smithi001.stderr: 2024-03-25T18:32:38.762 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:32:38.758+0000 7f398b5d6640 1 Processor -- start 2024-03-25T18:32:38.762 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:32:38.758+0000 7f398b5d6640 1 -- start start 2024-03-25T18:32:38.762 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:32:38.758+0000 7f398b5d6640 10 monclient: init 2024-03-25T18:32:38.762 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:32:38.759+0000 7f398b5d6640 10 monclient: _reopen_session rank -1 2024-03-25T18:32:38.762 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:32:38.759+0000 7f398b5d6640 10 monclient: _add_conns ranks=[2,1,0] 2024-03-25T18:32:38.762 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:32:38.759+0000 7f398b5d6640 10 monclient(hunting): picked mon.noname-b con 0x7f39840b4ff0 addr v1:172.21.15.1:6790/0 2024-03-25T18:32:38.762 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:32:38.759+0000 7f398b5d6640 10 monclient(hunting): picked mon.noname-c con 0x7f39840b4670 addr v1:172.21.15.115:6789/0 2024-03-25T18:32:38.762 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:32:38.759+0000 7f398b5d6640 10 monclient(hunting): picked mon.noname-a con 0x7f398413c480 addr v1:172.21.15.1:6789/0 2024-03-25T18:32:38.762 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:32:38.759+0000 7f398b5d6640 1 -- --> v1:172.21.15.1:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f39840ab0e0 con 0x7f398413c480 2024-03-25T18:32:38.762 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:32:38.759+0000 7f398b5d6640 1 -- --> v1:172.21.15.115:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f39840ab3c0 con 0x7f39840b4670 2024-03-25T18:32:38.762 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:32:38.759+0000 7f398b5d6640 1 -- --> v1:172.21.15.1:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f39840a9f30 con 0x7f39840b4ff0 2024-03-25T18:32:38.762 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:32:38.759+0000 7f398b5d6640 10 monclient(hunting): _renew_subs 2024-03-25T18:32:38.762 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:32:38.759+0000 7f398b5d6640 10 monclient(hunting): authenticate will time out at 7839.230441s 2024-03-25T18:32:38.768 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:32:38.766+0000 7f3989372640 1 --1- >> v1:172.21.15.1:6790/0 conn(0x7f39840b4ff0 0x7f398413bd70 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.1:6790/0 says I am v1:172.21.15.1:34818/0 (socket says 172.21.15.1:34818) 2024-03-25T18:32:38.768 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:32:38.766+0000 7f3989372640 1 -- 172.21.15.1:0/4163652812 learned_addr learned my addr 172.21.15.1:0/4163652812 (peer_addr_for_me v1:172.21.15.1:0/0) 2024-03-25T18:32:38.783 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:32:38.782+0000 7f397bfff640 1 -- 172.21.15.1:0/4163652812 <== mon.2 v1:172.21.15.1:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1790278630 0 0) 0x7f39840a9f30 con 0x7f39840b4ff0 2024-03-25T18:32:38.784 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:32:38.782+0000 7f397bfff640 10 monclient(hunting): _negotiate 2024-03-25T18:32:38.784 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:32:38.782+0000 7f397bfff640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:32:38.784 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:32:38.782+0000 7f397bfff640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:32:38.784 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:32:38.782+0000 7f397bfff640 10 monclient(hunting): my global_id is 4487 2024-03-25T18:32:38.784 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:32:38.782+0000 7f397bfff640 1 -- 172.21.15.1:0/4163652812 --> v1:172.21.15.1:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f3970003520 con 0x7f39840b4ff0 2024-03-25T18:32:38.786 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:32:38.784+0000 7f397bfff640 1 -- 172.21.15.1:0/4163652812 <== mon.1 v1:172.21.15.115:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3655054254 0 0) 0x7f39840ab3c0 con 0x7f39840b4670 2024-03-25T18:32:38.786 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:32:38.784+0000 7f397bfff640 10 monclient(hunting): _negotiate 2024-03-25T18:32:38.787 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:32:38.784+0000 7f397bfff640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:32:38.787 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:32:38.784+0000 7f397bfff640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:32:38.787 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:32:38.784+0000 7f397bfff640 10 monclient(hunting): my global_id is 4486 2024-03-25T18:32:38.787 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:32:38.784+0000 7f397bfff640 1 -- 172.21.15.1:0/4163652812 --> v1:172.21.15.115:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f39840a9f30 con 0x7f39840b4670 2024-03-25T18:32:38.799 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:32:38.797+0000 7f397bfff640 1 -- 172.21.15.1:0/4163652812 <== mon.0 v1:172.21.15.1:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 739855172 0 0) 0x7f39840ab0e0 con 0x7f398413c480 2024-03-25T18:32:38.799 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:32:38.797+0000 7f397bfff640 10 monclient(hunting): _negotiate 2024-03-25T18:32:38.799 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:32:38.797+0000 7f397bfff640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:32:38.799 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:32:38.797+0000 7f397bfff640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:32:38.799 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:32:38.797+0000 7f397bfff640 10 monclient(hunting): my global_id is 4473 2024-03-25T18:32:38.799 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:32:38.797+0000 7f397bfff640 1 -- 172.21.15.1:0/4163652812 --> v1:172.21.15.1:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f39840ab3c0 con 0x7f398413c480 2024-03-25T18:32:38.805 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:32:38.803+0000 7f397bfff640 1 -- 172.21.15.1:0/4163652812 <== mon.2 v1:172.21.15.1:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 864854777 0 0) 0x7f3970003520 con 0x7f39840b4ff0 2024-03-25T18:32:38.805 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:32:38.803+0000 7f397bfff640 1 -- 172.21.15.1:0/4163652812 >> v1:172.21.15.115:6789/0 conn(0x7f39840b4670 legacy=0x7f39841425c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:32:38.805 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:32:38.803+0000 7f397bfff640 1 -- 172.21.15.1:0/4163652812 >> v1:172.21.15.1:6789/0 conn(0x7f398413c480 legacy=0x7f398414dc00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:32:38.806 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:32:38.803+0000 7f397bfff640 10 monclient: _finish_hunting 0 2024-03-25T18:32:38.806 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:32:38.803+0000 7f397bfff640 1 monclient: found mon.noname-b 2024-03-25T18:32:38.806 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:32:38.803+0000 7f397bfff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:32:38.806 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:32:38.803+0000 7f397bfff640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.1:6790/0 2024-03-25T18:32:38.806 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:32:38.803+0000 7f397bfff640 1 -- 172.21.15.1:0/4163652812 --> v1:172.21.15.1:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f39840a9b70 con 0x7f39840b4ff0 2024-03-25T18:32:38.806 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:32:38.803+0000 7f397bfff640 10 monclient: _finish_auth 0 2024-03-25T18:32:38.806 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:32:38.803+0000 7f397bfff640 10 monclient: _check_auth_tickets 2024-03-25T18:32:38.806 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:32:38.803+0000 7f397bfff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:32:38.806 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:32:38.803+0000 7f397bfff640 1 -- 172.21.15.1:0/4163652812 <== mon.2 v1:172.21.15.1:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1586965025 0 0) 0x7f3980001690 con 0x7f39840b4ff0 2024-03-25T18:32:38.806 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:32:38.803+0000 7f398b5d6640 5 monclient: authenticate success, global_id 4487 2024-03-25T18:32:38.806 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:32:38.803+0000 7f397bfff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:32:38.806 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:32:38.803+0000 7f397bfff640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-03-25T18:32:38.806 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:32:38.803+0000 7f397bfff640 10 monclient: dump: 2024-03-25T18:32:38.806 INFO:teuthology.misc.health.smithi001.stderr:epoch 1 2024-03-25T18:32:38.806 INFO:teuthology.misc.health.smithi001.stderr:fsid c8ea72cf-9e99-4383-9b52-6c23d365fdfc 2024-03-25T18:32:38.807 INFO:teuthology.misc.health.smithi001.stderr:last_changed 2024-03-25T18:25:50.510519+0000 2024-03-25T18:32:38.807 INFO:teuthology.misc.health.smithi001.stderr:created 2024-03-25T18:25:50.510519+0000 2024-03-25T18:32:38.807 INFO:teuthology.misc.health.smithi001.stderr:min_mon_release 19 (squid) 2024-03-25T18:32:38.807 INFO:teuthology.misc.health.smithi001.stderr:election_strategy: 3 2024-03-25T18:32:38.807 INFO:teuthology.misc.health.smithi001.stderr:0: v1:172.21.15.1:6789/0 mon.a 2024-03-25T18:32:38.807 INFO:teuthology.misc.health.smithi001.stderr:1: v1:172.21.15.115:6789/0 mon.b 2024-03-25T18:32:38.807 INFO:teuthology.misc.health.smithi001.stderr:2: v1:172.21.15.1:6790/0 mon.c 2024-03-25T18:32:38.807 INFO:teuthology.misc.health.smithi001.stderr: 2024-03-25T18:32:38.807 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:32:38.803+0000 7f398b5d6640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T18:32:38.820 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:32:38.817+0000 7f397bfff640 1 -- 172.21.15.1:0/4163652812 <== mon.2 v1:172.21.15.1:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f39800017f0 con 0x7f39840b4ff0 2024-03-25T18:32:38.820 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:32:38.818+0000 7f397bfff640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:32:38.820 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:32:38.818+0000 7f397bfff640 1 -- 172.21.15.1:0/4163652812 <== mon.2 v1:172.21.15.1:6790/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1586965025 0 0) 0x7f3980001c00 con 0x7f39840b4ff0 2024-03-25T18:32:38.820 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:32:38.818+0000 7f397bfff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:32:38.820 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:32:38.818+0000 7f397bfff640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-25T18:32:38.820 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:32:38.818+0000 7f397bfff640 10 monclient: dump: 2024-03-25T18:32:38.820 INFO:teuthology.misc.health.smithi001.stderr:epoch 1 2024-03-25T18:32:38.820 INFO:teuthology.misc.health.smithi001.stderr:fsid c8ea72cf-9e99-4383-9b52-6c23d365fdfc 2024-03-25T18:32:38.820 INFO:teuthology.misc.health.smithi001.stderr:last_changed 2024-03-25T18:25:50.510519+0000 2024-03-25T18:32:38.820 INFO:teuthology.misc.health.smithi001.stderr:created 2024-03-25T18:25:50.510519+0000 2024-03-25T18:32:38.820 INFO:teuthology.misc.health.smithi001.stderr:min_mon_release 19 (squid) 2024-03-25T18:32:38.820 INFO:teuthology.misc.health.smithi001.stderr:election_strategy: 3 2024-03-25T18:32:38.820 INFO:teuthology.misc.health.smithi001.stderr:0: v1:172.21.15.1:6789/0 mon.a 2024-03-25T18:32:38.820 INFO:teuthology.misc.health.smithi001.stderr:1: v1:172.21.15.115:6789/0 mon.b 2024-03-25T18:32:38.821 INFO:teuthology.misc.health.smithi001.stderr:2: v1:172.21.15.1:6790/0 mon.c 2024-03-25T18:32:38.821 INFO:teuthology.misc.health.smithi001.stderr: 2024-03-25T18:32:38.821 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:32:38.818+0000 7f398b5d6640 10 monclient: get_monmap_and_config success 2024-03-25T18:32:38.821 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:32:38.818+0000 7f398b5d6640 4 set_mon_vals no callback set 2024-03-25T18:32:38.822 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:32:38.818+0000 7f398b5d6640 10 monclient: shutdown 2024-03-25T18:32:38.822 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:32:38.818+0000 7f398b5d6640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:32:38.822 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:32:38.818+0000 7f398b5d6640 1 -- 172.21.15.1:0/4163652812 >> v1:172.21.15.1:6790/0 conn(0x7f39840b4ff0 legacy=0x7f398413bd70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:32:38.822 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:32:38.819+0000 7f398b5d6640 1 -- 172.21.15.1:0/4163652812 shutdown_connections 2024-03-25T18:32:38.822 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:32:38.819+0000 7f398b5d6640 1 -- 172.21.15.1:0/4163652812 >> 172.21.15.1:0/4163652812 conn(0x7f39840617a0 msgr2=0x7f3984061bb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:32:38.822 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:32:38.819+0000 7f398b5d6640 1 -- 172.21.15.1:0/4163652812 shutdown_connections 2024-03-25T18:32:38.822 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:32:38.819+0000 7f398b5d6640 1 -- 172.21.15.1:0/4163652812 wait complete. 2024-03-25T18:32:38.822 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:32:38.820+0000 7f398b5d6640 10 monclient: build_initial_monmap 2024-03-25T18:32:38.822 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:32:38.820+0000 7f398b5d6640 10 monclient: monmap: 2024-03-25T18:32:38.822 INFO:teuthology.misc.health.smithi001.stderr:epoch 0 2024-03-25T18:32:38.823 INFO:teuthology.misc.health.smithi001.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T18:32:38.823 INFO:teuthology.misc.health.smithi001.stderr:last_changed 0.000000 2024-03-25T18:32:38.823 INFO:teuthology.misc.health.smithi001.stderr:created 0.000000 2024-03-25T18:32:38.823 INFO:teuthology.misc.health.smithi001.stderr:min_mon_release 0 (unknown) 2024-03-25T18:32:38.823 INFO:teuthology.misc.health.smithi001.stderr:election_strategy: 1 2024-03-25T18:32:38.823 INFO:teuthology.misc.health.smithi001.stderr:0: v1:172.21.15.1:6789/0 mon.noname-a 2024-03-25T18:32:38.823 INFO:teuthology.misc.health.smithi001.stderr:1: v1:172.21.15.115:6789/0 mon.noname-b 2024-03-25T18:32:38.823 INFO:teuthology.misc.health.smithi001.stderr:2: v1:172.21.15.1:6790/0 mon.noname-c 2024-03-25T18:32:38.823 INFO:teuthology.misc.health.smithi001.stderr: 2024-03-25T18:32:38.824 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:32:38.821+0000 7f398b5d6640 1 librados: starting msgr at 2024-03-25T18:32:38.824 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:32:38.821+0000 7f398b5d6640 1 librados: starting objecter 2024-03-25T18:32:38.824 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:32:38.821+0000 7f398b5d6640 1 Processor -- start 2024-03-25T18:32:38.825 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:32:38.821+0000 7f398b5d6640 1 -- start start 2024-03-25T18:32:38.825 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:32:38.821+0000 7f398b5d6640 1 librados: setting wanted keys 2024-03-25T18:32:38.825 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:32:38.821+0000 7f398b5d6640 1 librados: calling monclient init 2024-03-25T18:32:38.825 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:32:38.821+0000 7f398b5d6640 10 monclient: init 2024-03-25T18:32:38.825 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:32:38.822+0000 7f398b5d6640 10 monclient: _reopen_session rank -1 2024-03-25T18:32:38.825 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:32:38.822+0000 7f398b5d6640 10 monclient: _add_conns ranks=[2,1,0] 2024-03-25T18:32:38.825 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:32:38.822+0000 7f398b5d6640 10 monclient(hunting): picked mon.noname-c con 0x7f39840b4670 addr v1:172.21.15.1:6790/0 2024-03-25T18:32:38.825 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:32:38.822+0000 7f398b5d6640 10 monclient(hunting): picked mon.noname-b con 0x7f39840b4ff0 addr v1:172.21.15.115:6789/0 2024-03-25T18:32:38.825 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:32:38.822+0000 7f398b5d6640 10 monclient(hunting): picked mon.noname-a con 0x7f398413c480 addr v1:172.21.15.1:6789/0 2024-03-25T18:32:38.825 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:32:38.822+0000 7f398b5d6640 1 -- --> v1:172.21.15.1:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f39840b00e0 con 0x7f398413c480 2024-03-25T18:32:38.825 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:32:38.822+0000 7f398b5d6640 1 -- --> v1:172.21.15.115:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f39840b03c0 con 0x7f39840b4ff0 2024-03-25T18:32:38.825 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:32:38.822+0000 7f398b5d6640 1 -- --> v1:172.21.15.1:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f39840b06a0 con 0x7f39840b4670 2024-03-25T18:32:38.825 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:32:38.822+0000 7f398b5d6640 10 monclient(hunting): _renew_subs 2024-03-25T18:32:38.825 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:32:38.822+0000 7f398b5d6640 10 monclient(hunting): authenticate will time out at 7839.293359s 2024-03-25T18:32:38.839 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:32:38.838+0000 7f3989372640 1 --1- >> v1:172.21.15.1:6790/0 conn(0x7f39840b4670 0x7f39840ae240 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.1:6790/0 says I am v1:172.21.15.1:34872/0 (socket says 172.21.15.1:34872) 2024-03-25T18:32:38.840 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:32:38.838+0000 7f3989372640 1 -- 172.21.15.1:0/2690153085 learned_addr learned my addr 172.21.15.1:0/2690153085 (peer_addr_for_me v1:172.21.15.1:0/0) 2024-03-25T18:32:38.843 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:32:38.841+0000 7f3979ffb640 10 client.?.objecter ms_handle_connect 0x7f39840b4670 2024-03-25T18:32:38.843 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:32:38.841+0000 7f3979ffb640 10 client.?.objecter resend_mon_ops 2024-03-25T18:32:38.846 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:32:38.844+0000 7f3979ffb640 10 client.?.objecter ms_handle_connect 0x7f39840b4ff0 2024-03-25T18:32:38.846 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:32:38.844+0000 7f3979ffb640 10 client.?.objecter resend_mon_ops 2024-03-25T18:32:38.854 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:32:38.852+0000 7f3979ffb640 1 -- 172.21.15.1:0/2690153085 <== mon.1 v1:172.21.15.115:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2837694841 0 0) 0x7f39840b03c0 con 0x7f39840b4ff0 2024-03-25T18:32:38.854 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:32:38.852+0000 7f3979ffb640 10 monclient(hunting): _negotiate 2024-03-25T18:32:38.854 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:32:38.852+0000 7f3979ffb640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:32:38.854 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:32:38.852+0000 7f3979ffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:32:38.854 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:32:38.852+0000 7f3979ffb640 10 monclient(hunting): my global_id is 4489 2024-03-25T18:32:38.854 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:32:38.852+0000 7f3979ffb640 1 -- 172.21.15.1:0/2690153085 --> v1:172.21.15.115:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f3968004eb0 con 0x7f39840b4ff0 2024-03-25T18:32:38.860 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:32:38.858+0000 7f3979ffb640 10 client.?.objecter ms_handle_connect 0x7f398413c480 2024-03-25T18:32:38.860 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:32:38.858+0000 7f3979ffb640 10 client.?.objecter resend_mon_ops 2024-03-25T18:32:38.864 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:32:38.862+0000 7f3979ffb640 1 -- 172.21.15.1:0/2690153085 <== mon.2 v1:172.21.15.1:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 986616341 0 0) 0x7f39840b06a0 con 0x7f39840b4670 2024-03-25T18:32:38.864 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:32:38.862+0000 7f3979ffb640 10 monclient(hunting): _negotiate 2024-03-25T18:32:38.864 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:32:38.862+0000 7f3979ffb640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:32:38.864 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:32:38.862+0000 7f3979ffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:32:38.864 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:32:38.862+0000 7f3979ffb640 10 monclient(hunting): my global_id is 4490 2024-03-25T18:32:38.864 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:32:38.862+0000 7f3979ffb640 1 -- 172.21.15.1:0/2690153085 --> v1:172.21.15.1:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f39840b03c0 con 0x7f39840b4670 2024-03-25T18:32:38.872 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:32:38.870+0000 7f3979ffb640 1 -- 172.21.15.1:0/2690153085 <== mon.1 v1:172.21.15.115:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 4013498842 0 0) 0x7f3968004eb0 con 0x7f39840b4ff0 2024-03-25T18:32:38.872 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:32:38.870+0000 7f3979ffb640 1 -- 172.21.15.1:0/2690153085 --> v1:172.21.15.115:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f39840b06a0 con 0x7f39840b4ff0 2024-03-25T18:32:38.872 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:32:38.870+0000 7f3979ffb640 1 -- 172.21.15.1:0/2690153085 <== mon.1 v1:172.21.15.115:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1586965025 0 0) 0x7f3974001970 con 0x7f39840b4ff0 2024-03-25T18:32:38.872 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:32:38.870+0000 7f3979ffb640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:32:38.872 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:32:38.870+0000 7f3979ffb640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-03-25T18:32:38.872 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:32:38.870+0000 7f3979ffb640 10 monclient(hunting): dump: 2024-03-25T18:32:38.872 INFO:teuthology.misc.health.smithi001.stderr:epoch 1 2024-03-25T18:32:38.872 INFO:teuthology.misc.health.smithi001.stderr:fsid c8ea72cf-9e99-4383-9b52-6c23d365fdfc 2024-03-25T18:32:38.872 INFO:teuthology.misc.health.smithi001.stderr:last_changed 2024-03-25T18:25:50.510519+0000 2024-03-25T18:32:38.872 INFO:teuthology.misc.health.smithi001.stderr:created 2024-03-25T18:25:50.510519+0000 2024-03-25T18:32:38.872 INFO:teuthology.misc.health.smithi001.stderr:min_mon_release 19 (squid) 2024-03-25T18:32:38.872 INFO:teuthology.misc.health.smithi001.stderr:election_strategy: 3 2024-03-25T18:32:38.873 INFO:teuthology.misc.health.smithi001.stderr:0: v1:172.21.15.1:6789/0 mon.a 2024-03-25T18:32:38.873 INFO:teuthology.misc.health.smithi001.stderr:1: v1:172.21.15.115:6789/0 mon.b 2024-03-25T18:32:38.873 INFO:teuthology.misc.health.smithi001.stderr:2: v1:172.21.15.1:6790/0 mon.c 2024-03-25T18:32:38.873 INFO:teuthology.misc.health.smithi001.stderr: 2024-03-25T18:32:38.873 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:32:38.870+0000 7f3979ffb640 10 monclient(hunting): _finish_auth 0 2024-03-25T18:32:38.884 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:32:38.882+0000 7f3979ffb640 1 -- 172.21.15.1:0/2690153085 <== mon.0 v1:172.21.15.1:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2947276319 0 0) 0x7f39840b00e0 con 0x7f398413c480 2024-03-25T18:32:38.884 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:32:38.882+0000 7f3979ffb640 10 monclient(hunting): _negotiate 2024-03-25T18:32:38.884 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:32:38.882+0000 7f3979ffb640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:32:38.885 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:32:38.882+0000 7f3979ffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:32:38.885 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:32:38.882+0000 7f3979ffb640 10 monclient(hunting): my global_id is 4476 2024-03-25T18:32:38.885 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:32:38.882+0000 7f3979ffb640 1 -- 172.21.15.1:0/2690153085 --> v1:172.21.15.1:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f3968004eb0 con 0x7f398413c480 2024-03-25T18:32:38.888 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:32:38.886+0000 7f3979ffb640 1 -- 172.21.15.1:0/2690153085 <== mon.2 v1:172.21.15.1:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3323825174 0 0) 0x7f39840b03c0 con 0x7f39840b4670 2024-03-25T18:32:38.888 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:32:38.886+0000 7f3979ffb640 1 -- 172.21.15.1:0/2690153085 --> v1:172.21.15.1:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f39840b00e0 con 0x7f39840b4670 2024-03-25T18:32:38.888 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:32:38.886+0000 7f3979ffb640 1 -- 172.21.15.1:0/2690153085 <== mon.2 v1:172.21.15.1:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1586965025 0 0) 0x7f3980001b40 con 0x7f39840b4670 2024-03-25T18:32:38.888 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:32:38.886+0000 7f3979ffb640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:32:38.888 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:32:38.886+0000 7f3979ffb640 10 monclient(hunting): got monmap 1 from mon.c (according to old e1) 2024-03-25T18:32:38.889 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:32:38.886+0000 7f3979ffb640 10 monclient(hunting): dump: 2024-03-25T18:32:38.889 INFO:teuthology.misc.health.smithi001.stderr:epoch 1 2024-03-25T18:32:38.889 INFO:teuthology.misc.health.smithi001.stderr:fsid c8ea72cf-9e99-4383-9b52-6c23d365fdfc 2024-03-25T18:32:38.889 INFO:teuthology.misc.health.smithi001.stderr:last_changed 2024-03-25T18:25:50.510519+0000 2024-03-25T18:32:38.889 INFO:teuthology.misc.health.smithi001.stderr:created 2024-03-25T18:25:50.510519+0000 2024-03-25T18:32:38.889 INFO:teuthology.misc.health.smithi001.stderr:min_mon_release 19 (squid) 2024-03-25T18:32:38.889 INFO:teuthology.misc.health.smithi001.stderr:election_strategy: 3 2024-03-25T18:32:38.889 INFO:teuthology.misc.health.smithi001.stderr:0: v1:172.21.15.1:6789/0 mon.a 2024-03-25T18:32:38.889 INFO:teuthology.misc.health.smithi001.stderr:1: v1:172.21.15.115:6789/0 mon.b 2024-03-25T18:32:38.889 INFO:teuthology.misc.health.smithi001.stderr:2: v1:172.21.15.1:6790/0 mon.c 2024-03-25T18:32:38.889 INFO:teuthology.misc.health.smithi001.stderr: 2024-03-25T18:32:38.894 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:32:38.892+0000 7f3979ffb640 1 -- 172.21.15.1:0/2690153085 <== mon.1 v1:172.21.15.115:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 4205402175 0 0) 0x7f39840b06a0 con 0x7f39840b4ff0 2024-03-25T18:32:38.895 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:32:38.892+0000 7f3979ffb640 1 -- 172.21.15.1:0/2690153085 >> v1:172.21.15.1:6790/0 conn(0x7f39840b4670 legacy=0x7f39840ae240 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:32:38.895 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:32:38.892+0000 7f3979ffb640 1 -- 172.21.15.1:0/2690153085 >> v1:172.21.15.1:6789/0 conn(0x7f398413c480 legacy=0x7f398413a820 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:32:38.895 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:32:38.892+0000 7f3979ffb640 10 monclient: _finish_hunting 0 2024-03-25T18:32:38.895 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:32:38.892+0000 7f3979ffb640 1 monclient: found mon.b 2024-03-25T18:32:38.895 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:32:38.892+0000 7f3979ffb640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:32:38.895 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:32:38.892+0000 7f3979ffb640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.115:6789/0 2024-03-25T18:32:38.895 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:32:38.892+0000 7f3979ffb640 1 -- 172.21.15.1:0/2690153085 --> v1:172.21.15.115:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f39840af1d0 con 0x7f39840b4ff0 2024-03-25T18:32:38.895 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:32:38.892+0000 7f3979ffb640 10 monclient: _finish_auth 0 2024-03-25T18:32:38.895 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:32:38.892+0000 7f3979ffb640 10 monclient: _check_auth_tickets 2024-03-25T18:32:38.895 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:32:38.892+0000 7f3979ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:32:38.895 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:32:38.892+0000 7f398b5d6640 5 monclient: authenticate success, global_id 4489 2024-03-25T18:32:38.895 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:32:38.892+0000 7f398b5d6640 10 monclient: _renew_subs 2024-03-25T18:32:38.895 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:32:38.892+0000 7f398b5d6640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.115:6789/0 2024-03-25T18:32:38.895 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:32:38.892+0000 7f398b5d6640 1 -- 172.21.15.1:0/2690153085 --> v1:172.21.15.115:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f39840aeec0 con 0x7f39840b4ff0 2024-03-25T18:32:38.895 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:32:38.893+0000 7f398b5d6640 10 client.4489.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T18:32:38.896 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:32:38.893+0000 7f398b5d6640 10 monclient: _renew_subs 2024-03-25T18:32:38.896 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:32:38.893+0000 7f398b5d6640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.115:6789/0 2024-03-25T18:32:38.896 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:32:38.893+0000 7f398b5d6640 1 -- 172.21.15.1:0/2690153085 --> v1:172.21.15.115:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f39841e3570 con 0x7f39840b4ff0 2024-03-25T18:32:38.896 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:32:38.893+0000 7f398b5d6640 1 librados: init done 2024-03-25T18:32:38.896 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:32:38.894+0000 7f398b5d6640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T18:32:38.896 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:32:38.894+0000 7f398b5d6640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:32:38.896 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:32:38.894+0000 7f398b5d6640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.115:6789/0 2024-03-25T18:32:38.897 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:32:38.894+0000 7f398b5d6640 1 -- 172.21.15.1:0/2690153085 --> v1:172.21.15.115:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f398406dbd0 con 0x7f39840b4ff0 2024-03-25T18:32:39.188 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:32:39.186+0000 7f3979ffb640 1 -- 172.21.15.1:0/2690153085 <== mon.1 v1:172.21.15.115:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f39740035f0 con 0x7f39840b4ff0 2024-03-25T18:32:39.189 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:32:39.186+0000 7f3979ffb640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:32:39.189 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:32:39.186+0000 7f3979ffb640 1 -- 172.21.15.1:0/2690153085 <== mon.1 v1:172.21.15.115:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1586965025 0 0) 0x7f3974003900 con 0x7f39840b4ff0 2024-03-25T18:32:39.189 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:32:39.186+0000 7f3979ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:32:39.189 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:32:39.187+0000 7f3979ffb640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-25T18:32:39.189 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:32:39.187+0000 7f3979ffb640 10 monclient: dump: 2024-03-25T18:32:39.189 INFO:teuthology.misc.health.smithi001.stderr:epoch 1 2024-03-25T18:32:39.189 INFO:teuthology.misc.health.smithi001.stderr:fsid c8ea72cf-9e99-4383-9b52-6c23d365fdfc 2024-03-25T18:32:39.189 INFO:teuthology.misc.health.smithi001.stderr:last_changed 2024-03-25T18:25:50.510519+0000 2024-03-25T18:32:39.189 INFO:teuthology.misc.health.smithi001.stderr:created 2024-03-25T18:25:50.510519+0000 2024-03-25T18:32:39.189 INFO:teuthology.misc.health.smithi001.stderr:min_mon_release 19 (squid) 2024-03-25T18:32:39.189 INFO:teuthology.misc.health.smithi001.stderr:election_strategy: 3 2024-03-25T18:32:39.189 INFO:teuthology.misc.health.smithi001.stderr:0: v1:172.21.15.1:6789/0 mon.a 2024-03-25T18:32:39.189 INFO:teuthology.misc.health.smithi001.stderr:1: v1:172.21.15.115:6789/0 mon.b 2024-03-25T18:32:39.190 INFO:teuthology.misc.health.smithi001.stderr:2: v1:172.21.15.1:6790/0 mon.c 2024-03-25T18:32:39.190 INFO:teuthology.misc.health.smithi001.stderr: 2024-03-25T18:32:39.190 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:32:39.187+0000 7f397b7fe640 4 set_mon_vals no callback set 2024-03-25T18:32:39.190 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:32:39.187+0000 7f3979ffb640 1 -- 172.21.15.1:0/2690153085 <== mon.1 v1:172.21.15.115:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 21787975 0 0) 0x7f3974019950 con 0x7f39840b4ff0 2024-03-25T18:32:39.190 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:32:39.188+0000 7f3989372640 10 monclient: get_auth_request con 0x7f396806df70 auth_method 0 2024-03-25T18:32:39.190 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:32:39.188+0000 7f3979ffb640 10 client.4489.objecter ms_handle_connect 0x7f396806df70 2024-03-25T18:32:39.195 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:32:39.193+0000 7f3979ffb640 1 -- 172.21.15.1:0/2690153085 <== mon.1 v1:172.21.15.115:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 996254230 0 0) 0x7f3974053650 con 0x7f39840b4ff0 2024-03-25T18:32:39.195 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:32:39.193+0000 7f3979ffb640 10 client.4489.objecter ms_dispatch 0x7f3984000b90 osd_map(10..10 src has 1..10) v4 2024-03-25T18:32:39.195 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:32:39.193+0000 7f3979ffb640 3 client.4489.objecter handle_osd_map got epochs [10,10] > 0 2024-03-25T18:32:39.195 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:32:39.193+0000 7f3979ffb640 3 client.4489.objecter handle_osd_map decoding full epoch 10 2024-03-25T18:32:39.195 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:32:39.193+0000 7f3979ffb640 20 client.4489.objecter dump_active .. 0 homeless 2024-03-25T18:32:39.195 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:32:39.193+0000 7f3979ffb640 1 -- 172.21.15.1:0/2690153085 <== mon.1 v1:172.21.15.115:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f397404ef90 con 0x7f39840b4ff0 2024-03-25T18:32:39.195 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:32:39.193+0000 7f3979ffb640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:32:39.195 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:32:39.193+0000 7f3979ffb640 10 monclient: _finish_command 1 = system:0 2024-03-25T18:32:39.423 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:32:39.421+0000 7f398b5d6640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:32:39.423 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:32:39.421+0000 7f398b5d6640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:32:39.423 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:32:39.421+0000 7f398b5d6640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.115:6789/0 2024-03-25T18:32:39.423 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:32:39.421+0000 7f398b5d6640 1 -- 172.21.15.1:0/2690153085 --> v1:172.21.15.115:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f39840688c0 con 0x7f39840b4ff0 2024-03-25T18:32:39.458 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:32:39.456+0000 7f3979ffb640 1 -- 172.21.15.1:0/2690153085 <== mon.1 v1:172.21.15.115:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 4092213348) 0x7f3974051090 con 0x7f39840b4ff0 2024-03-25T18:32:39.458 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:32:39.456+0000 7f3979ffb640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:32:39.458 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:32:39.456+0000 7f3979ffb640 10 monclient: _finish_command 2 = system:0 2024-03-25T18:32:39.459 INFO:teuthology.misc.health.smithi001.stdout: 2024-03-25T18:32:39.459 INFO:teuthology.misc.health.smithi001.stdout:{"epoch":10,"fsid":"c8ea72cf-9e99-4383-9b52-6c23d365fdfc","created":"2024-03-25T18:26:38.856638+0000","modified":"2024-03-25T18:26:44.224208+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T18:26:44.224208+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"b9f34922-3a84-49be-8426-807c1eceaefc","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"42bf306f-55e3-4634-9874-8b4b73ac6644","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"292cb23a-18a9-4207-a7cc-4d5727feaec0","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"45e0e9ab-898d-4673-9590-b4117bc5e84a","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"831a5a09-73ab-4a61-9872-088e76bdbc66","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"88cd903d-3b55-4840-9396-6e3cfe005282","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"89db7355-bc9d-46f4-8ab0-e8d9e744e026","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"cac47bee-0914-4f08-896f-348838882866","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T18:32:39.461 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:32:39.459+0000 7f398b5d6640 10 librados: watch_flush enter 2024-03-25T18:32:39.461 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:32:39.459+0000 7f398b5d6640 10 librados: watch_flush exit 2024-03-25T18:32:39.462 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:32:39.459+0000 7f398b5d6640 20 client.4489.objecter shutdown clearing up homeless session... 2024-03-25T18:32:39.462 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:32:39.459+0000 7f398b5d6640 10 client.4489.objecter successfully canceled tick 2024-03-25T18:32:39.462 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:32:39.459+0000 7f398b5d6640 1 -- 172.21.15.1:0/2690153085 >> v1:172.21.15.115:6800/4237379176 conn(0x7f396806df70 legacy=0x7f3968070430 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:32:39.462 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:32:39.459+0000 7f398b5d6640 10 monclient: shutdown 2024-03-25T18:32:39.462 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:32:39.459+0000 7f398b5d6640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:32:39.462 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:32:39.459+0000 7f398b5d6640 1 -- 172.21.15.1:0/2690153085 >> v1:172.21.15.115:6789/0 conn(0x7f39840b4ff0 legacy=0x7f398413a110 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:32:39.462 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:32:39.459+0000 7f398b5d6640 1 -- 172.21.15.1:0/2690153085 shutdown_connections 2024-03-25T18:32:39.462 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:32:39.459+0000 7f398b5d6640 1 -- 172.21.15.1:0/2690153085 >> 172.21.15.1:0/2690153085 conn(0x7f39840617a0 msgr2=0x7f39840b7140 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:32:39.462 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:32:39.459+0000 7f398b5d6640 1 -- 172.21.15.1:0/2690153085 shutdown_connections 2024-03-25T18:32:39.462 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:32:39.459+0000 7f398b5d6640 1 -- 172.21.15.1:0/2690153085 wait complete. 2024-03-25T18:32:39.462 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:32:39.459+0000 7f398b5d6640 1 librados: shutdown 2024-03-25T18:32:39.473 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T18:32:42.060 INFO:tasks.ceph.mon.a.smithi001.stderr:==00:00:06:20.618 34876== Warning: unimplemented fcntl command: 1036 2024-03-25T18:32:42.067 INFO:tasks.ceph.mon.a.smithi001.stderr:==00:00:06:20.625 34876== Warning: unimplemented fcntl command: 1036 2024-03-25T18:32:42.094 INFO:tasks.ceph.mon.a.smithi001.stderr:==00:00:06:20.652 34876== Warning: unimplemented fcntl command: 1036 2024-03-25T18:32:42.114 INFO:tasks.ceph.mon.b.smithi115.stderr:==00:00:06:20.598 34836== Warning: unimplemented fcntl command: 1036 2024-03-25T18:32:42.122 INFO:tasks.ceph.mon.b.smithi115.stderr:==00:00:06:20.605 34836== Warning: unimplemented fcntl command: 1036 2024-03-25T18:32:42.124 INFO:tasks.ceph.mon.c.smithi001.stderr:==00:00:06:20.683 34877== Warning: unimplemented fcntl command: 1036 2024-03-25T18:32:42.135 INFO:tasks.ceph.mon.c.smithi001.stderr:==00:00:06:20.693 34877== Warning: unimplemented fcntl command: 1036 2024-03-25T18:32:42.143 INFO:tasks.ceph.mon.b.smithi115.stderr:==00:00:06:20.627 34836== Warning: unimplemented fcntl command: 1036 2024-03-25T18:32:42.171 INFO:tasks.ceph.mon.c.smithi001.stderr:==00:00:06:20.729 34877== Warning: unimplemented fcntl command: 1036 2024-03-25T18:32:46.281 DEBUG:teuthology.orchestra.run.smithi001:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T18:32:46.399 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:32:46.397+0000 7fb0f1a06640 10 monclient: get_monmap_and_config 2024-03-25T18:32:46.399 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:32:46.397+0000 7fb0f1a06640 10 monclient: build_initial_monmap 2024-03-25T18:32:46.399 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:32:46.397+0000 7fb0f1a06640 10 monclient: monmap: 2024-03-25T18:32:46.400 INFO:teuthology.misc.health.smithi001.stderr:epoch 0 2024-03-25T18:32:46.400 INFO:teuthology.misc.health.smithi001.stderr:fsid c8ea72cf-9e99-4383-9b52-6c23d365fdfc 2024-03-25T18:32:46.400 INFO:teuthology.misc.health.smithi001.stderr:last_changed 2024-03-25T18:32:46.399339+0000 2024-03-25T18:32:46.400 INFO:teuthology.misc.health.smithi001.stderr:created 2024-03-25T18:32:46.399339+0000 2024-03-25T18:32:46.400 INFO:teuthology.misc.health.smithi001.stderr:min_mon_release 0 (unknown) 2024-03-25T18:32:46.400 INFO:teuthology.misc.health.smithi001.stderr:election_strategy: 3 2024-03-25T18:32:46.400 INFO:teuthology.misc.health.smithi001.stderr:0: v1:172.21.15.1:6789/0 mon.noname-a 2024-03-25T18:32:46.400 INFO:teuthology.misc.health.smithi001.stderr:1: v1:172.21.15.115:6789/0 mon.noname-c 2024-03-25T18:32:46.400 INFO:teuthology.misc.health.smithi001.stderr:2: v1:172.21.15.1:6790/0 mon.noname-b 2024-03-25T18:32:46.400 INFO:teuthology.misc.health.smithi001.stderr: 2024-03-25T18:32:46.401 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:32:46.398+0000 7fb0f1a06640 1 Processor -- start 2024-03-25T18:32:46.401 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:32:46.398+0000 7fb0f1a06640 1 -- start start 2024-03-25T18:32:46.401 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:32:46.398+0000 7fb0f1a06640 10 monclient: init 2024-03-25T18:32:46.402 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:32:46.399+0000 7fb0f1a06640 10 monclient: _reopen_session rank -1 2024-03-25T18:32:46.402 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:32:46.399+0000 7fb0f1a06640 10 monclient: _add_conns ranks=[1,0,2] 2024-03-25T18:32:46.402 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:32:46.399+0000 7fb0f1a06640 10 monclient(hunting): picked mon.noname-c con 0x7fb0ec0b4ff0 addr v1:172.21.15.115:6789/0 2024-03-25T18:32:46.402 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:32:46.399+0000 7fb0f1a06640 10 monclient(hunting): picked mon.noname-a con 0x7fb0ec0b4670 addr v1:172.21.15.1:6789/0 2024-03-25T18:32:46.402 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:32:46.399+0000 7fb0f1a06640 10 monclient(hunting): picked mon.noname-b con 0x7fb0ec13c480 addr v1:172.21.15.1:6790/0 2024-03-25T18:32:46.402 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:32:46.399+0000 7fb0f1a06640 1 -- --> v1:172.21.15.1:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb0ec0ab0e0 con 0x7fb0ec0b4670 2024-03-25T18:32:46.402 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:32:46.399+0000 7fb0f1a06640 1 -- --> v1:172.21.15.115:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb0ec0ab3c0 con 0x7fb0ec0b4ff0 2024-03-25T18:32:46.402 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:32:46.399+0000 7fb0f1a06640 1 -- --> v1:172.21.15.1:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb0ec0a9f30 con 0x7fb0ec13c480 2024-03-25T18:32:46.402 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:32:46.399+0000 7fb0f1a06640 10 monclient(hunting): _renew_subs 2024-03-25T18:32:46.402 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:32:46.399+0000 7fb0f1a06640 10 monclient(hunting): authenticate will time out at 7846.870044s 2024-03-25T18:32:46.407 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:32:46.405+0000 7fb0eb7fe640 1 --1- >> v1:172.21.15.1:6790/0 conn(0x7fb0ec13c480 0x7fb0ec14dc00 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.1:6790/0 says I am v1:172.21.15.1:54272/0 (socket says 172.21.15.1:54272) 2024-03-25T18:32:46.407 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:32:46.405+0000 7fb0eb7fe640 1 -- 172.21.15.1:0/3930605694 learned_addr learned my addr 172.21.15.1:0/3930605694 (peer_addr_for_me v1:172.21.15.1:0/0) 2024-03-25T18:32:46.430 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:32:46.428+0000 7fb0e9ffb640 1 -- 172.21.15.1:0/3930605694 <== mon.2 v1:172.21.15.1:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3159371224 0 0) 0x7fb0ec0a9f30 con 0x7fb0ec13c480 2024-03-25T18:32:46.430 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:32:46.428+0000 7fb0e9ffb640 10 monclient(hunting): _negotiate 2024-03-25T18:32:46.430 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:32:46.428+0000 7fb0e9ffb640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:32:46.430 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:32:46.428+0000 7fb0e9ffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:32:46.430 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:32:46.428+0000 7fb0e9ffb640 10 monclient(hunting): my global_id is 4493 2024-03-25T18:32:46.430 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:32:46.428+0000 7fb0e9ffb640 1 -- 172.21.15.1:0/3930605694 --> v1:172.21.15.1:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fb0d4003520 con 0x7fb0ec13c480 2024-03-25T18:32:46.438 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:32:46.436+0000 7fb0e9ffb640 1 -- 172.21.15.1:0/3930605694 <== mon.0 v1:172.21.15.1:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3063290248 0 0) 0x7fb0ec0ab0e0 con 0x7fb0ec0b4670 2024-03-25T18:32:46.438 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:32:46.436+0000 7fb0e9ffb640 10 monclient(hunting): _negotiate 2024-03-25T18:32:46.438 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:32:46.436+0000 7fb0e9ffb640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:32:46.438 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:32:46.436+0000 7fb0e9ffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:32:46.438 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:32:46.436+0000 7fb0e9ffb640 10 monclient(hunting): my global_id is 4479 2024-03-25T18:32:46.438 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:32:46.436+0000 7fb0e9ffb640 1 -- 172.21.15.1:0/3930605694 --> v1:172.21.15.1:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fb0ec0a9f30 con 0x7fb0ec0b4670 2024-03-25T18:32:46.446 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:32:46.445+0000 7fb0e9ffb640 1 -- 172.21.15.1:0/3930605694 <== mon.1 v1:172.21.15.115:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3788873716 0 0) 0x7fb0ec0ab3c0 con 0x7fb0ec0b4ff0 2024-03-25T18:32:46.446 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:32:46.445+0000 7fb0e9ffb640 10 monclient(hunting): _negotiate 2024-03-25T18:32:46.447 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:32:46.445+0000 7fb0e9ffb640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:32:46.447 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:32:46.445+0000 7fb0e9ffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:32:46.447 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:32:46.445+0000 7fb0e9ffb640 10 monclient(hunting): my global_id is 4492 2024-03-25T18:32:46.447 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:32:46.445+0000 7fb0e9ffb640 1 -- 172.21.15.1:0/3930605694 --> v1:172.21.15.115:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fb0ec0ab0e0 con 0x7fb0ec0b4ff0 2024-03-25T18:32:46.455 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:32:46.452+0000 7fb0e9ffb640 1 -- 172.21.15.1:0/3930605694 <== mon.2 v1:172.21.15.1:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2696183112 0 0) 0x7fb0d4003520 con 0x7fb0ec13c480 2024-03-25T18:32:46.455 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:32:46.452+0000 7fb0e9ffb640 1 -- 172.21.15.1:0/3930605694 >> v1:172.21.15.115:6789/0 conn(0x7fb0ec0b4ff0 legacy=0x7fb0ec13bd70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:32:46.455 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:32:46.452+0000 7fb0e9ffb640 1 -- 172.21.15.1:0/3930605694 >> v1:172.21.15.1:6789/0 conn(0x7fb0ec0b4670 legacy=0x7fb0ec1425c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:32:46.455 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:32:46.452+0000 7fb0e9ffb640 10 monclient: _finish_hunting 0 2024-03-25T18:32:46.455 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:32:46.452+0000 7fb0e9ffb640 1 monclient: found mon.noname-b 2024-03-25T18:32:46.455 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:32:46.452+0000 7fb0e9ffb640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:32:46.455 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:32:46.452+0000 7fb0e9ffb640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.1:6790/0 2024-03-25T18:32:46.455 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:32:46.452+0000 7fb0e9ffb640 1 -- 172.21.15.1:0/3930605694 --> v1:172.21.15.1:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb0ec0a9b70 con 0x7fb0ec13c480 2024-03-25T18:32:46.455 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:32:46.452+0000 7fb0e9ffb640 10 monclient: _finish_auth 0 2024-03-25T18:32:46.455 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:32:46.452+0000 7fb0e9ffb640 10 monclient: _check_auth_tickets 2024-03-25T18:32:46.455 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:32:46.452+0000 7fb0e9ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:32:46.455 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:32:46.452+0000 7fb0e9ffb640 1 -- 172.21.15.1:0/3930605694 <== mon.2 v1:172.21.15.1:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1586965025 0 0) 0x7fb0e4001a10 con 0x7fb0ec13c480 2024-03-25T18:32:46.455 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:32:46.452+0000 7fb0e9ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:32:46.456 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:32:46.452+0000 7fb0e9ffb640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-03-25T18:32:46.456 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:32:46.452+0000 7fb0e9ffb640 10 monclient: dump: 2024-03-25T18:32:46.456 INFO:teuthology.misc.health.smithi001.stderr:epoch 1 2024-03-25T18:32:46.456 INFO:teuthology.misc.health.smithi001.stderr:fsid c8ea72cf-9e99-4383-9b52-6c23d365fdfc 2024-03-25T18:32:46.456 INFO:teuthology.misc.health.smithi001.stderr:last_changed 2024-03-25T18:25:50.510519+0000 2024-03-25T18:32:46.456 INFO:teuthology.misc.health.smithi001.stderr:created 2024-03-25T18:25:50.510519+0000 2024-03-25T18:32:46.456 INFO:teuthology.misc.health.smithi001.stderr:min_mon_release 19 (squid) 2024-03-25T18:32:46.456 INFO:teuthology.misc.health.smithi001.stderr:election_strategy: 3 2024-03-25T18:32:46.456 INFO:teuthology.misc.health.smithi001.stderr:0: v1:172.21.15.1:6789/0 mon.a 2024-03-25T18:32:46.456 INFO:teuthology.misc.health.smithi001.stderr:1: v1:172.21.15.115:6789/0 mon.b 2024-03-25T18:32:46.456 INFO:teuthology.misc.health.smithi001.stderr:2: v1:172.21.15.1:6790/0 mon.c 2024-03-25T18:32:46.456 INFO:teuthology.misc.health.smithi001.stderr: 2024-03-25T18:32:46.456 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:32:46.452+0000 7fb0f1a06640 5 monclient: authenticate success, global_id 4493 2024-03-25T18:32:46.456 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:32:46.452+0000 7fb0f1a06640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T18:32:46.498 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:32:46.496+0000 7fb0e9ffb640 1 -- 172.21.15.1:0/3930605694 <== mon.2 v1:172.21.15.1:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fb0e4001b70 con 0x7fb0ec13c480 2024-03-25T18:32:46.498 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:32:46.496+0000 7fb0e9ffb640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:32:46.498 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:32:46.496+0000 7fb0e9ffb640 1 -- 172.21.15.1:0/3930605694 <== mon.2 v1:172.21.15.1:6790/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1586965025 0 0) 0x7fb0e4001f80 con 0x7fb0ec13c480 2024-03-25T18:32:46.498 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:32:46.496+0000 7fb0e9ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:32:46.498 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:32:46.496+0000 7fb0e9ffb640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-25T18:32:46.499 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:32:46.496+0000 7fb0e9ffb640 10 monclient: dump: 2024-03-25T18:32:46.499 INFO:teuthology.misc.health.smithi001.stderr:epoch 1 2024-03-25T18:32:46.499 INFO:teuthology.misc.health.smithi001.stderr:fsid c8ea72cf-9e99-4383-9b52-6c23d365fdfc 2024-03-25T18:32:46.499 INFO:teuthology.misc.health.smithi001.stderr:last_changed 2024-03-25T18:25:50.510519+0000 2024-03-25T18:32:46.499 INFO:teuthology.misc.health.smithi001.stderr:created 2024-03-25T18:25:50.510519+0000 2024-03-25T18:32:46.499 INFO:teuthology.misc.health.smithi001.stderr:min_mon_release 19 (squid) 2024-03-25T18:32:46.499 INFO:teuthology.misc.health.smithi001.stderr:election_strategy: 3 2024-03-25T18:32:46.499 INFO:teuthology.misc.health.smithi001.stderr:0: v1:172.21.15.1:6789/0 mon.a 2024-03-25T18:32:46.499 INFO:teuthology.misc.health.smithi001.stderr:1: v1:172.21.15.115:6789/0 mon.b 2024-03-25T18:32:46.499 INFO:teuthology.misc.health.smithi001.stderr:2: v1:172.21.15.1:6790/0 mon.c 2024-03-25T18:32:46.499 INFO:teuthology.misc.health.smithi001.stderr: 2024-03-25T18:32:46.499 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:32:46.496+0000 7fb0f1a06640 10 monclient: get_monmap_and_config success 2024-03-25T18:32:46.499 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:32:46.496+0000 7fb0f1a06640 4 set_mon_vals no callback set 2024-03-25T18:32:46.501 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:32:46.497+0000 7fb0f1a06640 10 monclient: shutdown 2024-03-25T18:32:46.501 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:32:46.497+0000 7fb0f1a06640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:32:46.501 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:32:46.497+0000 7fb0f1a06640 1 -- 172.21.15.1:0/3930605694 >> v1:172.21.15.1:6790/0 conn(0x7fb0ec13c480 legacy=0x7fb0ec14dc00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:32:46.501 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:32:46.497+0000 7fb0f1a06640 1 -- 172.21.15.1:0/3930605694 shutdown_connections 2024-03-25T18:32:46.501 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:32:46.497+0000 7fb0f1a06640 1 -- 172.21.15.1:0/3930605694 >> 172.21.15.1:0/3930605694 conn(0x7fb0ec0617a0 msgr2=0x7fb0ec061bb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:32:46.501 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:32:46.497+0000 7fb0f1a06640 1 -- 172.21.15.1:0/3930605694 shutdown_connections 2024-03-25T18:32:46.501 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:32:46.497+0000 7fb0f1a06640 1 -- 172.21.15.1:0/3930605694 wait complete. 2024-03-25T18:32:46.501 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:32:46.498+0000 7fb0f1a06640 10 monclient: build_initial_monmap 2024-03-25T18:32:46.501 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:32:46.498+0000 7fb0f1a06640 10 monclient: monmap: 2024-03-25T18:32:46.501 INFO:teuthology.misc.health.smithi001.stderr:epoch 0 2024-03-25T18:32:46.501 INFO:teuthology.misc.health.smithi001.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T18:32:46.502 INFO:teuthology.misc.health.smithi001.stderr:last_changed 0.000000 2024-03-25T18:32:46.502 INFO:teuthology.misc.health.smithi001.stderr:created 0.000000 2024-03-25T18:32:46.502 INFO:teuthology.misc.health.smithi001.stderr:min_mon_release 0 (unknown) 2024-03-25T18:32:46.502 INFO:teuthology.misc.health.smithi001.stderr:election_strategy: 1 2024-03-25T18:32:46.502 INFO:teuthology.misc.health.smithi001.stderr:0: v1:172.21.15.1:6789/0 mon.noname-a 2024-03-25T18:32:46.502 INFO:teuthology.misc.health.smithi001.stderr:1: v1:172.21.15.115:6789/0 mon.noname-b 2024-03-25T18:32:46.502 INFO:teuthology.misc.health.smithi001.stderr:2: v1:172.21.15.1:6790/0 mon.noname-c 2024-03-25T18:32:46.502 INFO:teuthology.misc.health.smithi001.stderr: 2024-03-25T18:32:46.502 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:32:46.498+0000 7fb0f1a06640 1 librados: starting msgr at 2024-03-25T18:32:46.502 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:32:46.498+0000 7fb0f1a06640 1 librados: starting objecter 2024-03-25T18:32:46.502 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:32:46.499+0000 7fb0f1a06640 1 Processor -- start 2024-03-25T18:32:46.502 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:32:46.499+0000 7fb0f1a06640 1 -- start start 2024-03-25T18:32:46.502 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:32:46.499+0000 7fb0f1a06640 1 librados: setting wanted keys 2024-03-25T18:32:46.502 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:32:46.499+0000 7fb0f1a06640 1 librados: calling monclient init 2024-03-25T18:32:46.503 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:32:46.499+0000 7fb0f1a06640 10 monclient: init 2024-03-25T18:32:46.503 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:32:46.500+0000 7fb0f1a06640 10 monclient: _reopen_session rank -1 2024-03-25T18:32:46.503 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:32:46.500+0000 7fb0f1a06640 10 monclient: _add_conns ranks=[0,2,1] 2024-03-25T18:32:46.503 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:32:46.500+0000 7fb0f1a06640 10 monclient(hunting): picked mon.noname-a con 0x7fb0ec0b4670 addr v1:172.21.15.1:6789/0 2024-03-25T18:32:46.503 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:32:46.500+0000 7fb0f1a06640 10 monclient(hunting): picked mon.noname-c con 0x7fb0ec0b4ff0 addr v1:172.21.15.1:6790/0 2024-03-25T18:32:46.504 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:32:46.500+0000 7fb0f1a06640 10 monclient(hunting): picked mon.noname-b con 0x7fb0ec13c480 addr v1:172.21.15.115:6789/0 2024-03-25T18:32:46.504 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:32:46.500+0000 7fb0f1a06640 1 -- --> v1:172.21.15.1:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb0ec146780 con 0x7fb0ec0b4670 2024-03-25T18:32:46.504 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:32:46.500+0000 7fb0f1a06640 1 -- --> v1:172.21.15.115:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb0ec144c10 con 0x7fb0ec13c480 2024-03-25T18:32:46.504 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:32:46.500+0000 7fb0f1a06640 1 -- --> v1:172.21.15.1:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb0ec144ef0 con 0x7fb0ec0b4ff0 2024-03-25T18:32:46.504 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:32:46.500+0000 7fb0f1a06640 10 monclient(hunting): _renew_subs 2024-03-25T18:32:46.504 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:32:46.500+0000 7fb0f1a06640 10 monclient(hunting): authenticate will time out at 7846.970963s 2024-03-25T18:32:46.510 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:32:46.508+0000 7fb0eb7fe640 1 --1- >> v1:172.21.15.115:6789/0 conn(0x7fb0ec13c480 0x7fb0ec1e2590 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.115:6789/0 says I am v1:172.21.15.1:41830/0 (socket says 172.21.15.1:41830) 2024-03-25T18:32:46.510 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:32:46.508+0000 7fb0eb7fe640 1 -- 172.21.15.1:0/3226427439 learned_addr learned my addr 172.21.15.1:0/3226427439 (peer_addr_for_me v1:172.21.15.1:0/0) 2024-03-25T18:32:46.511 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:32:46.509+0000 7fb0d3fff640 10 client.?.objecter ms_handle_connect 0x7fb0ec13c480 2024-03-25T18:32:46.511 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:32:46.509+0000 7fb0d3fff640 10 client.?.objecter resend_mon_ops 2024-03-25T18:32:46.518 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:32:46.517+0000 7fb0d3fff640 10 client.?.objecter ms_handle_connect 0x7fb0ec0b4ff0 2024-03-25T18:32:46.519 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:32:46.517+0000 7fb0d3fff640 10 client.?.objecter resend_mon_ops 2024-03-25T18:32:46.521 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:32:46.520+0000 7fb0d3fff640 1 -- 172.21.15.1:0/3226427439 <== mon.1 v1:172.21.15.115:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2218623841 0 0) 0x7fb0ec144c10 con 0x7fb0ec13c480 2024-03-25T18:32:46.522 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:32:46.520+0000 7fb0d3fff640 10 monclient(hunting): _negotiate 2024-03-25T18:32:46.522 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:32:46.520+0000 7fb0d3fff640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:32:46.522 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:32:46.520+0000 7fb0d3fff640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:32:46.522 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:32:46.520+0000 7fb0d3fff640 10 monclient(hunting): my global_id is 4495 2024-03-25T18:32:46.522 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:32:46.520+0000 7fb0d3fff640 1 -- 172.21.15.1:0/3226427439 --> v1:172.21.15.115:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fb0d8004eb0 con 0x7fb0ec13c480 2024-03-25T18:32:46.523 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:32:46.521+0000 7fb0d3fff640 10 client.?.objecter ms_handle_connect 0x7fb0ec0b4670 2024-03-25T18:32:46.523 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:32:46.521+0000 7fb0d3fff640 10 client.?.objecter resend_mon_ops 2024-03-25T18:32:46.532 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:32:46.530+0000 7fb0d3fff640 1 -- 172.21.15.1:0/3226427439 <== mon.0 v1:172.21.15.1:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1262022093 0 0) 0x7fb0ec0ab0e0 con 0x7fb0ec0b4670 2024-03-25T18:32:46.532 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:32:46.530+0000 7fb0d3fff640 10 monclient(hunting): _negotiate 2024-03-25T18:32:46.532 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:32:46.530+0000 7fb0d3fff640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:32:46.532 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:32:46.530+0000 7fb0d3fff640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:32:46.532 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:32:46.530+0000 7fb0d3fff640 10 monclient(hunting): my global_id is 4482 2024-03-25T18:32:46.533 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:32:46.530+0000 7fb0d3fff640 1 -- 172.21.15.1:0/3226427439 --> v1:172.21.15.1:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fb0ec144c10 con 0x7fb0ec0b4670 2024-03-25T18:32:46.533 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:32:46.532+0000 7fb0d3fff640 1 -- 172.21.15.1:0/3226427439 <== mon.2 v1:172.21.15.1:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2870245092 0 0) 0x7fb0ec144ef0 con 0x7fb0ec0b4ff0 2024-03-25T18:32:46.534 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:32:46.532+0000 7fb0d3fff640 10 monclient(hunting): _negotiate 2024-03-25T18:32:46.534 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:32:46.532+0000 7fb0d3fff640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:32:46.534 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:32:46.532+0000 7fb0d3fff640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:32:46.534 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:32:46.532+0000 7fb0d3fff640 10 monclient(hunting): my global_id is 4496 2024-03-25T18:32:46.534 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:32:46.532+0000 7fb0d3fff640 1 -- 172.21.15.1:0/3226427439 --> v1:172.21.15.1:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fb0ec0ab0e0 con 0x7fb0ec0b4ff0 2024-03-25T18:32:46.549 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:32:46.547+0000 7fb0d3fff640 1 -- 172.21.15.1:0/3226427439 <== mon.1 v1:172.21.15.115:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1752036737 0 0) 0x7fb0d8004eb0 con 0x7fb0ec13c480 2024-03-25T18:32:46.549 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:32:46.547+0000 7fb0d3fff640 1 -- 172.21.15.1:0/3226427439 --> v1:172.21.15.115:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fb0ec144ef0 con 0x7fb0ec13c480 2024-03-25T18:32:46.549 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:32:46.547+0000 7fb0d3fff640 1 -- 172.21.15.1:0/3226427439 <== mon.1 v1:172.21.15.115:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1586965025 0 0) 0x7fb0e4001ec0 con 0x7fb0ec13c480 2024-03-25T18:32:46.549 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:32:46.547+0000 7fb0d3fff640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:32:46.549 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:32:46.547+0000 7fb0d3fff640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-03-25T18:32:46.549 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:32:46.547+0000 7fb0d3fff640 10 monclient(hunting): dump: 2024-03-25T18:32:46.549 INFO:teuthology.misc.health.smithi001.stderr:epoch 1 2024-03-25T18:32:46.549 INFO:teuthology.misc.health.smithi001.stderr:fsid c8ea72cf-9e99-4383-9b52-6c23d365fdfc 2024-03-25T18:32:46.549 INFO:teuthology.misc.health.smithi001.stderr:last_changed 2024-03-25T18:25:50.510519+0000 2024-03-25T18:32:46.549 INFO:teuthology.misc.health.smithi001.stderr:created 2024-03-25T18:25:50.510519+0000 2024-03-25T18:32:46.549 INFO:teuthology.misc.health.smithi001.stderr:min_mon_release 19 (squid) 2024-03-25T18:32:46.550 INFO:teuthology.misc.health.smithi001.stderr:election_strategy: 3 2024-03-25T18:32:46.550 INFO:teuthology.misc.health.smithi001.stderr:0: v1:172.21.15.1:6789/0 mon.a 2024-03-25T18:32:46.550 INFO:teuthology.misc.health.smithi001.stderr:1: v1:172.21.15.115:6789/0 mon.b 2024-03-25T18:32:46.550 INFO:teuthology.misc.health.smithi001.stderr:2: v1:172.21.15.1:6790/0 mon.c 2024-03-25T18:32:46.550 INFO:teuthology.misc.health.smithi001.stderr: 2024-03-25T18:32:46.550 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:32:46.547+0000 7fb0d3fff640 10 monclient(hunting): _finish_auth 0 2024-03-25T18:32:46.555 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:32:46.553+0000 7fb0d3fff640 1 -- 172.21.15.1:0/3226427439 <== mon.2 v1:172.21.15.1:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 712922810 0 0) 0x7fb0ec0ab0e0 con 0x7fb0ec0b4ff0 2024-03-25T18:32:46.556 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:32:46.554+0000 7fb0d3fff640 1 -- 172.21.15.1:0/3226427439 --> v1:172.21.15.1:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fb0d8004eb0 con 0x7fb0ec0b4ff0 2024-03-25T18:32:46.556 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:32:46.554+0000 7fb0d3fff640 1 -- 172.21.15.1:0/3226427439 <== mon.2 v1:172.21.15.1:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1586965025 0 0) 0x7fb0e00018f0 con 0x7fb0ec0b4ff0 2024-03-25T18:32:46.556 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:32:46.554+0000 7fb0d3fff640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:32:46.556 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:32:46.554+0000 7fb0d3fff640 10 monclient(hunting): got monmap 1 from mon.c (according to old e1) 2024-03-25T18:32:46.556 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:32:46.554+0000 7fb0d3fff640 10 monclient(hunting): dump: 2024-03-25T18:32:46.556 INFO:teuthology.misc.health.smithi001.stderr:epoch 1 2024-03-25T18:32:46.556 INFO:teuthology.misc.health.smithi001.stderr:fsid c8ea72cf-9e99-4383-9b52-6c23d365fdfc 2024-03-25T18:32:46.556 INFO:teuthology.misc.health.smithi001.stderr:last_changed 2024-03-25T18:25:50.510519+0000 2024-03-25T18:32:46.556 INFO:teuthology.misc.health.smithi001.stderr:created 2024-03-25T18:25:50.510519+0000 2024-03-25T18:32:46.556 INFO:teuthology.misc.health.smithi001.stderr:min_mon_release 19 (squid) 2024-03-25T18:32:46.556 INFO:teuthology.misc.health.smithi001.stderr:election_strategy: 3 2024-03-25T18:32:46.556 INFO:teuthology.misc.health.smithi001.stderr:0: v1:172.21.15.1:6789/0 mon.a 2024-03-25T18:32:46.556 INFO:teuthology.misc.health.smithi001.stderr:1: v1:172.21.15.115:6789/0 mon.b 2024-03-25T18:32:46.556 INFO:teuthology.misc.health.smithi001.stderr:2: v1:172.21.15.1:6790/0 mon.c 2024-03-25T18:32:46.557 INFO:teuthology.misc.health.smithi001.stderr: 2024-03-25T18:32:46.568 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:32:46.565+0000 7fb0d3fff640 1 -- 172.21.15.1:0/3226427439 <== mon.1 v1:172.21.15.115:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 1483880886 0 0) 0x7fb0ec144ef0 con 0x7fb0ec13c480 2024-03-25T18:32:46.568 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:32:46.565+0000 7fb0d3fff640 1 -- 172.21.15.1:0/3226427439 >> v1:172.21.15.1:6790/0 conn(0x7fb0ec0b4ff0 legacy=0x7fb0ec146070 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:32:46.568 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:32:46.565+0000 7fb0d3fff640 1 -- 172.21.15.1:0/3226427439 >> v1:172.21.15.1:6789/0 conn(0x7fb0ec0b4670 legacy=0x7fb0ec145960 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:32:46.568 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:32:46.565+0000 7fb0d3fff640 10 monclient: _finish_hunting 0 2024-03-25T18:32:46.568 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:32:46.565+0000 7fb0d3fff640 1 monclient: found mon.b 2024-03-25T18:32:46.568 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:32:46.566+0000 7fb0d3fff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:32:46.568 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:32:46.566+0000 7fb0d3fff640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.115:6789/0 2024-03-25T18:32:46.568 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:32:46.566+0000 7fb0d3fff640 1 -- 172.21.15.1:0/3226427439 --> v1:172.21.15.115:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb0ec13f0b0 con 0x7fb0ec13c480 2024-03-25T18:32:46.568 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:32:46.566+0000 7fb0d3fff640 10 monclient: _finish_auth 0 2024-03-25T18:32:46.569 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:32:46.566+0000 7fb0d3fff640 10 monclient: _check_auth_tickets 2024-03-25T18:32:46.569 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:32:46.566+0000 7fb0d3fff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:32:46.569 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:32:46.566+0000 7fb0f1a06640 5 monclient: authenticate success, global_id 4495 2024-03-25T18:32:46.569 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:32:46.566+0000 7fb0f1a06640 10 monclient: _renew_subs 2024-03-25T18:32:46.569 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:32:46.566+0000 7fb0f1a06640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.115:6789/0 2024-03-25T18:32:46.569 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:32:46.566+0000 7fb0f1a06640 1 -- 172.21.15.1:0/3226427439 --> v1:172.21.15.115:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb0ec143840 con 0x7fb0ec13c480 2024-03-25T18:32:46.569 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:32:46.566+0000 7fb0f1a06640 10 client.4495.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T18:32:46.569 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:32:46.566+0000 7fb0f1a06640 10 monclient: _renew_subs 2024-03-25T18:32:46.569 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:32:46.566+0000 7fb0f1a06640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.115:6789/0 2024-03-25T18:32:46.569 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:32:46.566+0000 7fb0f1a06640 1 -- 172.21.15.1:0/3226427439 --> v1:172.21.15.115:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7fb0ec13df00 con 0x7fb0ec13c480 2024-03-25T18:32:46.569 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:32:46.566+0000 7fb0f1a06640 1 librados: init done 2024-03-25T18:32:46.570 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:32:46.567+0000 7fb0f1a06640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T18:32:46.570 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:32:46.567+0000 7fb0f1a06640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:32:46.570 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:32:46.567+0000 7fb0f1a06640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.115:6789/0 2024-03-25T18:32:46.570 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:32:46.567+0000 7fb0f1a06640 1 -- 172.21.15.1:0/3226427439 --> v1:172.21.15.115:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb0ec06dbd0 con 0x7fb0ec13c480 2024-03-25T18:32:46.861 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:32:46.858+0000 7fb0d3fff640 1 -- 172.21.15.1:0/3226427439 <== mon.1 v1:172.21.15.115:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fb0e4004e10 con 0x7fb0ec13c480 2024-03-25T18:32:46.861 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:32:46.858+0000 7fb0d3fff640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:32:46.861 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:32:46.858+0000 7fb0d3fff640 1 -- 172.21.15.1:0/3226427439 <== mon.1 v1:172.21.15.115:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1586965025 0 0) 0x7fb0e4002100 con 0x7fb0ec13c480 2024-03-25T18:32:46.861 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:32:46.858+0000 7fb0d3fff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:32:46.861 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:32:46.858+0000 7fb0d3fff640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-25T18:32:46.861 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:32:46.858+0000 7fb0d3fff640 10 monclient: dump: 2024-03-25T18:32:46.861 INFO:teuthology.misc.health.smithi001.stderr:epoch 1 2024-03-25T18:32:46.861 INFO:teuthology.misc.health.smithi001.stderr:fsid c8ea72cf-9e99-4383-9b52-6c23d365fdfc 2024-03-25T18:32:46.861 INFO:teuthology.misc.health.smithi001.stderr:last_changed 2024-03-25T18:25:50.510519+0000 2024-03-25T18:32:46.861 INFO:teuthology.misc.health.smithi001.stderr:created 2024-03-25T18:25:50.510519+0000 2024-03-25T18:32:46.861 INFO:teuthology.misc.health.smithi001.stderr:min_mon_release 19 (squid) 2024-03-25T18:32:46.861 INFO:teuthology.misc.health.smithi001.stderr:election_strategy: 3 2024-03-25T18:32:46.861 INFO:teuthology.misc.health.smithi001.stderr:0: v1:172.21.15.1:6789/0 mon.a 2024-03-25T18:32:46.862 INFO:teuthology.misc.health.smithi001.stderr:1: v1:172.21.15.115:6789/0 mon.b 2024-03-25T18:32:46.862 INFO:teuthology.misc.health.smithi001.stderr:2: v1:172.21.15.1:6790/0 mon.c 2024-03-25T18:32:46.862 INFO:teuthology.misc.health.smithi001.stderr: 2024-03-25T18:32:46.862 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:32:46.858+0000 7fb0e97fa640 4 set_mon_vals no callback set 2024-03-25T18:32:46.862 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:32:46.859+0000 7fb0d3fff640 1 -- 172.21.15.1:0/3226427439 <== mon.1 v1:172.21.15.115:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 21787975 0 0) 0x7fb0e401b010 con 0x7fb0ec13c480 2024-03-25T18:32:46.862 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:32:46.860+0000 7fb0eaffd640 10 monclient: get_auth_request con 0x7fb0d806df70 auth_method 0 2024-03-25T18:32:46.863 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:32:46.860+0000 7fb0d3fff640 10 client.4495.objecter ms_handle_connect 0x7fb0d806df70 2024-03-25T18:32:46.866 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:32:46.864+0000 7fb0d3fff640 1 -- 172.21.15.1:0/3226427439 <== mon.1 v1:172.21.15.115:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 996254230 0 0) 0x7fb0e4054cb0 con 0x7fb0ec13c480 2024-03-25T18:32:46.866 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:32:46.864+0000 7fb0d3fff640 10 client.4495.objecter ms_dispatch 0x7fb0ec000b90 osd_map(10..10 src has 1..10) v4 2024-03-25T18:32:46.866 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:32:46.864+0000 7fb0d3fff640 3 client.4495.objecter handle_osd_map got epochs [10,10] > 0 2024-03-25T18:32:46.866 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:32:46.864+0000 7fb0d3fff640 3 client.4495.objecter handle_osd_map decoding full epoch 10 2024-03-25T18:32:46.866 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:32:46.864+0000 7fb0d3fff640 20 client.4495.objecter dump_active .. 0 homeless 2024-03-25T18:32:46.866 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:32:46.864+0000 7fb0d3fff640 1 -- 172.21.15.1:0/3226427439 <== mon.1 v1:172.21.15.115:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7fb0e40505f0 con 0x7fb0ec13c480 2024-03-25T18:32:46.866 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:32:46.864+0000 7fb0d3fff640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:32:46.866 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:32:46.864+0000 7fb0d3fff640 10 monclient: _finish_command 1 = system:0 2024-03-25T18:32:47.120 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:32:47.118+0000 7fb0f1a06640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:32:47.120 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:32:47.118+0000 7fb0f1a06640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:32:47.120 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:32:47.118+0000 7fb0f1a06640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.115:6789/0 2024-03-25T18:32:47.120 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:32:47.118+0000 7fb0f1a06640 1 -- 172.21.15.1:0/3226427439 --> v1:172.21.15.115:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fb0ec13d940 con 0x7fb0ec13c480 2024-03-25T18:32:47.156 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:32:47.154+0000 7fb0d3fff640 1 -- 172.21.15.1:0/3226427439 <== mon.1 v1:172.21.15.115:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 4092213348) 0x7fb0e40526f0 con 0x7fb0ec13c480 2024-03-25T18:32:47.156 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:32:47.154+0000 7fb0d3fff640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:32:47.156 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:32:47.154+0000 7fb0d3fff640 10 monclient: _finish_command 2 = system:0 2024-03-25T18:32:47.156 INFO:teuthology.misc.health.smithi001.stdout: 2024-03-25T18:32:47.157 INFO:teuthology.misc.health.smithi001.stdout:{"epoch":10,"fsid":"c8ea72cf-9e99-4383-9b52-6c23d365fdfc","created":"2024-03-25T18:26:38.856638+0000","modified":"2024-03-25T18:26:44.224208+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T18:26:44.224208+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"b9f34922-3a84-49be-8426-807c1eceaefc","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"42bf306f-55e3-4634-9874-8b4b73ac6644","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"292cb23a-18a9-4207-a7cc-4d5727feaec0","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"45e0e9ab-898d-4673-9590-b4117bc5e84a","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"831a5a09-73ab-4a61-9872-088e76bdbc66","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"88cd903d-3b55-4840-9396-6e3cfe005282","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"89db7355-bc9d-46f4-8ab0-e8d9e744e026","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"cac47bee-0914-4f08-896f-348838882866","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T18:32:47.159 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:32:47.156+0000 7fb0f1a06640 10 librados: watch_flush enter 2024-03-25T18:32:47.159 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:32:47.156+0000 7fb0f1a06640 10 librados: watch_flush exit 2024-03-25T18:32:47.159 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:32:47.156+0000 7fb0f1a06640 20 client.4495.objecter shutdown clearing up homeless session... 2024-03-25T18:32:47.159 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:32:47.156+0000 7fb0f1a06640 10 client.4495.objecter successfully canceled tick 2024-03-25T18:32:47.159 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:32:47.157+0000 7fb0f1a06640 1 -- 172.21.15.1:0/3226427439 >> v1:172.21.15.115:6800/4237379176 conn(0x7fb0d806df70 legacy=0x7fb0d8070430 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:32:47.159 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:32:47.157+0000 7fb0f1a06640 10 monclient: shutdown 2024-03-25T18:32:47.159 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:32:47.157+0000 7fb0f1a06640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:32:47.159 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:32:47.157+0000 7fb0f1a06640 1 -- 172.21.15.1:0/3226427439 >> v1:172.21.15.115:6789/0 conn(0x7fb0ec13c480 legacy=0x7fb0ec1e2590 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:32:47.159 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:32:47.157+0000 7fb0f1a06640 1 -- 172.21.15.1:0/3226427439 shutdown_connections 2024-03-25T18:32:47.160 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:32:47.157+0000 7fb0f1a06640 1 -- 172.21.15.1:0/3226427439 >> 172.21.15.1:0/3226427439 conn(0x7fb0ec0617a0 msgr2=0x7fb0ec141b80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:32:47.160 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:32:47.157+0000 7fb0f1a06640 1 -- 172.21.15.1:0/3226427439 shutdown_connections 2024-03-25T18:32:47.160 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:32:47.157+0000 7fb0f1a06640 1 -- 172.21.15.1:0/3226427439 wait complete. 2024-03-25T18:32:47.160 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:32:47.157+0000 7fb0f1a06640 1 librados: shutdown 2024-03-25T18:32:47.171 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T18:32:53.978 DEBUG:teuthology.orchestra.run.smithi001:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T18:32:54.093 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:32:54.091+0000 7fbee9655640 10 monclient: get_monmap_and_config 2024-03-25T18:32:54.094 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:32:54.091+0000 7fbee9655640 10 monclient: build_initial_monmap 2024-03-25T18:32:54.094 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:32:54.091+0000 7fbee9655640 10 monclient: monmap: 2024-03-25T18:32:54.094 INFO:teuthology.misc.health.smithi001.stderr:epoch 0 2024-03-25T18:32:54.094 INFO:teuthology.misc.health.smithi001.stderr:fsid c8ea72cf-9e99-4383-9b52-6c23d365fdfc 2024-03-25T18:32:54.094 INFO:teuthology.misc.health.smithi001.stderr:last_changed 2024-03-25T18:32:54.093268+0000 2024-03-25T18:32:54.094 INFO:teuthology.misc.health.smithi001.stderr:created 2024-03-25T18:32:54.093268+0000 2024-03-25T18:32:54.094 INFO:teuthology.misc.health.smithi001.stderr:min_mon_release 0 (unknown) 2024-03-25T18:32:54.094 INFO:teuthology.misc.health.smithi001.stderr:election_strategy: 3 2024-03-25T18:32:54.094 INFO:teuthology.misc.health.smithi001.stderr:0: v1:172.21.15.1:6789/0 mon.noname-a 2024-03-25T18:32:54.094 INFO:teuthology.misc.health.smithi001.stderr:1: v1:172.21.15.115:6789/0 mon.noname-c 2024-03-25T18:32:54.094 INFO:teuthology.misc.health.smithi001.stderr:2: v1:172.21.15.1:6790/0 mon.noname-b 2024-03-25T18:32:54.094 INFO:teuthology.misc.health.smithi001.stderr: 2024-03-25T18:32:54.096 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:32:54.092+0000 7fbee9655640 1 Processor -- start 2024-03-25T18:32:54.096 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:32:54.092+0000 7fbee9655640 1 -- start start 2024-03-25T18:32:54.096 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:32:54.092+0000 7fbee9655640 10 monclient: init 2024-03-25T18:32:54.096 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:32:54.093+0000 7fbee9655640 10 monclient: _reopen_session rank -1 2024-03-25T18:32:54.096 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:32:54.093+0000 7fbee9655640 10 monclient: _add_conns ranks=[0,1,2] 2024-03-25T18:32:54.096 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:32:54.093+0000 7fbee9655640 10 monclient(hunting): picked mon.noname-a con 0x7fbee40b4ff0 addr v1:172.21.15.1:6789/0 2024-03-25T18:32:54.096 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:32:54.093+0000 7fbee9655640 10 monclient(hunting): picked mon.noname-c con 0x7fbee40b4670 addr v1:172.21.15.115:6789/0 2024-03-25T18:32:54.096 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:32:54.093+0000 7fbee9655640 10 monclient(hunting): picked mon.noname-b con 0x7fbee413c480 addr v1:172.21.15.1:6790/0 2024-03-25T18:32:54.096 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:32:54.093+0000 7fbee9655640 1 -- --> v1:172.21.15.1:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fbee40ab0e0 con 0x7fbee40b4ff0 2024-03-25T18:32:54.097 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:32:54.093+0000 7fbee9655640 1 -- --> v1:172.21.15.115:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fbee40ab3c0 con 0x7fbee40b4670 2024-03-25T18:32:54.097 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:32:54.093+0000 7fbee9655640 1 -- --> v1:172.21.15.1:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fbee40a9f30 con 0x7fbee413c480 2024-03-25T18:32:54.097 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:32:54.093+0000 7fbee9655640 10 monclient(hunting): _renew_subs 2024-03-25T18:32:54.097 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:32:54.093+0000 7fbee9655640 10 monclient(hunting): authenticate will time out at 7854.564158s 2024-03-25T18:32:54.112 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:32:54.110+0000 7fbee37fe640 1 --1- >> v1:172.21.15.1:6790/0 conn(0x7fbee413c480 0x7fbee414dc00 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.1:6790/0 says I am v1:172.21.15.1:37666/0 (socket says 172.21.15.1:37666) 2024-03-25T18:32:54.112 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:32:54.110+0000 7fbee37fe640 1 -- 172.21.15.1:0/3828054395 learned_addr learned my addr 172.21.15.1:0/3828054395 (peer_addr_for_me v1:172.21.15.1:0/0) 2024-03-25T18:32:54.125 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:32:54.123+0000 7fbee27fc640 1 -- 172.21.15.1:0/3828054395 >> v1:172.21.15.115:6789/0 conn(0x7fbee40b4670 legacy=0x7fbee41425c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 15 2024-03-25T18:32:54.125 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:32:54.123+0000 7fbee27fc640 1 -- 172.21.15.1:0/3828054395 >> v1:172.21.15.115:6789/0 conn(0x7fbee40b4670 legacy=0x7fbee41425c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-03-25T18:32:54.125 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:32:54.123+0000 7fbee27fc640 1 --1- 172.21.15.1:0/3828054395 >> v1:172.21.15.115:6789/0 conn(0x7fbee40b4670 0x7fbee41425c0 :-1 s=OPENED pgs=45132 cs=1 l=1).handle_message read tag failed 2024-03-25T18:32:54.125 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:32:54.123+0000 7fbee27fc640 1 --1- 172.21.15.1:0/3828054395 >> v1:172.21.15.115:6789/0 conn(0x7fbee40b4670 0x7fbee41425c0 :-1 s=OPENED pgs=45132 cs=1 l=1).fault on lossy channel, failing 2024-03-25T18:32:54.125 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:32:54.123+0000 7fbee1ffb640 10 monclient(hunting): ms_handle_reset hunted mon v1:172.21.15.115:6789/0 2024-03-25T18:32:54.127 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:32:54.125+0000 7fbee1ffb640 1 -- 172.21.15.1:0/3828054395 <== mon.2 v1:172.21.15.1:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 619618623 0 0) 0x7fbee40a9f30 con 0x7fbee413c480 2024-03-25T18:32:54.127 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:32:54.125+0000 7fbee1ffb640 10 monclient(hunting): _negotiate 2024-03-25T18:32:54.127 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:32:54.125+0000 7fbee1ffb640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:32:54.127 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:32:54.125+0000 7fbee1ffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:32:54.127 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:32:54.125+0000 7fbee1ffb640 10 monclient(hunting): my global_id is 4499 2024-03-25T18:32:54.127 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:32:54.125+0000 7fbee1ffb640 1 -- 172.21.15.1:0/3828054395 --> v1:172.21.15.1:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fbecc003520 con 0x7fbee413c480 2024-03-25T18:32:54.135 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:32:54.133+0000 7fbee1ffb640 1 -- 172.21.15.1:0/3828054395 <== mon.2 v1:172.21.15.1:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2003227313 0 0) 0x7fbecc003520 con 0x7fbee413c480 2024-03-25T18:32:54.135 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:32:54.133+0000 7fbee1ffb640 1 -- 172.21.15.1:0/3828054395 >> v1:172.21.15.115:6789/0 conn(0x7fbee40b4670 legacy=0x7fbee41425c0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-03-25T18:32:54.135 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:32:54.133+0000 7fbee1ffb640 1 -- 172.21.15.1:0/3828054395 >> v1:172.21.15.1:6789/0 conn(0x7fbee40b4ff0 legacy=0x7fbee413bd70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:32:54.136 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:32:54.133+0000 7fbee1ffb640 10 monclient: _finish_hunting 0 2024-03-25T18:32:54.136 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:32:54.133+0000 7fbee1ffb640 1 monclient: found mon.noname-b 2024-03-25T18:32:54.136 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:32:54.133+0000 7fbee1ffb640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:32:54.136 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:32:54.133+0000 7fbee1ffb640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.1:6790/0 2024-03-25T18:32:54.136 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:32:54.133+0000 7fbee1ffb640 1 -- 172.21.15.1:0/3828054395 --> v1:172.21.15.1:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbee40a9b70 con 0x7fbee413c480 2024-03-25T18:32:54.136 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:32:54.133+0000 7fbee1ffb640 10 monclient: _finish_auth 0 2024-03-25T18:32:54.136 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:32:54.133+0000 7fbee1ffb640 10 monclient: _check_auth_tickets 2024-03-25T18:32:54.136 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:32:54.133+0000 7fbee1ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:32:54.136 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:32:54.133+0000 7fbee1ffb640 1 -- 172.21.15.1:0/3828054395 <== mon.2 v1:172.21.15.1:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1586965025 0 0) 0x7fbedc001690 con 0x7fbee413c480 2024-03-25T18:32:54.136 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:32:54.133+0000 7fbee1ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:32:54.136 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:32:54.133+0000 7fbee1ffb640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-03-25T18:32:54.136 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:32:54.133+0000 7fbee1ffb640 10 monclient: dump: 2024-03-25T18:32:54.136 INFO:teuthology.misc.health.smithi001.stderr:epoch 1 2024-03-25T18:32:54.137 INFO:teuthology.misc.health.smithi001.stderr:fsid c8ea72cf-9e99-4383-9b52-6c23d365fdfc 2024-03-25T18:32:54.137 INFO:teuthology.misc.health.smithi001.stderr:last_changed 2024-03-25T18:25:50.510519+0000 2024-03-25T18:32:54.137 INFO:teuthology.misc.health.smithi001.stderr:created 2024-03-25T18:25:50.510519+0000 2024-03-25T18:32:54.137 INFO:teuthology.misc.health.smithi001.stderr:min_mon_release 19 (squid) 2024-03-25T18:32:54.137 INFO:teuthology.misc.health.smithi001.stderr:election_strategy: 3 2024-03-25T18:32:54.137 INFO:teuthology.misc.health.smithi001.stderr:0: v1:172.21.15.1:6789/0 mon.a 2024-03-25T18:32:54.137 INFO:teuthology.misc.health.smithi001.stderr:1: v1:172.21.15.115:6789/0 mon.b 2024-03-25T18:32:54.137 INFO:teuthology.misc.health.smithi001.stderr:2: v1:172.21.15.1:6790/0 mon.c 2024-03-25T18:32:54.137 INFO:teuthology.misc.health.smithi001.stderr: 2024-03-25T18:32:54.137 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:32:54.133+0000 7fbee9655640 5 monclient: authenticate success, global_id 4499 2024-03-25T18:32:54.137 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:32:54.133+0000 7fbee9655640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T18:32:54.146 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:32:54.144+0000 7fbee1ffb640 1 -- 172.21.15.1:0/3828054395 <== mon.2 v1:172.21.15.1:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fbedc0017f0 con 0x7fbee413c480 2024-03-25T18:32:54.146 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:32:54.144+0000 7fbee1ffb640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:32:54.146 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:32:54.144+0000 7fbee1ffb640 1 -- 172.21.15.1:0/3828054395 <== mon.2 v1:172.21.15.1:6790/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1586965025 0 0) 0x7fbedc001c40 con 0x7fbee413c480 2024-03-25T18:32:54.146 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:32:54.144+0000 7fbee1ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:32:54.146 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:32:54.144+0000 7fbee1ffb640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-25T18:32:54.146 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:32:54.144+0000 7fbee1ffb640 10 monclient: dump: 2024-03-25T18:32:54.146 INFO:teuthology.misc.health.smithi001.stderr:epoch 1 2024-03-25T18:32:54.146 INFO:teuthology.misc.health.smithi001.stderr:fsid c8ea72cf-9e99-4383-9b52-6c23d365fdfc 2024-03-25T18:32:54.146 INFO:teuthology.misc.health.smithi001.stderr:last_changed 2024-03-25T18:25:50.510519+0000 2024-03-25T18:32:54.146 INFO:teuthology.misc.health.smithi001.stderr:created 2024-03-25T18:25:50.510519+0000 2024-03-25T18:32:54.147 INFO:teuthology.misc.health.smithi001.stderr:min_mon_release 19 (squid) 2024-03-25T18:32:54.147 INFO:teuthology.misc.health.smithi001.stderr:election_strategy: 3 2024-03-25T18:32:54.147 INFO:teuthology.misc.health.smithi001.stderr:0: v1:172.21.15.1:6789/0 mon.a 2024-03-25T18:32:54.147 INFO:teuthology.misc.health.smithi001.stderr:1: v1:172.21.15.115:6789/0 mon.b 2024-03-25T18:32:54.147 INFO:teuthology.misc.health.smithi001.stderr:2: v1:172.21.15.1:6790/0 mon.c 2024-03-25T18:32:54.147 INFO:teuthology.misc.health.smithi001.stderr: 2024-03-25T18:32:54.147 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:32:54.144+0000 7fbee9655640 10 monclient: get_monmap_and_config success 2024-03-25T18:32:54.147 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:32:54.144+0000 7fbee9655640 4 set_mon_vals no callback set 2024-03-25T18:32:54.149 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:32:54.144+0000 7fbee9655640 10 monclient: shutdown 2024-03-25T18:32:54.149 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:32:54.144+0000 7fbee9655640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:32:54.149 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:32:54.144+0000 7fbee9655640 1 -- 172.21.15.1:0/3828054395 >> v1:172.21.15.1:6790/0 conn(0x7fbee413c480 legacy=0x7fbee414dc00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:32:54.149 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:32:54.144+0000 7fbee9655640 1 -- 172.21.15.1:0/3828054395 shutdown_connections 2024-03-25T18:32:54.149 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:32:54.144+0000 7fbee9655640 1 -- 172.21.15.1:0/3828054395 >> 172.21.15.1:0/3828054395 conn(0x7fbee40617a0 msgr2=0x7fbee4061bb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:32:54.150 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:32:54.144+0000 7fbee9655640 1 -- 172.21.15.1:0/3828054395 shutdown_connections 2024-03-25T18:32:54.150 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:32:54.144+0000 7fbee9655640 1 -- 172.21.15.1:0/3828054395 wait complete. 2024-03-25T18:32:54.150 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:32:54.145+0000 7fbee9655640 10 monclient: build_initial_monmap 2024-03-25T18:32:54.150 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:32:54.145+0000 7fbee9655640 10 monclient: monmap: 2024-03-25T18:32:54.150 INFO:teuthology.misc.health.smithi001.stderr:epoch 0 2024-03-25T18:32:54.150 INFO:teuthology.misc.health.smithi001.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T18:32:54.150 INFO:teuthology.misc.health.smithi001.stderr:last_changed 0.000000 2024-03-25T18:32:54.150 INFO:teuthology.misc.health.smithi001.stderr:created 0.000000 2024-03-25T18:32:54.150 INFO:teuthology.misc.health.smithi001.stderr:min_mon_release 0 (unknown) 2024-03-25T18:32:54.150 INFO:teuthology.misc.health.smithi001.stderr:election_strategy: 1 2024-03-25T18:32:54.150 INFO:teuthology.misc.health.smithi001.stderr:0: v1:172.21.15.1:6789/0 mon.noname-a 2024-03-25T18:32:54.150 INFO:teuthology.misc.health.smithi001.stderr:1: v1:172.21.15.115:6789/0 mon.noname-b 2024-03-25T18:32:54.150 INFO:teuthology.misc.health.smithi001.stderr:2: v1:172.21.15.1:6790/0 mon.noname-c 2024-03-25T18:32:54.151 INFO:teuthology.misc.health.smithi001.stderr: 2024-03-25T18:32:54.151 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:32:54.145+0000 7fbee9655640 1 librados: starting msgr at 2024-03-25T18:32:54.151 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:32:54.145+0000 7fbee9655640 1 librados: starting objecter 2024-03-25T18:32:54.151 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:32:54.146+0000 7fbee9655640 1 Processor -- start 2024-03-25T18:32:54.151 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:32:54.146+0000 7fbee9655640 1 -- start start 2024-03-25T18:32:54.151 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:32:54.146+0000 7fbee9655640 1 librados: setting wanted keys 2024-03-25T18:32:54.151 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:32:54.146+0000 7fbee9655640 1 librados: calling monclient init 2024-03-25T18:32:54.151 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:32:54.146+0000 7fbee9655640 10 monclient: init 2024-03-25T18:32:54.151 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:32:54.146+0000 7fbee9655640 10 monclient: _reopen_session rank -1 2024-03-25T18:32:54.151 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:32:54.146+0000 7fbee9655640 10 monclient: _add_conns ranks=[1,0,2] 2024-03-25T18:32:54.151 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:32:54.146+0000 7fbee9655640 10 monclient(hunting): picked mon.noname-b con 0x7fbee40b4670 addr v1:172.21.15.115:6789/0 2024-03-25T18:32:54.151 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:32:54.146+0000 7fbee9655640 10 monclient(hunting): picked mon.noname-a con 0x7fbee40b4ff0 addr v1:172.21.15.1:6789/0 2024-03-25T18:32:54.151 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:32:54.146+0000 7fbee9655640 10 monclient(hunting): picked mon.noname-c con 0x7fbee413c480 addr v1:172.21.15.1:6790/0 2024-03-25T18:32:54.152 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:32:54.146+0000 7fbee9655640 1 -- --> v1:172.21.15.1:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fbee41467d0 con 0x7fbee40b4ff0 2024-03-25T18:32:54.152 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:32:54.146+0000 7fbee9655640 1 -- --> v1:172.21.15.115:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fbee413e680 con 0x7fbee40b4670 2024-03-25T18:32:54.152 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:32:54.146+0000 7fbee9655640 1 -- --> v1:172.21.15.1:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fbee4144490 con 0x7fbee413c480 2024-03-25T18:32:54.152 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:32:54.146+0000 7fbee9655640 10 monclient(hunting): _renew_subs 2024-03-25T18:32:54.152 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:32:54.146+0000 7fbee9655640 10 monclient(hunting): authenticate will time out at 7854.617494s 2024-03-25T18:32:54.153 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:32:54.151+0000 7fbee2ffd640 1 --1- >> v1:172.21.15.115:6789/0 conn(0x7fbee40b4670 0x7fbee413cc10 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.115:6789/0 says I am v1:172.21.15.1:58368/0 (socket says 172.21.15.1:58368) 2024-03-25T18:32:54.153 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:32:54.151+0000 7fbee2ffd640 1 -- 172.21.15.1:0/2339385148 learned_addr learned my addr 172.21.15.1:0/2339385148 (peer_addr_for_me v1:172.21.15.1:0/0) 2024-03-25T18:32:54.154 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:32:54.152+0000 7fbecb7fe640 10 client.?.objecter ms_handle_connect 0x7fbee40b4670 2024-03-25T18:32:54.154 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:32:54.152+0000 7fbecb7fe640 10 client.?.objecter resend_mon_ops 2024-03-25T18:32:54.157 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:32:54.155+0000 7fbecb7fe640 1 -- 172.21.15.1:0/2339385148 <== mon.1 v1:172.21.15.115:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3505300512 0 0) 0x7fbee413e680 con 0x7fbee40b4670 2024-03-25T18:32:54.157 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:32:54.156+0000 7fbecb7fe640 10 monclient(hunting): _negotiate 2024-03-25T18:32:54.157 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:32:54.156+0000 7fbecb7fe640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:32:54.158 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:32:54.156+0000 7fbecb7fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:32:54.158 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:32:54.156+0000 7fbecb7fe640 10 monclient(hunting): my global_id is 4501 2024-03-25T18:32:54.158 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:32:54.156+0000 7fbecb7fe640 1 -- 172.21.15.1:0/2339385148 --> v1:172.21.15.115:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fbed0004eb0 con 0x7fbee40b4670 2024-03-25T18:32:54.168 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:32:54.166+0000 7fbecb7fe640 1 -- 172.21.15.1:0/2339385148 <== mon.1 v1:172.21.15.115:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3759642471 0 0) 0x7fbed0004eb0 con 0x7fbee40b4670 2024-03-25T18:32:54.168 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:32:54.166+0000 7fbecb7fe640 1 -- 172.21.15.1:0/2339385148 --> v1:172.21.15.115:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fbee413e680 con 0x7fbee40b4670 2024-03-25T18:32:54.168 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:32:54.166+0000 7fbecb7fe640 1 -- 172.21.15.1:0/2339385148 <== mon.1 v1:172.21.15.115:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1586965025 0 0) 0x7fbed4001d50 con 0x7fbee40b4670 2024-03-25T18:32:54.168 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:32:54.166+0000 7fbecb7fe640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:32:54.168 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:32:54.166+0000 7fbecb7fe640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-03-25T18:32:54.168 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:32:54.166+0000 7fbecb7fe640 10 monclient(hunting): dump: 2024-03-25T18:32:54.168 INFO:teuthology.misc.health.smithi001.stderr:epoch 1 2024-03-25T18:32:54.168 INFO:teuthology.misc.health.smithi001.stderr:fsid c8ea72cf-9e99-4383-9b52-6c23d365fdfc 2024-03-25T18:32:54.169 INFO:teuthology.misc.health.smithi001.stderr:last_changed 2024-03-25T18:25:50.510519+0000 2024-03-25T18:32:54.169 INFO:teuthology.misc.health.smithi001.stderr:created 2024-03-25T18:25:50.510519+0000 2024-03-25T18:32:54.169 INFO:teuthology.misc.health.smithi001.stderr:min_mon_release 19 (squid) 2024-03-25T18:32:54.169 INFO:teuthology.misc.health.smithi001.stderr:election_strategy: 3 2024-03-25T18:32:54.169 INFO:teuthology.misc.health.smithi001.stderr:0: v1:172.21.15.1:6789/0 mon.a 2024-03-25T18:32:54.169 INFO:teuthology.misc.health.smithi001.stderr:1: v1:172.21.15.115:6789/0 mon.b 2024-03-25T18:32:54.169 INFO:teuthology.misc.health.smithi001.stderr:2: v1:172.21.15.1:6790/0 mon.c 2024-03-25T18:32:54.169 INFO:teuthology.misc.health.smithi001.stderr: 2024-03-25T18:32:54.169 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:32:54.166+0000 7fbecb7fe640 10 monclient(hunting): _finish_auth 0 2024-03-25T18:32:54.169 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:32:54.167+0000 7fbecb7fe640 10 client.?.objecter ms_handle_connect 0x7fbee40b4ff0 2024-03-25T18:32:54.170 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:32:54.167+0000 7fbecb7fe640 10 client.?.objecter resend_mon_ops 2024-03-25T18:32:54.171 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:32:54.169+0000 7fbecb7fe640 10 client.?.objecter ms_handle_connect 0x7fbee413c480 2024-03-25T18:32:54.171 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:32:54.169+0000 7fbecb7fe640 10 client.?.objecter resend_mon_ops 2024-03-25T18:32:54.177 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:32:54.175+0000 7fbecb7fe640 1 -- 172.21.15.1:0/2339385148 <== mon.1 v1:172.21.15.115:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 2576840914 0 0) 0x7fbee413e680 con 0x7fbee40b4670 2024-03-25T18:32:54.177 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:32:54.175+0000 7fbecb7fe640 1 -- 172.21.15.1:0/2339385148 >> v1:172.21.15.1:6790/0 conn(0x7fbee413c480 legacy=0x7fbee41e2380 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:32:54.177 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:32:54.175+0000 7fbecb7fe640 1 -- 172.21.15.1:0/2339385148 >> v1:172.21.15.1:6789/0 conn(0x7fbee40b4ff0 legacy=0x7fbee413d320 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:32:54.177 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:32:54.175+0000 7fbecb7fe640 10 monclient: _finish_hunting 0 2024-03-25T18:32:54.178 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:32:54.175+0000 7fbecb7fe640 1 monclient: found mon.b 2024-03-25T18:32:54.178 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:32:54.175+0000 7fbecb7fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:32:54.178 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:32:54.175+0000 7fbecb7fe640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.115:6789/0 2024-03-25T18:32:54.178 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:32:54.175+0000 7fbecb7fe640 1 -- 172.21.15.1:0/2339385148 --> v1:172.21.15.115:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbee4144770 con 0x7fbee40b4670 2024-03-25T18:32:54.178 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:32:54.175+0000 7fbecb7fe640 10 monclient: _finish_auth 0 2024-03-25T18:32:54.178 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:32:54.175+0000 7fbecb7fe640 10 monclient: _check_auth_tickets 2024-03-25T18:32:54.178 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:32:54.175+0000 7fbecb7fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:32:54.178 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:32:54.175+0000 7fbee9655640 5 monclient: authenticate success, global_id 4501 2024-03-25T18:32:54.178 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:32:54.175+0000 7fbee9655640 10 monclient: _renew_subs 2024-03-25T18:32:54.178 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:32:54.175+0000 7fbee9655640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.115:6789/0 2024-03-25T18:32:54.178 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:32:54.175+0000 7fbee9655640 1 -- 172.21.15.1:0/2339385148 --> v1:172.21.15.115:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fbee4144a50 con 0x7fbee40b4670 2024-03-25T18:32:54.178 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:32:54.175+0000 7fbee9655640 10 client.4501.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T18:32:54.179 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:32:54.175+0000 7fbee9655640 10 monclient: _renew_subs 2024-03-25T18:32:54.179 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:32:54.175+0000 7fbee9655640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.115:6789/0 2024-03-25T18:32:54.179 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:32:54.175+0000 7fbee9655640 1 -- 172.21.15.1:0/2339385148 --> v1:172.21.15.115:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7fbee41463b0 con 0x7fbee40b4670 2024-03-25T18:32:54.179 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:32:54.175+0000 7fbee9655640 1 librados: init done 2024-03-25T18:32:54.179 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:32:54.176+0000 7fbee9655640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T18:32:54.179 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:32:54.176+0000 7fbee9655640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:32:54.179 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:32:54.176+0000 7fbee9655640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.115:6789/0 2024-03-25T18:32:54.179 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:32:54.176+0000 7fbee9655640 1 -- 172.21.15.1:0/2339385148 --> v1:172.21.15.115:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fbee406dbd0 con 0x7fbee40b4670 2024-03-25T18:32:54.446 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:32:54.444+0000 7fbecb7fe640 1 -- 172.21.15.1:0/2339385148 <== mon.1 v1:172.21.15.115:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fbed4001eb0 con 0x7fbee40b4670 2024-03-25T18:32:54.446 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:32:54.444+0000 7fbecb7fe640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:32:54.446 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:32:54.444+0000 7fbecb7fe640 1 -- 172.21.15.1:0/2339385148 <== mon.1 v1:172.21.15.115:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1586965025 0 0) 0x7fbed4003de0 con 0x7fbee40b4670 2024-03-25T18:32:54.446 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:32:54.444+0000 7fbecb7fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:32:54.446 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:32:54.444+0000 7fbecb7fe640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-25T18:32:54.447 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:32:54.444+0000 7fbecb7fe640 10 monclient: dump: 2024-03-25T18:32:54.447 INFO:teuthology.misc.health.smithi001.stderr:epoch 1 2024-03-25T18:32:54.447 INFO:teuthology.misc.health.smithi001.stderr:fsid c8ea72cf-9e99-4383-9b52-6c23d365fdfc 2024-03-25T18:32:54.447 INFO:teuthology.misc.health.smithi001.stderr:last_changed 2024-03-25T18:25:50.510519+0000 2024-03-25T18:32:54.447 INFO:teuthology.misc.health.smithi001.stderr:created 2024-03-25T18:25:50.510519+0000 2024-03-25T18:32:54.447 INFO:teuthology.misc.health.smithi001.stderr:min_mon_release 19 (squid) 2024-03-25T18:32:54.447 INFO:teuthology.misc.health.smithi001.stderr:election_strategy: 3 2024-03-25T18:32:54.447 INFO:teuthology.misc.health.smithi001.stderr:0: v1:172.21.15.1:6789/0 mon.a 2024-03-25T18:32:54.447 INFO:teuthology.misc.health.smithi001.stderr:1: v1:172.21.15.115:6789/0 mon.b 2024-03-25T18:32:54.447 INFO:teuthology.misc.health.smithi001.stderr:2: v1:172.21.15.1:6790/0 mon.c 2024-03-25T18:32:54.447 INFO:teuthology.misc.health.smithi001.stderr: 2024-03-25T18:32:54.447 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:32:54.444+0000 7fbee17fa640 4 set_mon_vals no callback set 2024-03-25T18:32:54.448 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:32:54.445+0000 7fbecb7fe640 1 -- 172.21.15.1:0/2339385148 <== mon.1 v1:172.21.15.115:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 21787975 0 0) 0x7fbed4004000 con 0x7fbee40b4670 2024-03-25T18:32:54.448 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:32:54.446+0000 7fbee27fc640 10 monclient: get_auth_request con 0x7fbed006dbf0 auth_method 0 2024-03-25T18:32:54.448 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:32:54.446+0000 7fbecb7fe640 10 client.4501.objecter ms_handle_connect 0x7fbed006dbf0 2024-03-25T18:32:54.452 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:32:54.449+0000 7fbecb7fe640 1 -- 172.21.15.1:0/2339385148 <== mon.1 v1:172.21.15.115:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 996254230 0 0) 0x7fbed4054c10 con 0x7fbee40b4670 2024-03-25T18:32:54.452 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:32:54.449+0000 7fbecb7fe640 10 client.4501.objecter ms_dispatch 0x7fbee4000b90 osd_map(10..10 src has 1..10) v4 2024-03-25T18:32:54.452 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:32:54.449+0000 7fbecb7fe640 3 client.4501.objecter handle_osd_map got epochs [10,10] > 0 2024-03-25T18:32:54.452 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:32:54.449+0000 7fbecb7fe640 3 client.4501.objecter handle_osd_map decoding full epoch 10 2024-03-25T18:32:54.452 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:32:54.450+0000 7fbecb7fe640 20 client.4501.objecter dump_active .. 0 homeless 2024-03-25T18:32:54.452 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:32:54.450+0000 7fbecb7fe640 1 -- 172.21.15.1:0/2339385148 <== mon.1 v1:172.21.15.115:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7fbed4050720 con 0x7fbee40b4670 2024-03-25T18:32:54.452 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:32:54.450+0000 7fbecb7fe640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:32:54.452 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:32:54.450+0000 7fbecb7fe640 10 monclient: _finish_command 1 = system:0 2024-03-25T18:32:54.723 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:32:54.721+0000 7fbee9655640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:32:54.723 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:32:54.721+0000 7fbee9655640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:32:54.723 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:32:54.721+0000 7fbee9655640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.115:6789/0 2024-03-25T18:32:54.723 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:32:54.721+0000 7fbee9655640 1 -- 172.21.15.1:0/2339385148 --> v1:172.21.15.115:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fbee40688c0 con 0x7fbee40b4670 2024-03-25T18:32:54.742 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:32:54.740+0000 7fbecb7fe640 1 -- 172.21.15.1:0/2339385148 <== mon.1 v1:172.21.15.115:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 4092213348) 0x7fbed4052650 con 0x7fbee40b4670 2024-03-25T18:32:54.742 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:32:54.740+0000 7fbecb7fe640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:32:54.742 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:32:54.740+0000 7fbecb7fe640 10 monclient: _finish_command 2 = system:0 2024-03-25T18:32:54.742 INFO:teuthology.misc.health.smithi001.stdout: 2024-03-25T18:32:54.742 INFO:teuthology.misc.health.smithi001.stdout:{"epoch":10,"fsid":"c8ea72cf-9e99-4383-9b52-6c23d365fdfc","created":"2024-03-25T18:26:38.856638+0000","modified":"2024-03-25T18:26:44.224208+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T18:26:44.224208+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"b9f34922-3a84-49be-8426-807c1eceaefc","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"42bf306f-55e3-4634-9874-8b4b73ac6644","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"292cb23a-18a9-4207-a7cc-4d5727feaec0","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"45e0e9ab-898d-4673-9590-b4117bc5e84a","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"831a5a09-73ab-4a61-9872-088e76bdbc66","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"88cd903d-3b55-4840-9396-6e3cfe005282","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"89db7355-bc9d-46f4-8ab0-e8d9e744e026","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"cac47bee-0914-4f08-896f-348838882866","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T18:32:54.744 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:32:54.742+0000 7fbee9655640 10 librados: watch_flush enter 2024-03-25T18:32:54.745 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:32:54.742+0000 7fbee9655640 10 librados: watch_flush exit 2024-03-25T18:32:54.745 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:32:54.742+0000 7fbee9655640 20 client.4501.objecter shutdown clearing up homeless session... 2024-03-25T18:32:54.745 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:32:54.742+0000 7fbee9655640 10 client.4501.objecter successfully canceled tick 2024-03-25T18:32:54.745 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:32:54.742+0000 7fbee9655640 1 -- 172.21.15.1:0/2339385148 >> v1:172.21.15.115:6800/4237379176 conn(0x7fbed006dbf0 legacy=0x7fbed00700b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:32:54.745 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:32:54.742+0000 7fbee9655640 10 monclient: shutdown 2024-03-25T18:32:54.745 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:32:54.742+0000 7fbee9655640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:32:54.745 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:32:54.742+0000 7fbee9655640 1 -- 172.21.15.1:0/2339385148 >> v1:172.21.15.115:6789/0 conn(0x7fbee40b4670 legacy=0x7fbee413cc10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:32:54.745 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:32:54.742+0000 7fbee9655640 1 -- 172.21.15.1:0/2339385148 shutdown_connections 2024-03-25T18:32:54.745 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:32:54.742+0000 7fbee9655640 1 -- 172.21.15.1:0/2339385148 >> 172.21.15.1:0/2339385148 conn(0x7fbee40617a0 msgr2=0x7fbee4140760 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:32:54.745 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:32:54.743+0000 7fbee9655640 1 -- 172.21.15.1:0/2339385148 shutdown_connections 2024-03-25T18:32:54.745 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:32:54.743+0000 7fbee9655640 1 -- 172.21.15.1:0/2339385148 wait complete. 2024-03-25T18:32:54.745 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:32:54.743+0000 7fbee9655640 1 librados: shutdown 2024-03-25T18:32:54.758 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T18:33:01.566 DEBUG:teuthology.orchestra.run.smithi001:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T18:33:01.682 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:33:01.680+0000 7f18c52d6640 10 monclient: get_monmap_and_config 2024-03-25T18:33:01.682 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:33:01.680+0000 7f18c52d6640 10 monclient: build_initial_monmap 2024-03-25T18:33:01.682 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:33:01.680+0000 7f18c52d6640 10 monclient: monmap: 2024-03-25T18:33:01.682 INFO:teuthology.misc.health.smithi001.stderr:epoch 0 2024-03-25T18:33:01.682 INFO:teuthology.misc.health.smithi001.stderr:fsid c8ea72cf-9e99-4383-9b52-6c23d365fdfc 2024-03-25T18:33:01.682 INFO:teuthology.misc.health.smithi001.stderr:last_changed 2024-03-25T18:33:01.681597+0000 2024-03-25T18:33:01.682 INFO:teuthology.misc.health.smithi001.stderr:created 2024-03-25T18:33:01.681597+0000 2024-03-25T18:33:01.682 INFO:teuthology.misc.health.smithi001.stderr:min_mon_release 0 (unknown) 2024-03-25T18:33:01.683 INFO:teuthology.misc.health.smithi001.stderr:election_strategy: 3 2024-03-25T18:33:01.683 INFO:teuthology.misc.health.smithi001.stderr:0: v1:172.21.15.1:6789/0 mon.noname-a 2024-03-25T18:33:01.683 INFO:teuthology.misc.health.smithi001.stderr:1: v1:172.21.15.115:6789/0 mon.noname-c 2024-03-25T18:33:01.683 INFO:teuthology.misc.health.smithi001.stderr:2: v1:172.21.15.1:6790/0 mon.noname-b 2024-03-25T18:33:01.683 INFO:teuthology.misc.health.smithi001.stderr: 2024-03-25T18:33:01.684 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:33:01.680+0000 7f18c52d6640 1 Processor -- start 2024-03-25T18:33:01.684 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:33:01.680+0000 7f18c52d6640 1 -- start start 2024-03-25T18:33:01.684 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:33:01.680+0000 7f18c52d6640 10 monclient: init 2024-03-25T18:33:01.684 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:33:01.681+0000 7f18c52d6640 10 monclient: _reopen_session rank -1 2024-03-25T18:33:01.684 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:33:01.681+0000 7f18c52d6640 10 monclient: _add_conns ranks=[2,1,0] 2024-03-25T18:33:01.684 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:33:01.681+0000 7f18c52d6640 10 monclient(hunting): picked mon.noname-b con 0x7f18c00b4ff0 addr v1:172.21.15.1:6790/0 2024-03-25T18:33:01.684 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:33:01.681+0000 7f18c52d6640 10 monclient(hunting): picked mon.noname-c con 0x7f18c00b4670 addr v1:172.21.15.115:6789/0 2024-03-25T18:33:01.685 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:33:01.681+0000 7f18c52d6640 10 monclient(hunting): picked mon.noname-a con 0x7f18c013c480 addr v1:172.21.15.1:6789/0 2024-03-25T18:33:01.685 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:33:01.681+0000 7f18c52d6640 1 -- --> v1:172.21.15.1:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f18c00ab0e0 con 0x7f18c013c480 2024-03-25T18:33:01.685 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:33:01.681+0000 7f18c52d6640 1 -- --> v1:172.21.15.115:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f18c00ab3c0 con 0x7f18c00b4670 2024-03-25T18:33:01.685 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:33:01.681+0000 7f18c52d6640 1 -- --> v1:172.21.15.1:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f18c00a9f30 con 0x7f18c00b4ff0 2024-03-25T18:33:01.685 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:33:01.681+0000 7f18c52d6640 10 monclient(hunting): _renew_subs 2024-03-25T18:33:01.685 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:33:01.681+0000 7f18c52d6640 10 monclient(hunting): authenticate will time out at 7862.152461s 2024-03-25T18:33:01.698 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:33:01.695+0000 7f18be7fc640 1 --1- >> v1:172.21.15.115:6789/0 conn(0x7f18c00b4670 0x7f18c01425c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.115:6789/0 says I am v1:172.21.15.1:34820/0 (socket says 172.21.15.1:34820) 2024-03-25T18:33:01.698 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:33:01.695+0000 7f18be7fc640 1 -- 172.21.15.1:0/3820343633 learned_addr learned my addr 172.21.15.1:0/3820343633 (peer_addr_for_me v1:172.21.15.1:0/0) 2024-03-25T18:33:01.717 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:33:01.715+0000 7f18bdffb640 1 -- 172.21.15.1:0/3820343633 <== mon.1 v1:172.21.15.115:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 617659745 0 0) 0x7f18c00ab3c0 con 0x7f18c00b4670 2024-03-25T18:33:01.717 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:33:01.715+0000 7f18bdffb640 10 monclient(hunting): _negotiate 2024-03-25T18:33:01.717 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:33:01.715+0000 7f18bdffb640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:33:01.717 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:33:01.715+0000 7f18bdffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:33:01.717 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:33:01.715+0000 7f18bdffb640 10 monclient(hunting): my global_id is 4504 2024-03-25T18:33:01.717 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:33:01.715+0000 7f18bdffb640 1 -- 172.21.15.1:0/3820343633 --> v1:172.21.15.115:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f18a8003520 con 0x7f18c00b4670 2024-03-25T18:33:01.719 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:33:01.717+0000 7f18bdffb640 1 -- 172.21.15.1:0/3820343633 <== mon.2 v1:172.21.15.1:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2046038082 0 0) 0x7f18c00a9f30 con 0x7f18c00b4ff0 2024-03-25T18:33:01.719 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:33:01.717+0000 7f18bdffb640 10 monclient(hunting): _negotiate 2024-03-25T18:33:01.719 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:33:01.717+0000 7f18bdffb640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:33:01.719 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:33:01.717+0000 7f18bdffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:33:01.719 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:33:01.717+0000 7f18bdffb640 10 monclient(hunting): my global_id is 4505 2024-03-25T18:33:01.719 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:33:01.717+0000 7f18bdffb640 1 -- 172.21.15.1:0/3820343633 --> v1:172.21.15.1:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f18c00ab3c0 con 0x7f18c00b4ff0 2024-03-25T18:33:01.727 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:33:01.725+0000 7f18bdffb640 1 -- 172.21.15.1:0/3820343633 <== mon.0 v1:172.21.15.1:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 152725076 0 0) 0x7f18c00ab0e0 con 0x7f18c013c480 2024-03-25T18:33:01.727 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:33:01.725+0000 7f18bdffb640 10 monclient(hunting): _negotiate 2024-03-25T18:33:01.728 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:33:01.725+0000 7f18bdffb640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:33:01.728 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:33:01.725+0000 7f18bdffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:33:01.728 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:33:01.725+0000 7f18bdffb640 10 monclient(hunting): my global_id is 4491 2024-03-25T18:33:01.728 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:33:01.725+0000 7f18bdffb640 1 -- 172.21.15.1:0/3820343633 --> v1:172.21.15.1:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f18c00a9f30 con 0x7f18c013c480 2024-03-25T18:33:01.729 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:33:01.726+0000 7f18bdffb640 1 -- 172.21.15.1:0/3820343633 <== mon.1 v1:172.21.15.115:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2662190133 0 0) 0x7f18a8003520 con 0x7f18c00b4670 2024-03-25T18:33:01.729 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:33:01.726+0000 7f18bdffb640 1 -- 172.21.15.1:0/3820343633 >> v1:172.21.15.1:6790/0 conn(0x7f18c00b4ff0 legacy=0x7f18c013bd70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:33:01.729 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:33:01.726+0000 7f18bdffb640 1 -- 172.21.15.1:0/3820343633 >> v1:172.21.15.1:6789/0 conn(0x7f18c013c480 legacy=0x7f18c014dc00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:33:01.729 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:33:01.726+0000 7f18bdffb640 10 monclient: _finish_hunting 0 2024-03-25T18:33:01.729 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:33:01.726+0000 7f18bdffb640 1 monclient: found mon.noname-c 2024-03-25T18:33:01.729 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:33:01.726+0000 7f18bdffb640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:33:01.729 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:33:01.726+0000 7f18bdffb640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.115:6789/0 2024-03-25T18:33:01.730 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:33:01.726+0000 7f18bdffb640 1 -- 172.21.15.1:0/3820343633 --> v1:172.21.15.115:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f18c00a9b70 con 0x7f18c00b4670 2024-03-25T18:33:01.730 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:33:01.726+0000 7f18bdffb640 10 monclient: _finish_auth 0 2024-03-25T18:33:01.730 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:33:01.726+0000 7f18bdffb640 10 monclient: _check_auth_tickets 2024-03-25T18:33:01.730 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:33:01.726+0000 7f18bdffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:33:01.730 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:33:01.726+0000 7f18c52d6640 5 monclient: authenticate success, global_id 4504 2024-03-25T18:33:01.730 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:33:01.726+0000 7f18bdffb640 1 -- 172.21.15.1:0/3820343633 <== mon.1 v1:172.21.15.115:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1586965025 0 0) 0x7f18b4001690 con 0x7f18c00b4670 2024-03-25T18:33:01.730 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:33:01.726+0000 7f18bdffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:33:01.730 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:33:01.726+0000 7f18bdffb640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-03-25T18:33:01.730 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:33:01.726+0000 7f18bdffb640 10 monclient: dump: 2024-03-25T18:33:01.730 INFO:teuthology.misc.health.smithi001.stderr:epoch 1 2024-03-25T18:33:01.730 INFO:teuthology.misc.health.smithi001.stderr:fsid c8ea72cf-9e99-4383-9b52-6c23d365fdfc 2024-03-25T18:33:01.731 INFO:teuthology.misc.health.smithi001.stderr:last_changed 2024-03-25T18:25:50.510519+0000 2024-03-25T18:33:01.731 INFO:teuthology.misc.health.smithi001.stderr:created 2024-03-25T18:25:50.510519+0000 2024-03-25T18:33:01.731 INFO:teuthology.misc.health.smithi001.stderr:min_mon_release 19 (squid) 2024-03-25T18:33:01.731 INFO:teuthology.misc.health.smithi001.stderr:election_strategy: 3 2024-03-25T18:33:01.731 INFO:teuthology.misc.health.smithi001.stderr:0: v1:172.21.15.1:6789/0 mon.a 2024-03-25T18:33:01.731 INFO:teuthology.misc.health.smithi001.stderr:1: v1:172.21.15.115:6789/0 mon.b 2024-03-25T18:33:01.731 INFO:teuthology.misc.health.smithi001.stderr:2: v1:172.21.15.1:6790/0 mon.c 2024-03-25T18:33:01.731 INFO:teuthology.misc.health.smithi001.stderr: 2024-03-25T18:33:01.731 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:33:01.726+0000 7f18c52d6640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T18:33:01.743 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:33:01.741+0000 7f18bdffb640 1 -- 172.21.15.1:0/3820343633 <== mon.1 v1:172.21.15.115:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f18b40017f0 con 0x7f18c00b4670 2024-03-25T18:33:01.743 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:33:01.741+0000 7f18bdffb640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:33:01.743 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:33:01.741+0000 7f18bdffb640 1 -- 172.21.15.1:0/3820343633 <== mon.1 v1:172.21.15.115:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1586965025 0 0) 0x7f18b4001c00 con 0x7f18c00b4670 2024-03-25T18:33:01.743 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:33:01.741+0000 7f18bdffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:33:01.743 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:33:01.741+0000 7f18bdffb640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-25T18:33:01.743 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:33:01.741+0000 7f18bdffb640 10 monclient: dump: 2024-03-25T18:33:01.743 INFO:teuthology.misc.health.smithi001.stderr:epoch 1 2024-03-25T18:33:01.743 INFO:teuthology.misc.health.smithi001.stderr:fsid c8ea72cf-9e99-4383-9b52-6c23d365fdfc 2024-03-25T18:33:01.744 INFO:teuthology.misc.health.smithi001.stderr:last_changed 2024-03-25T18:25:50.510519+0000 2024-03-25T18:33:01.744 INFO:teuthology.misc.health.smithi001.stderr:created 2024-03-25T18:25:50.510519+0000 2024-03-25T18:33:01.744 INFO:teuthology.misc.health.smithi001.stderr:min_mon_release 19 (squid) 2024-03-25T18:33:01.744 INFO:teuthology.misc.health.smithi001.stderr:election_strategy: 3 2024-03-25T18:33:01.744 INFO:teuthology.misc.health.smithi001.stderr:0: v1:172.21.15.1:6789/0 mon.a 2024-03-25T18:33:01.744 INFO:teuthology.misc.health.smithi001.stderr:1: v1:172.21.15.115:6789/0 mon.b 2024-03-25T18:33:01.744 INFO:teuthology.misc.health.smithi001.stderr:2: v1:172.21.15.1:6790/0 mon.c 2024-03-25T18:33:01.744 INFO:teuthology.misc.health.smithi001.stderr: 2024-03-25T18:33:01.744 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:33:01.741+0000 7f18c52d6640 10 monclient: get_monmap_and_config success 2024-03-25T18:33:01.744 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:33:01.741+0000 7f18c52d6640 4 set_mon_vals no callback set 2024-03-25T18:33:01.746 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:33:01.742+0000 7f18c52d6640 10 monclient: shutdown 2024-03-25T18:33:01.747 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:33:01.742+0000 7f18c52d6640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:33:01.747 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:33:01.742+0000 7f18c52d6640 1 -- 172.21.15.1:0/3820343633 >> v1:172.21.15.115:6789/0 conn(0x7f18c00b4670 legacy=0x7f18c01425c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:33:01.747 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:33:01.742+0000 7f18c52d6640 1 -- 172.21.15.1:0/3820343633 shutdown_connections 2024-03-25T18:33:01.747 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:33:01.742+0000 7f18c52d6640 1 -- 172.21.15.1:0/3820343633 >> 172.21.15.1:0/3820343633 conn(0x7f18c00617a0 msgr2=0x7f18c0061bb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:33:01.747 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:33:01.742+0000 7f18c52d6640 1 -- 172.21.15.1:0/3820343633 shutdown_connections 2024-03-25T18:33:01.747 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:33:01.742+0000 7f18c52d6640 1 -- 172.21.15.1:0/3820343633 wait complete. 2024-03-25T18:33:01.747 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:33:01.743+0000 7f18c52d6640 10 monclient: build_initial_monmap 2024-03-25T18:33:01.747 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:33:01.743+0000 7f18c52d6640 10 monclient: monmap: 2024-03-25T18:33:01.747 INFO:teuthology.misc.health.smithi001.stderr:epoch 0 2024-03-25T18:33:01.747 INFO:teuthology.misc.health.smithi001.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T18:33:01.747 INFO:teuthology.misc.health.smithi001.stderr:last_changed 0.000000 2024-03-25T18:33:01.748 INFO:teuthology.misc.health.smithi001.stderr:created 0.000000 2024-03-25T18:33:01.748 INFO:teuthology.misc.health.smithi001.stderr:min_mon_release 0 (unknown) 2024-03-25T18:33:01.748 INFO:teuthology.misc.health.smithi001.stderr:election_strategy: 1 2024-03-25T18:33:01.748 INFO:teuthology.misc.health.smithi001.stderr:0: v1:172.21.15.1:6789/0 mon.noname-a 2024-03-25T18:33:01.748 INFO:teuthology.misc.health.smithi001.stderr:1: v1:172.21.15.115:6789/0 mon.noname-b 2024-03-25T18:33:01.748 INFO:teuthology.misc.health.smithi001.stderr:2: v1:172.21.15.1:6790/0 mon.noname-c 2024-03-25T18:33:01.748 INFO:teuthology.misc.health.smithi001.stderr: 2024-03-25T18:33:01.748 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:33:01.743+0000 7f18c52d6640 1 librados: starting msgr at 2024-03-25T18:33:01.748 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:33:01.743+0000 7f18c52d6640 1 librados: starting objecter 2024-03-25T18:33:01.748 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:33:01.743+0000 7f18c52d6640 1 Processor -- start 2024-03-25T18:33:01.748 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:33:01.744+0000 7f18c52d6640 1 -- start start 2024-03-25T18:33:01.748 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:33:01.744+0000 7f18c52d6640 1 librados: setting wanted keys 2024-03-25T18:33:01.749 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:33:01.744+0000 7f18c52d6640 1 librados: calling monclient init 2024-03-25T18:33:01.749 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:33:01.744+0000 7f18c52d6640 10 monclient: init 2024-03-25T18:33:01.749 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:33:01.744+0000 7f18c52d6640 10 monclient: _reopen_session rank -1 2024-03-25T18:33:01.749 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:33:01.744+0000 7f18c52d6640 10 monclient: _add_conns ranks=[0,2,1] 2024-03-25T18:33:01.749 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:33:01.744+0000 7f18c52d6640 10 monclient(hunting): picked mon.noname-a con 0x7f18c00b4670 addr v1:172.21.15.1:6789/0 2024-03-25T18:33:01.749 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:33:01.744+0000 7f18c52d6640 10 monclient(hunting): picked mon.noname-c con 0x7f18c00b4ff0 addr v1:172.21.15.1:6790/0 2024-03-25T18:33:01.749 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:33:01.744+0000 7f18c52d6640 10 monclient(hunting): picked mon.noname-b con 0x7f18c013c480 addr v1:172.21.15.115:6789/0 2024-03-25T18:33:01.749 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:33:01.744+0000 7f18c52d6640 1 -- --> v1:172.21.15.1:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f18c00af4f0 con 0x7f18c00b4670 2024-03-25T18:33:01.749 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:33:01.744+0000 7f18c52d6640 1 -- --> v1:172.21.15.115:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f18c00ada40 con 0x7f18c013c480 2024-03-25T18:33:01.749 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:33:01.744+0000 7f18c52d6640 1 -- --> v1:172.21.15.1:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f18c00afcd0 con 0x7f18c00b4ff0 2024-03-25T18:33:01.749 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:33:01.744+0000 7f18c52d6640 10 monclient(hunting): _renew_subs 2024-03-25T18:33:01.749 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:33:01.744+0000 7f18c52d6640 10 monclient(hunting): authenticate will time out at 7862.215419s 2024-03-25T18:33:01.762 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:33:01.760+0000 7f18bf7fe640 1 --1- >> v1:172.21.15.115:6789/0 conn(0x7f18c013c480 0x7f18c00b10a0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.115:6789/0 says I am v1:172.21.15.1:34858/0 (socket says 172.21.15.1:34858) 2024-03-25T18:33:01.762 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:33:01.760+0000 7f18bf7fe640 1 -- 172.21.15.1:0/3621274813 learned_addr learned my addr 172.21.15.1:0/3621274813 (peer_addr_for_me v1:172.21.15.1:0/0) 2024-03-25T18:33:01.762 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:33:01.760+0000 7f18a77fe640 10 client.?.objecter ms_handle_connect 0x7f18c013c480 2024-03-25T18:33:01.762 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:33:01.760+0000 7f18a77fe640 10 client.?.objecter resend_mon_ops 2024-03-25T18:33:01.765 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:33:01.763+0000 7f18a77fe640 10 client.?.objecter ms_handle_connect 0x7f18c00b4ff0 2024-03-25T18:33:01.765 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:33:01.763+0000 7f18a77fe640 10 client.?.objecter resend_mon_ops 2024-03-25T18:33:01.774 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:33:01.772+0000 7f18a77fe640 1 -- 172.21.15.1:0/3621274813 <== mon.1 v1:172.21.15.115:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3081734089 0 0) 0x7f18c00ada40 con 0x7f18c013c480 2024-03-25T18:33:01.774 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:33:01.772+0000 7f18a77fe640 10 monclient(hunting): _negotiate 2024-03-25T18:33:01.774 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:33:01.772+0000 7f18a77fe640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:33:01.774 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:33:01.772+0000 7f18a77fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:33:01.775 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:33:01.772+0000 7f18a77fe640 10 monclient(hunting): my global_id is 4507 2024-03-25T18:33:01.775 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:33:01.772+0000 7f18a77fe640 1 -- 172.21.15.1:0/3621274813 --> v1:172.21.15.115:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f18ac004eb0 con 0x7f18c013c480 2024-03-25T18:33:01.779 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:33:01.777+0000 7f18a77fe640 10 client.?.objecter ms_handle_connect 0x7f18c00b4670 2024-03-25T18:33:01.779 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:33:01.777+0000 7f18a77fe640 10 client.?.objecter resend_mon_ops 2024-03-25T18:33:01.779 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:33:01.778+0000 7f18beffd640 1 -- 172.21.15.1:0/3621274813 >> v1:172.21.15.1:6789/0 conn(0x7f18c00b4670 legacy=0x7f18c013a0f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 16 2024-03-25T18:33:01.780 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:33:01.778+0000 7f18beffd640 1 -- 172.21.15.1:0/3621274813 >> v1:172.21.15.1:6789/0 conn(0x7f18c00b4670 legacy=0x7f18c013a0f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-03-25T18:33:01.780 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:33:01.778+0000 7f18beffd640 1 --1- 172.21.15.1:0/3621274813 >> v1:172.21.15.1:6789/0 conn(0x7f18c00b4670 0x7f18c013a0f0 :-1 s=OPENED pgs=45334 cs=1 l=1).handle_message read tag failed 2024-03-25T18:33:01.780 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:33:01.778+0000 7f18beffd640 1 --1- 172.21.15.1:0/3621274813 >> v1:172.21.15.1:6789/0 conn(0x7f18c00b4670 0x7f18c013a0f0 :-1 s=OPENED pgs=45334 cs=1 l=1).fault on lossy channel, failing 2024-03-25T18:33:01.780 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:33:01.778+0000 7f18a77fe640 10 monclient(hunting): ms_handle_reset hunted mon v1:172.21.15.1:6789/0 2024-03-25T18:33:01.787 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:33:01.785+0000 7f18a77fe640 1 -- 172.21.15.1:0/3621274813 <== mon.2 v1:172.21.15.1:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1893631263 0 0) 0x7f18c00afcd0 con 0x7f18c00b4ff0 2024-03-25T18:33:01.787 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:33:01.785+0000 7f18a77fe640 10 monclient(hunting): _negotiate 2024-03-25T18:33:01.787 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:33:01.785+0000 7f18a77fe640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:33:01.787 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:33:01.785+0000 7f18a77fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:33:01.787 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:33:01.785+0000 7f18a77fe640 10 monclient(hunting): my global_id is 4508 2024-03-25T18:33:01.787 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:33:01.785+0000 7f18a77fe640 1 -- 172.21.15.1:0/3621274813 --> v1:172.21.15.1:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f18c00ada40 con 0x7f18c00b4ff0 2024-03-25T18:33:01.797 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:33:01.795+0000 7f18a77fe640 1 -- 172.21.15.1:0/3621274813 <== mon.1 v1:172.21.15.115:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 491406923 0 0) 0x7f18ac004eb0 con 0x7f18c013c480 2024-03-25T18:33:01.797 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:33:01.795+0000 7f18a77fe640 1 -- 172.21.15.1:0/3621274813 --> v1:172.21.15.115:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f18c00afcd0 con 0x7f18c013c480 2024-03-25T18:33:01.797 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:33:01.795+0000 7f18a77fe640 1 -- 172.21.15.1:0/3621274813 <== mon.1 v1:172.21.15.115:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1586965025 0 0) 0x7f18b8001c70 con 0x7f18c013c480 2024-03-25T18:33:01.797 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:33:01.795+0000 7f18a77fe640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:33:01.797 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:33:01.795+0000 7f18a77fe640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-03-25T18:33:01.797 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:33:01.795+0000 7f18a77fe640 10 monclient(hunting): dump: 2024-03-25T18:33:01.797 INFO:teuthology.misc.health.smithi001.stderr:epoch 1 2024-03-25T18:33:01.797 INFO:teuthology.misc.health.smithi001.stderr:fsid c8ea72cf-9e99-4383-9b52-6c23d365fdfc 2024-03-25T18:33:01.798 INFO:teuthology.misc.health.smithi001.stderr:last_changed 2024-03-25T18:25:50.510519+0000 2024-03-25T18:33:01.798 INFO:teuthology.misc.health.smithi001.stderr:created 2024-03-25T18:25:50.510519+0000 2024-03-25T18:33:01.798 INFO:teuthology.misc.health.smithi001.stderr:min_mon_release 19 (squid) 2024-03-25T18:33:01.798 INFO:teuthology.misc.health.smithi001.stderr:election_strategy: 3 2024-03-25T18:33:01.798 INFO:teuthology.misc.health.smithi001.stderr:0: v1:172.21.15.1:6789/0 mon.a 2024-03-25T18:33:01.798 INFO:teuthology.misc.health.smithi001.stderr:1: v1:172.21.15.115:6789/0 mon.b 2024-03-25T18:33:01.798 INFO:teuthology.misc.health.smithi001.stderr:2: v1:172.21.15.1:6790/0 mon.c 2024-03-25T18:33:01.798 INFO:teuthology.misc.health.smithi001.stderr: 2024-03-25T18:33:01.798 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:33:01.795+0000 7f18a77fe640 10 monclient(hunting): _finish_auth 0 2024-03-25T18:33:01.807 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:33:01.805+0000 7f18a77fe640 1 -- 172.21.15.1:0/3621274813 <== mon.1 v1:172.21.15.115:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 3222433477 0 0) 0x7f18c00afcd0 con 0x7f18c013c480 2024-03-25T18:33:01.807 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:33:01.805+0000 7f18a77fe640 1 -- 172.21.15.1:0/3621274813 >> v1:172.21.15.1:6790/0 conn(0x7f18c00b4ff0 legacy=0x7f18c00b0990 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:33:01.807 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:33:01.805+0000 7f18a77fe640 1 -- 172.21.15.1:0/3621274813 >> v1:172.21.15.1:6789/0 conn(0x7f18c00b4670 legacy=0x7f18c013a0f0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-03-25T18:33:01.807 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:33:01.805+0000 7f18a77fe640 10 monclient: _finish_hunting 0 2024-03-25T18:33:01.808 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:33:01.805+0000 7f18a77fe640 1 monclient: found mon.b 2024-03-25T18:33:01.808 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:33:01.805+0000 7f18a77fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:33:01.808 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:33:01.805+0000 7f18a77fe640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.115:6789/0 2024-03-25T18:33:01.808 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:33:01.805+0000 7f18a77fe640 1 -- 172.21.15.1:0/3621274813 --> v1:172.21.15.115:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f18c00affb0 con 0x7f18c013c480 2024-03-25T18:33:01.808 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:33:01.805+0000 7f18a77fe640 10 monclient: _finish_auth 0 2024-03-25T18:33:01.808 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:33:01.805+0000 7f18a77fe640 10 monclient: _check_auth_tickets 2024-03-25T18:33:01.808 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:33:01.805+0000 7f18a77fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:33:01.808 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:33:01.805+0000 7f18c52d6640 5 monclient: authenticate success, global_id 4507 2024-03-25T18:33:01.808 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:33:01.805+0000 7f18c52d6640 10 monclient: _renew_subs 2024-03-25T18:33:01.808 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:33:01.805+0000 7f18c52d6640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.115:6789/0 2024-03-25T18:33:01.808 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:33:01.805+0000 7f18c52d6640 1 -- 172.21.15.1:0/3621274813 --> v1:172.21.15.115:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f18c00b0290 con 0x7f18c013c480 2024-03-25T18:33:01.809 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:33:01.805+0000 7f18c52d6640 10 client.4507.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T18:33:01.809 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:33:01.805+0000 7f18c52d6640 10 monclient: _renew_subs 2024-03-25T18:33:01.809 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:33:01.805+0000 7f18c52d6640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.115:6789/0 2024-03-25T18:33:01.809 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:33:01.805+0000 7f18c52d6640 1 -- 172.21.15.1:0/3621274813 --> v1:172.21.15.115:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f18c00aefd0 con 0x7f18c013c480 2024-03-25T18:33:01.809 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:33:01.805+0000 7f18c52d6640 1 librados: init done 2024-03-25T18:33:01.809 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:33:01.806+0000 7f18c52d6640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T18:33:01.810 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:33:01.806+0000 7f18c52d6640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:33:01.810 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:33:01.806+0000 7f18c52d6640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.115:6789/0 2024-03-25T18:33:01.810 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:33:01.806+0000 7f18c52d6640 1 -- 172.21.15.1:0/3621274813 --> v1:172.21.15.115:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f18c006dbd0 con 0x7f18c013c480 2024-03-25T18:33:02.080 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:33:02.078+0000 7f18a77fe640 1 -- 172.21.15.1:0/3621274813 <== mon.1 v1:172.21.15.115:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f18b8003990 con 0x7f18c013c480 2024-03-25T18:33:02.080 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:33:02.078+0000 7f18a77fe640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:33:02.080 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:33:02.078+0000 7f18a77fe640 1 -- 172.21.15.1:0/3621274813 <== mon.1 v1:172.21.15.115:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1586965025 0 0) 0x7f18b8003da0 con 0x7f18c013c480 2024-03-25T18:33:02.080 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:33:02.078+0000 7f18a77fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:33:02.080 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:33:02.078+0000 7f18a77fe640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-25T18:33:02.080 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:33:02.078+0000 7f18a77fe640 10 monclient: dump: 2024-03-25T18:33:02.081 INFO:teuthology.misc.health.smithi001.stderr:epoch 1 2024-03-25T18:33:02.081 INFO:teuthology.misc.health.smithi001.stderr:fsid c8ea72cf-9e99-4383-9b52-6c23d365fdfc 2024-03-25T18:33:02.081 INFO:teuthology.misc.health.smithi001.stderr:last_changed 2024-03-25T18:25:50.510519+0000 2024-03-25T18:33:02.081 INFO:teuthology.misc.health.smithi001.stderr:created 2024-03-25T18:25:50.510519+0000 2024-03-25T18:33:02.081 INFO:teuthology.misc.health.smithi001.stderr:min_mon_release 19 (squid) 2024-03-25T18:33:02.081 INFO:teuthology.misc.health.smithi001.stderr:election_strategy: 3 2024-03-25T18:33:02.081 INFO:teuthology.misc.health.smithi001.stderr:0: v1:172.21.15.1:6789/0 mon.a 2024-03-25T18:33:02.081 INFO:teuthology.misc.health.smithi001.stderr:1: v1:172.21.15.115:6789/0 mon.b 2024-03-25T18:33:02.081 INFO:teuthology.misc.health.smithi001.stderr:2: v1:172.21.15.1:6790/0 mon.c 2024-03-25T18:33:02.081 INFO:teuthology.misc.health.smithi001.stderr: 2024-03-25T18:33:02.081 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:33:02.078+0000 7f18bd7fa640 4 set_mon_vals no callback set 2024-03-25T18:33:02.082 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:33:02.079+0000 7f18a77fe640 1 -- 172.21.15.1:0/3621274813 <== mon.1 v1:172.21.15.115:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 21787975 0 0) 0x7f18b8003f00 con 0x7f18c013c480 2024-03-25T18:33:02.082 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:33:02.080+0000 7f18beffd640 10 monclient: get_auth_request con 0x7f18ac06dcf0 auth_method 0 2024-03-25T18:33:02.082 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:33:02.080+0000 7f18a77fe640 10 client.4507.objecter ms_handle_connect 0x7f18ac06dcf0 2024-03-25T18:33:02.083 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:33:02.081+0000 7f18a77fe640 1 -- 172.21.15.1:0/3621274813 <== mon.1 v1:172.21.15.115:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 996254230 0 0) 0x7f18b8054ae0 con 0x7f18c013c480 2024-03-25T18:33:02.083 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:33:02.081+0000 7f18a77fe640 10 client.4507.objecter ms_dispatch 0x7f18c0000b90 osd_map(10..10 src has 1..10) v4 2024-03-25T18:33:02.083 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:33:02.081+0000 7f18a77fe640 3 client.4507.objecter handle_osd_map got epochs [10,10] > 0 2024-03-25T18:33:02.084 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:33:02.081+0000 7f18a77fe640 3 client.4507.objecter handle_osd_map decoding full epoch 10 2024-03-25T18:33:02.084 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:33:02.081+0000 7f18a77fe640 20 client.4507.objecter dump_active .. 0 homeless 2024-03-25T18:33:02.085 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:33:02.083+0000 7f18a77fe640 1 -- 172.21.15.1:0/3621274813 <== mon.1 v1:172.21.15.115:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f18b80505f0 con 0x7f18c013c480 2024-03-25T18:33:02.085 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:33:02.083+0000 7f18a77fe640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:33:02.085 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:33:02.083+0000 7f18a77fe640 10 monclient: _finish_command 1 = system:0 2024-03-25T18:33:02.178 INFO:tasks.ceph.mon.a.smithi001.stderr:==00:00:06:40.736 34876== Warning: unimplemented fcntl command: 1036 2024-03-25T18:33:02.185 INFO:tasks.ceph.mon.a.smithi001.stderr:==00:00:06:40.743 34876== Warning: unimplemented fcntl command: 1036 2024-03-25T18:33:02.218 INFO:tasks.ceph.mon.a.smithi001.stderr:==00:00:06:40.776 34876== Warning: unimplemented fcntl command: 1036 2024-03-25T18:33:02.230 INFO:tasks.ceph.mon.b.smithi115.stderr:==00:00:06:40.713 34836== Warning: unimplemented fcntl command: 1036 2024-03-25T18:33:02.239 INFO:tasks.ceph.mon.b.smithi115.stderr:==00:00:06:40.723 34836== Warning: unimplemented fcntl command: 1036 2024-03-25T18:33:02.262 INFO:tasks.ceph.mon.b.smithi115.stderr:==00:00:06:40.745 34836== Warning: unimplemented fcntl command: 1036 2024-03-25T18:33:02.297 INFO:tasks.ceph.mon.c.smithi001.stderr:==00:00:06:40.855 34877== Warning: unimplemented fcntl command: 1036 2024-03-25T18:33:02.303 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:33:02.301+0000 7f18c52d6640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:33:02.304 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:33:02.301+0000 7f18c52d6640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:33:02.304 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:33:02.301+0000 7f18c52d6640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.115:6789/0 2024-03-25T18:33:02.304 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:33:02.301+0000 7f18c52d6640 1 -- 172.21.15.1:0/3621274813 --> v1:172.21.15.115:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f18c00688c0 con 0x7f18c013c480 2024-03-25T18:33:02.317 INFO:tasks.ceph.mon.c.smithi001.stderr:==00:00:06:40.875 34877== Warning: unimplemented fcntl command: 1036 2024-03-25T18:33:02.340 INFO:tasks.ceph.mon.c.smithi001.stderr:==00:00:06:40.898 34877== Warning: unimplemented fcntl command: 1036 2024-03-25T18:33:02.370 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:33:02.368+0000 7f18a77fe640 1 -- 172.21.15.1:0/3621274813 <== mon.1 v1:172.21.15.115:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 4092213348) 0x7f18b8052520 con 0x7f18c013c480 2024-03-25T18:33:02.370 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:33:02.368+0000 7f18a77fe640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:33:02.370 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:33:02.368+0000 7f18a77fe640 10 monclient: _finish_command 2 = system:0 2024-03-25T18:33:02.370 INFO:teuthology.misc.health.smithi001.stdout: 2024-03-25T18:33:02.370 INFO:teuthology.misc.health.smithi001.stdout:{"epoch":10,"fsid":"c8ea72cf-9e99-4383-9b52-6c23d365fdfc","created":"2024-03-25T18:26:38.856638+0000","modified":"2024-03-25T18:26:44.224208+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T18:26:44.224208+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"b9f34922-3a84-49be-8426-807c1eceaefc","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"42bf306f-55e3-4634-9874-8b4b73ac6644","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"292cb23a-18a9-4207-a7cc-4d5727feaec0","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"45e0e9ab-898d-4673-9590-b4117bc5e84a","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"831a5a09-73ab-4a61-9872-088e76bdbc66","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"88cd903d-3b55-4840-9396-6e3cfe005282","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"89db7355-bc9d-46f4-8ab0-e8d9e744e026","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"cac47bee-0914-4f08-896f-348838882866","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T18:33:02.373 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:33:02.370+0000 7f18c52d6640 10 librados: watch_flush enter 2024-03-25T18:33:02.373 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:33:02.370+0000 7f18c52d6640 10 librados: watch_flush exit 2024-03-25T18:33:02.373 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:33:02.370+0000 7f18c52d6640 20 client.4507.objecter shutdown clearing up homeless session... 2024-03-25T18:33:02.373 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:33:02.370+0000 7f18c52d6640 10 client.4507.objecter successfully canceled tick 2024-03-25T18:33:02.373 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:33:02.370+0000 7f18c52d6640 1 -- 172.21.15.1:0/3621274813 >> v1:172.21.15.115:6800/4237379176 conn(0x7f18ac06dcf0 legacy=0x7f18ac0701b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:33:02.373 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:33:02.370+0000 7f18c52d6640 10 monclient: shutdown 2024-03-25T18:33:02.373 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:33:02.370+0000 7f18c52d6640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:33:02.373 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:33:02.370+0000 7f18c52d6640 1 -- 172.21.15.1:0/3621274813 >> v1:172.21.15.115:6789/0 conn(0x7f18c013c480 legacy=0x7f18c00b10a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:33:02.373 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:33:02.370+0000 7f18c52d6640 1 -- 172.21.15.1:0/3621274813 shutdown_connections 2024-03-25T18:33:02.373 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:33:02.370+0000 7f18c52d6640 1 -- 172.21.15.1:0/3621274813 >> 172.21.15.1:0/3621274813 conn(0x7f18c00617a0 msgr2=0x7f18c00b71c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:33:02.373 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:33:02.370+0000 7f18c52d6640 1 -- 172.21.15.1:0/3621274813 shutdown_connections 2024-03-25T18:33:02.373 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:33:02.370+0000 7f18c52d6640 1 -- 172.21.15.1:0/3621274813 wait complete. 2024-03-25T18:33:02.374 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:33:02.370+0000 7f18c52d6640 1 librados: shutdown 2024-03-25T18:33:02.384 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T18:33:09.191 DEBUG:teuthology.orchestra.run.smithi001:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T18:33:09.309 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:33:09.306+0000 7f31143e8640 10 monclient: get_monmap_and_config 2024-03-25T18:33:09.309 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:33:09.306+0000 7f31143e8640 10 monclient: build_initial_monmap 2024-03-25T18:33:09.309 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:33:09.306+0000 7f31143e8640 10 monclient: monmap: 2024-03-25T18:33:09.309 INFO:teuthology.misc.health.smithi001.stderr:epoch 0 2024-03-25T18:33:09.309 INFO:teuthology.misc.health.smithi001.stderr:fsid c8ea72cf-9e99-4383-9b52-6c23d365fdfc 2024-03-25T18:33:09.309 INFO:teuthology.misc.health.smithi001.stderr:last_changed 2024-03-25T18:33:09.308465+0000 2024-03-25T18:33:09.309 INFO:teuthology.misc.health.smithi001.stderr:created 2024-03-25T18:33:09.308465+0000 2024-03-25T18:33:09.309 INFO:teuthology.misc.health.smithi001.stderr:min_mon_release 0 (unknown) 2024-03-25T18:33:09.309 INFO:teuthology.misc.health.smithi001.stderr:election_strategy: 3 2024-03-25T18:33:09.309 INFO:teuthology.misc.health.smithi001.stderr:0: v1:172.21.15.1:6789/0 mon.noname-a 2024-03-25T18:33:09.309 INFO:teuthology.misc.health.smithi001.stderr:1: v1:172.21.15.115:6789/0 mon.noname-c 2024-03-25T18:33:09.309 INFO:teuthology.misc.health.smithi001.stderr:2: v1:172.21.15.1:6790/0 mon.noname-b 2024-03-25T18:33:09.310 INFO:teuthology.misc.health.smithi001.stderr: 2024-03-25T18:33:09.311 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:33:09.307+0000 7f31143e8640 1 Processor -- start 2024-03-25T18:33:09.311 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:33:09.307+0000 7f31143e8640 1 -- start start 2024-03-25T18:33:09.311 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:33:09.307+0000 7f31143e8640 10 monclient: init 2024-03-25T18:33:09.311 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:33:09.308+0000 7f31143e8640 10 monclient: _reopen_session rank -1 2024-03-25T18:33:09.311 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:33:09.308+0000 7f31143e8640 10 monclient: _add_conns ranks=[2,1,0] 2024-03-25T18:33:09.311 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:33:09.308+0000 7f31143e8640 10 monclient(hunting): picked mon.noname-b con 0x7f310c0b4ff0 addr v1:172.21.15.1:6790/0 2024-03-25T18:33:09.311 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:33:09.308+0000 7f31143e8640 10 monclient(hunting): picked mon.noname-c con 0x7f310c0b4670 addr v1:172.21.15.115:6789/0 2024-03-25T18:33:09.311 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:33:09.308+0000 7f31143e8640 10 monclient(hunting): picked mon.noname-a con 0x7f310c13c480 addr v1:172.21.15.1:6789/0 2024-03-25T18:33:09.311 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:33:09.308+0000 7f31143e8640 1 -- --> v1:172.21.15.1:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f310c0ab0e0 con 0x7f310c13c480 2024-03-25T18:33:09.311 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:33:09.308+0000 7f31143e8640 1 -- --> v1:172.21.15.115:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f310c0ab3c0 con 0x7f310c0b4670 2024-03-25T18:33:09.311 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:33:09.308+0000 7f31143e8640 1 -- --> v1:172.21.15.1:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f310c0a9f30 con 0x7f310c0b4ff0 2024-03-25T18:33:09.311 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:33:09.308+0000 7f31143e8640 10 monclient(hunting): _renew_subs 2024-03-25T18:33:09.312 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:33:09.308+0000 7f31143e8640 10 monclient(hunting): authenticate will time out at 7869.779611s 2024-03-25T18:33:09.320 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:33:09.318+0000 7f3112184640 1 --1- >> v1:172.21.15.1:6790/0 conn(0x7f310c0b4ff0 0x7f310c13bd70 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.1:6790/0 says I am v1:172.21.15.1:53638/0 (socket says 172.21.15.1:53638) 2024-03-25T18:33:09.320 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:33:09.318+0000 7f3112184640 1 -- 172.21.15.1:0/9660228 learned_addr learned my addr 172.21.15.1:0/9660228 (peer_addr_for_me v1:172.21.15.1:0/0) 2024-03-25T18:33:09.325 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:33:09.323+0000 7f3111182640 1 -- 172.21.15.1:0/9660228 <== mon.1 v1:172.21.15.115:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 382590870 0 0) 0x7f310c0ab3c0 con 0x7f310c0b4670 2024-03-25T18:33:09.325 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:33:09.323+0000 7f3111182640 10 monclient(hunting): _negotiate 2024-03-25T18:33:09.325 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:33:09.323+0000 7f3111182640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:33:09.325 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:33:09.323+0000 7f3111182640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:33:09.325 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:33:09.323+0000 7f3111182640 10 monclient(hunting): my global_id is 4510 2024-03-25T18:33:09.325 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:33:09.323+0000 7f3111182640 1 -- 172.21.15.1:0/9660228 --> v1:172.21.15.115:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f30f8003520 con 0x7f310c0b4670 2024-03-25T18:33:09.330 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:33:09.328+0000 7f3111182640 1 -- 172.21.15.1:0/9660228 <== mon.0 v1:172.21.15.1:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3013540644 0 0) 0x7f310c0ab0e0 con 0x7f310c13c480 2024-03-25T18:33:09.330 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:33:09.328+0000 7f3111182640 10 monclient(hunting): _negotiate 2024-03-25T18:33:09.330 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:33:09.328+0000 7f3111182640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:33:09.330 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:33:09.328+0000 7f3111182640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:33:09.330 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:33:09.328+0000 7f3111182640 10 monclient(hunting): my global_id is 4497 2024-03-25T18:33:09.330 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:33:09.328+0000 7f3111182640 1 -- 172.21.15.1:0/9660228 --> v1:172.21.15.1:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f310c0ab3c0 con 0x7f310c13c480 2024-03-25T18:33:09.332 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:33:09.329+0000 7f3111182640 1 -- 172.21.15.1:0/9660228 <== mon.1 v1:172.21.15.115:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1509683317 0 0) 0x7f30f8003520 con 0x7f310c0b4670 2024-03-25T18:33:09.332 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:33:09.330+0000 7f3111182640 1 -- 172.21.15.1:0/9660228 >> v1:172.21.15.1:6790/0 conn(0x7f310c0b4ff0 legacy=0x7f310c13bd70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:33:09.332 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:33:09.330+0000 7f3111182640 1 -- 172.21.15.1:0/9660228 >> v1:172.21.15.1:6789/0 conn(0x7f310c13c480 legacy=0x7f310c14dc00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:33:09.332 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:33:09.330+0000 7f3111182640 10 monclient: _finish_hunting 0 2024-03-25T18:33:09.332 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:33:09.330+0000 7f3111182640 1 monclient: found mon.noname-c 2024-03-25T18:33:09.332 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:33:09.330+0000 7f3111182640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:33:09.333 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:33:09.330+0000 7f3111182640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.115:6789/0 2024-03-25T18:33:09.333 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:33:09.330+0000 7f3111182640 1 -- 172.21.15.1:0/9660228 --> v1:172.21.15.115:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f310c0a9b70 con 0x7f310c0b4670 2024-03-25T18:33:09.333 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:33:09.330+0000 7f3111182640 10 monclient: _finish_auth 0 2024-03-25T18:33:09.333 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:33:09.330+0000 7f3111182640 10 monclient: _check_auth_tickets 2024-03-25T18:33:09.333 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:33:09.330+0000 7f3111182640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:33:09.333 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:33:09.330+0000 7f3111182640 1 -- 172.21.15.1:0/9660228 <== mon.1 v1:172.21.15.115:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1586965025 0 0) 0x7f30fc001690 con 0x7f310c0b4670 2024-03-25T18:33:09.333 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:33:09.330+0000 7f3111182640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:33:09.333 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:33:09.330+0000 7f3111182640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-03-25T18:33:09.333 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:33:09.330+0000 7f3111182640 10 monclient: dump: 2024-03-25T18:33:09.333 INFO:teuthology.misc.health.smithi001.stderr:epoch 1 2024-03-25T18:33:09.333 INFO:teuthology.misc.health.smithi001.stderr:fsid c8ea72cf-9e99-4383-9b52-6c23d365fdfc 2024-03-25T18:33:09.333 INFO:teuthology.misc.health.smithi001.stderr:last_changed 2024-03-25T18:25:50.510519+0000 2024-03-25T18:33:09.334 INFO:teuthology.misc.health.smithi001.stderr:created 2024-03-25T18:25:50.510519+0000 2024-03-25T18:33:09.334 INFO:teuthology.misc.health.smithi001.stderr:min_mon_release 19 (squid) 2024-03-25T18:33:09.334 INFO:teuthology.misc.health.smithi001.stderr:election_strategy: 3 2024-03-25T18:33:09.334 INFO:teuthology.misc.health.smithi001.stderr:0: v1:172.21.15.1:6789/0 mon.a 2024-03-25T18:33:09.334 INFO:teuthology.misc.health.smithi001.stderr:1: v1:172.21.15.115:6789/0 mon.b 2024-03-25T18:33:09.334 INFO:teuthology.misc.health.smithi001.stderr:2: v1:172.21.15.1:6790/0 mon.c 2024-03-25T18:33:09.334 INFO:teuthology.misc.health.smithi001.stderr: 2024-03-25T18:33:09.334 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:33:09.330+0000 7f31143e8640 5 monclient: authenticate success, global_id 4510 2024-03-25T18:33:09.334 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:33:09.330+0000 7f31143e8640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T18:33:09.339 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:33:09.337+0000 7f3111182640 1 -- 172.21.15.1:0/9660228 <== mon.1 v1:172.21.15.115:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f30fc0017f0 con 0x7f310c0b4670 2024-03-25T18:33:09.339 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:33:09.337+0000 7f3111182640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:33:09.339 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:33:09.337+0000 7f3111182640 1 -- 172.21.15.1:0/9660228 <== mon.1 v1:172.21.15.115:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1586965025 0 0) 0x7f30fc001c00 con 0x7f310c0b4670 2024-03-25T18:33:09.339 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:33:09.337+0000 7f31143e8640 10 monclient: get_monmap_and_config success 2024-03-25T18:33:09.339 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:33:09.337+0000 7f31143e8640 4 set_mon_vals no callback set 2024-03-25T18:33:09.339 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:33:09.337+0000 7f3111182640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:33:09.339 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:33:09.337+0000 7f3111182640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-25T18:33:09.339 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:33:09.337+0000 7f3111182640 10 monclient: dump: 2024-03-25T18:33:09.339 INFO:teuthology.misc.health.smithi001.stderr:epoch 1 2024-03-25T18:33:09.339 INFO:teuthology.misc.health.smithi001.stderr:fsid c8ea72cf-9e99-4383-9b52-6c23d365fdfc 2024-03-25T18:33:09.339 INFO:teuthology.misc.health.smithi001.stderr:last_changed 2024-03-25T18:25:50.510519+0000 2024-03-25T18:33:09.340 INFO:teuthology.misc.health.smithi001.stderr:created 2024-03-25T18:25:50.510519+0000 2024-03-25T18:33:09.340 INFO:teuthology.misc.health.smithi001.stderr:min_mon_release 19 (squid) 2024-03-25T18:33:09.340 INFO:teuthology.misc.health.smithi001.stderr:election_strategy: 3 2024-03-25T18:33:09.340 INFO:teuthology.misc.health.smithi001.stderr:0: v1:172.21.15.1:6789/0 mon.a 2024-03-25T18:33:09.340 INFO:teuthology.misc.health.smithi001.stderr:1: v1:172.21.15.115:6789/0 mon.b 2024-03-25T18:33:09.340 INFO:teuthology.misc.health.smithi001.stderr:2: v1:172.21.15.1:6790/0 mon.c 2024-03-25T18:33:09.340 INFO:teuthology.misc.health.smithi001.stderr: 2024-03-25T18:33:09.342 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:33:09.337+0000 7f31143e8640 10 monclient: shutdown 2024-03-25T18:33:09.342 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:33:09.337+0000 7f31143e8640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:33:09.342 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:33:09.337+0000 7f31143e8640 1 -- 172.21.15.1:0/9660228 >> v1:172.21.15.115:6789/0 conn(0x7f310c0b4670 legacy=0x7f310c1425c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:33:09.342 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:33:09.338+0000 7f31143e8640 1 -- 172.21.15.1:0/9660228 shutdown_connections 2024-03-25T18:33:09.343 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:33:09.338+0000 7f31143e8640 1 -- 172.21.15.1:0/9660228 >> 172.21.15.1:0/9660228 conn(0x7f310c0617a0 msgr2=0x7f310c061bb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:33:09.343 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:33:09.338+0000 7f31143e8640 1 -- 172.21.15.1:0/9660228 shutdown_connections 2024-03-25T18:33:09.343 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:33:09.338+0000 7f31143e8640 1 -- 172.21.15.1:0/9660228 wait complete. 2024-03-25T18:33:09.343 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:33:09.338+0000 7f31143e8640 10 monclient: build_initial_monmap 2024-03-25T18:33:09.343 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:33:09.338+0000 7f31143e8640 10 monclient: monmap: 2024-03-25T18:33:09.343 INFO:teuthology.misc.health.smithi001.stderr:epoch 0 2024-03-25T18:33:09.343 INFO:teuthology.misc.health.smithi001.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T18:33:09.343 INFO:teuthology.misc.health.smithi001.stderr:last_changed 0.000000 2024-03-25T18:33:09.343 INFO:teuthology.misc.health.smithi001.stderr:created 0.000000 2024-03-25T18:33:09.343 INFO:teuthology.misc.health.smithi001.stderr:min_mon_release 0 (unknown) 2024-03-25T18:33:09.343 INFO:teuthology.misc.health.smithi001.stderr:election_strategy: 1 2024-03-25T18:33:09.343 INFO:teuthology.misc.health.smithi001.stderr:0: v1:172.21.15.1:6789/0 mon.noname-a 2024-03-25T18:33:09.343 INFO:teuthology.misc.health.smithi001.stderr:1: v1:172.21.15.115:6789/0 mon.noname-b 2024-03-25T18:33:09.343 INFO:teuthology.misc.health.smithi001.stderr:2: v1:172.21.15.1:6790/0 mon.noname-c 2024-03-25T18:33:09.344 INFO:teuthology.misc.health.smithi001.stderr: 2024-03-25T18:33:09.344 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:33:09.339+0000 7f31143e8640 1 librados: starting msgr at 2024-03-25T18:33:09.344 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:33:09.339+0000 7f31143e8640 1 librados: starting objecter 2024-03-25T18:33:09.344 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:33:09.339+0000 7f31143e8640 1 Processor -- start 2024-03-25T18:33:09.344 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:33:09.339+0000 7f31143e8640 1 -- start start 2024-03-25T18:33:09.344 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:33:09.339+0000 7f31143e8640 1 librados: setting wanted keys 2024-03-25T18:33:09.344 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:33:09.339+0000 7f31143e8640 1 librados: calling monclient init 2024-03-25T18:33:09.344 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:33:09.339+0000 7f31143e8640 10 monclient: init 2024-03-25T18:33:09.344 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:33:09.339+0000 7f31143e8640 10 monclient: _reopen_session rank -1 2024-03-25T18:33:09.344 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:33:09.339+0000 7f31143e8640 10 monclient: _add_conns ranks=[1,2,0] 2024-03-25T18:33:09.344 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:33:09.339+0000 7f31143e8640 10 monclient(hunting): picked mon.noname-b con 0x7f310c0b4670 addr v1:172.21.15.115:6789/0 2024-03-25T18:33:09.344 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:33:09.339+0000 7f31143e8640 10 monclient(hunting): picked mon.noname-c con 0x7f310c0b4ff0 addr v1:172.21.15.1:6790/0 2024-03-25T18:33:09.344 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:33:09.339+0000 7f31143e8640 10 monclient(hunting): picked mon.noname-a con 0x7f310c13c480 addr v1:172.21.15.1:6789/0 2024-03-25T18:33:09.344 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:33:09.339+0000 7f31143e8640 1 -- --> v1:172.21.15.1:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f310c0af4f0 con 0x7f310c13c480 2024-03-25T18:33:09.345 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:33:09.340+0000 7f31143e8640 1 -- --> v1:172.21.15.115:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f310c0ada40 con 0x7f310c0b4670 2024-03-25T18:33:09.345 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:33:09.340+0000 7f31143e8640 1 -- --> v1:172.21.15.1:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f310c0afcd0 con 0x7f310c0b4ff0 2024-03-25T18:33:09.345 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:33:09.340+0000 7f31143e8640 10 monclient(hunting): _renew_subs 2024-03-25T18:33:09.345 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:33:09.340+0000 7f31143e8640 10 monclient(hunting): authenticate will time out at 7869.810728s 2024-03-25T18:33:09.355 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:33:09.353+0000 7f3112184640 1 --1- >> v1:172.21.15.115:6789/0 conn(0x7f310c0b4670 0x7f310c13a0f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.115:6789/0 says I am v1:172.21.15.1:48970/0 (socket says 172.21.15.1:48970) 2024-03-25T18:33:09.355 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:33:09.353+0000 7f3112184640 1 -- 172.21.15.1:0/688258845 learned_addr learned my addr 172.21.15.1:0/688258845 (peer_addr_for_me v1:172.21.15.1:0/0) 2024-03-25T18:33:09.355 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:33:09.353+0000 7f3102ffd640 10 client.?.objecter ms_handle_connect 0x7f310c0b4670 2024-03-25T18:33:09.355 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:33:09.353+0000 7f3102ffd640 10 client.?.objecter resend_mon_ops 2024-03-25T18:33:09.361 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:33:09.359+0000 7f3102ffd640 10 client.?.objecter ms_handle_connect 0x7f310c0b4ff0 2024-03-25T18:33:09.361 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:33:09.359+0000 7f3102ffd640 10 client.?.objecter resend_mon_ops 2024-03-25T18:33:09.364 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:33:09.362+0000 7f3102ffd640 1 -- 172.21.15.1:0/688258845 <== mon.1 v1:172.21.15.115:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3019779403 0 0) 0x7f310c0ada40 con 0x7f310c0b4670 2024-03-25T18:33:09.364 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:33:09.362+0000 7f3102ffd640 10 monclient(hunting): _negotiate 2024-03-25T18:33:09.364 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:33:09.362+0000 7f3102ffd640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:33:09.364 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:33:09.362+0000 7f3102ffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:33:09.364 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:33:09.362+0000 7f3102ffd640 10 monclient(hunting): my global_id is 4513 2024-03-25T18:33:09.365 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:33:09.362+0000 7f3102ffd640 1 -- 172.21.15.1:0/688258845 --> v1:172.21.15.115:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f30f0004eb0 con 0x7f310c0b4670 2024-03-25T18:33:09.371 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:33:09.369+0000 7f3102ffd640 1 -- 172.21.15.1:0/688258845 <== mon.2 v1:172.21.15.1:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 243977971 0 0) 0x7f310c0afcd0 con 0x7f310c0b4ff0 2024-03-25T18:33:09.371 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:33:09.369+0000 7f3102ffd640 10 monclient(hunting): _negotiate 2024-03-25T18:33:09.371 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:33:09.369+0000 7f3102ffd640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:33:09.371 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:33:09.369+0000 7f3102ffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:33:09.371 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:33:09.369+0000 7f3102ffd640 10 monclient(hunting): my global_id is 4514 2024-03-25T18:33:09.371 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:33:09.369+0000 7f3102ffd640 1 -- 172.21.15.1:0/688258845 --> v1:172.21.15.1:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f310c0ada40 con 0x7f310c0b4ff0 2024-03-25T18:33:09.373 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:33:09.371+0000 7f3102ffd640 10 client.?.objecter ms_handle_connect 0x7f310c13c480 2024-03-25T18:33:09.373 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:33:09.372+0000 7f3102ffd640 10 client.?.objecter resend_mon_ops 2024-03-25T18:33:09.382 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:33:09.380+0000 7f3102ffd640 1 -- 172.21.15.1:0/688258845 <== mon.1 v1:172.21.15.115:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2015492416 0 0) 0x7f30f0004eb0 con 0x7f310c0b4670 2024-03-25T18:33:09.383 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:33:09.380+0000 7f3102ffd640 1 -- 172.21.15.1:0/688258845 --> v1:172.21.15.115:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f310c0afcd0 con 0x7f310c0b4670 2024-03-25T18:33:09.383 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:33:09.380+0000 7f3102ffd640 1 -- 172.21.15.1:0/688258845 <== mon.1 v1:172.21.15.115:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1586965025 0 0) 0x7f31080018b0 con 0x7f310c0b4670 2024-03-25T18:33:09.383 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:33:09.380+0000 7f3102ffd640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:33:09.383 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:33:09.380+0000 7f3102ffd640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-03-25T18:33:09.383 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:33:09.380+0000 7f3102ffd640 10 monclient(hunting): dump: 2024-03-25T18:33:09.383 INFO:teuthology.misc.health.smithi001.stderr:epoch 1 2024-03-25T18:33:09.383 INFO:teuthology.misc.health.smithi001.stderr:fsid c8ea72cf-9e99-4383-9b52-6c23d365fdfc 2024-03-25T18:33:09.383 INFO:teuthology.misc.health.smithi001.stderr:last_changed 2024-03-25T18:25:50.510519+0000 2024-03-25T18:33:09.383 INFO:teuthology.misc.health.smithi001.stderr:created 2024-03-25T18:25:50.510519+0000 2024-03-25T18:33:09.383 INFO:teuthology.misc.health.smithi001.stderr:min_mon_release 19 (squid) 2024-03-25T18:33:09.383 INFO:teuthology.misc.health.smithi001.stderr:election_strategy: 3 2024-03-25T18:33:09.383 INFO:teuthology.misc.health.smithi001.stderr:0: v1:172.21.15.1:6789/0 mon.a 2024-03-25T18:33:09.383 INFO:teuthology.misc.health.smithi001.stderr:1: v1:172.21.15.115:6789/0 mon.b 2024-03-25T18:33:09.383 INFO:teuthology.misc.health.smithi001.stderr:2: v1:172.21.15.1:6790/0 mon.c 2024-03-25T18:33:09.384 INFO:teuthology.misc.health.smithi001.stderr: 2024-03-25T18:33:09.384 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:33:09.380+0000 7f3102ffd640 10 monclient(hunting): _finish_auth 0 2024-03-25T18:33:09.385 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:33:09.383+0000 7f3102ffd640 1 -- 172.21.15.1:0/688258845 <== mon.0 v1:172.21.15.1:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 736280575 0 0) 0x7f310c0af4f0 con 0x7f310c13c480 2024-03-25T18:33:09.385 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:33:09.383+0000 7f3102ffd640 10 monclient(hunting): _negotiate 2024-03-25T18:33:09.385 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:33:09.383+0000 7f3102ffd640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:33:09.385 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:33:09.383+0000 7f3102ffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:33:09.385 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:33:09.383+0000 7f3102ffd640 10 monclient(hunting): my global_id is 4500 2024-03-25T18:33:09.385 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:33:09.383+0000 7f3102ffd640 1 -- 172.21.15.1:0/688258845 --> v1:172.21.15.1:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f30f0004eb0 con 0x7f310c13c480 2024-03-25T18:33:09.392 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:33:09.389+0000 7f3102ffd640 1 -- 172.21.15.1:0/688258845 <== mon.2 v1:172.21.15.1:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1091633697 0 0) 0x7f310c0ada40 con 0x7f310c0b4ff0 2024-03-25T18:33:09.392 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:33:09.389+0000 7f3102ffd640 1 -- 172.21.15.1:0/688258845 --> v1:172.21.15.1:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f310c0af4f0 con 0x7f310c0b4ff0 2024-03-25T18:33:09.392 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:33:09.389+0000 7f3102ffd640 1 -- 172.21.15.1:0/688258845 <== mon.2 v1:172.21.15.1:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1586965025 0 0) 0x7f30fc001bc0 con 0x7f310c0b4ff0 2024-03-25T18:33:09.392 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:33:09.389+0000 7f3102ffd640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:33:09.392 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:33:09.389+0000 7f3102ffd640 10 monclient(hunting): got monmap 1 from mon.c (according to old e1) 2024-03-25T18:33:09.392 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:33:09.389+0000 7f3102ffd640 10 monclient(hunting): dump: 2024-03-25T18:33:09.392 INFO:teuthology.misc.health.smithi001.stderr:epoch 1 2024-03-25T18:33:09.392 INFO:teuthology.misc.health.smithi001.stderr:fsid c8ea72cf-9e99-4383-9b52-6c23d365fdfc 2024-03-25T18:33:09.392 INFO:teuthology.misc.health.smithi001.stderr:last_changed 2024-03-25T18:25:50.510519+0000 2024-03-25T18:33:09.392 INFO:teuthology.misc.health.smithi001.stderr:created 2024-03-25T18:25:50.510519+0000 2024-03-25T18:33:09.392 INFO:teuthology.misc.health.smithi001.stderr:min_mon_release 19 (squid) 2024-03-25T18:33:09.393 INFO:teuthology.misc.health.smithi001.stderr:election_strategy: 3 2024-03-25T18:33:09.393 INFO:teuthology.misc.health.smithi001.stderr:0: v1:172.21.15.1:6789/0 mon.a 2024-03-25T18:33:09.393 INFO:teuthology.misc.health.smithi001.stderr:1: v1:172.21.15.115:6789/0 mon.b 2024-03-25T18:33:09.393 INFO:teuthology.misc.health.smithi001.stderr:2: v1:172.21.15.1:6790/0 mon.c 2024-03-25T18:33:09.393 INFO:teuthology.misc.health.smithi001.stderr: 2024-03-25T18:33:09.393 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:33:09.389+0000 7f3102ffd640 1 -- 172.21.15.1:0/688258845 <== mon.1 v1:172.21.15.115:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 4145368587 0 0) 0x7f310c0afcd0 con 0x7f310c0b4670 2024-03-25T18:33:09.393 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:33:09.389+0000 7f3102ffd640 1 -- 172.21.15.1:0/688258845 >> v1:172.21.15.1:6790/0 conn(0x7f310c0b4ff0 legacy=0x7f310c0b0990 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:33:09.393 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:33:09.389+0000 7f3102ffd640 1 -- 172.21.15.1:0/688258845 >> v1:172.21.15.1:6789/0 conn(0x7f310c13c480 legacy=0x7f310c0b10a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:33:09.393 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:33:09.389+0000 7f3102ffd640 10 monclient: _finish_hunting 0 2024-03-25T18:33:09.393 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:33:09.389+0000 7f3102ffd640 1 monclient: found mon.b 2024-03-25T18:33:09.393 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:33:09.389+0000 7f3102ffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:33:09.393 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:33:09.389+0000 7f3102ffd640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.115:6789/0 2024-03-25T18:33:09.393 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:33:09.389+0000 7f3102ffd640 1 -- 172.21.15.1:0/688258845 --> v1:172.21.15.115:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f310c0affb0 con 0x7f310c0b4670 2024-03-25T18:33:09.393 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:33:09.389+0000 7f3102ffd640 10 monclient: _finish_auth 0 2024-03-25T18:33:09.394 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:33:09.389+0000 7f3102ffd640 10 monclient: _check_auth_tickets 2024-03-25T18:33:09.394 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:33:09.389+0000 7f3102ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:33:09.394 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:33:09.389+0000 7f31143e8640 5 monclient: authenticate success, global_id 4513 2024-03-25T18:33:09.394 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:33:09.389+0000 7f31143e8640 10 monclient: _renew_subs 2024-03-25T18:33:09.394 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:33:09.389+0000 7f31143e8640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.115:6789/0 2024-03-25T18:33:09.394 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:33:09.389+0000 7f31143e8640 1 -- 172.21.15.1:0/688258845 --> v1:172.21.15.115:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f310c0b0290 con 0x7f310c0b4670 2024-03-25T18:33:09.394 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:33:09.389+0000 7f31143e8640 10 client.4513.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T18:33:09.394 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:33:09.389+0000 7f31143e8640 10 monclient: _renew_subs 2024-03-25T18:33:09.394 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:33:09.389+0000 7f31143e8640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.115:6789/0 2024-03-25T18:33:09.394 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:33:09.389+0000 7f31143e8640 1 -- 172.21.15.1:0/688258845 --> v1:172.21.15.115:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f310c0aefd0 con 0x7f310c0b4670 2024-03-25T18:33:09.394 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:33:09.389+0000 7f31143e8640 1 librados: init done 2024-03-25T18:33:09.395 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:33:09.390+0000 7f31143e8640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T18:33:09.395 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:33:09.390+0000 7f31143e8640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:33:09.395 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:33:09.390+0000 7f31143e8640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.115:6789/0 2024-03-25T18:33:09.395 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:33:09.390+0000 7f31143e8640 1 -- 172.21.15.1:0/688258845 --> v1:172.21.15.115:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f310c06dbd0 con 0x7f310c0b4670 2024-03-25T18:33:09.650 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:33:09.648+0000 7f3102ffd640 1 -- 172.21.15.1:0/688258845 <== mon.1 v1:172.21.15.115:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f31080047e0 con 0x7f310c0b4670 2024-03-25T18:33:09.651 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:33:09.648+0000 7f3102ffd640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:33:09.651 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:33:09.648+0000 7f3110981640 4 set_mon_vals no callback set 2024-03-25T18:33:09.660 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:33:09.658+0000 7f3102ffd640 1 -- 172.21.15.1:0/688258845 <== mon.1 v1:172.21.15.115:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1586965025 0 0) 0x7f3108004c30 con 0x7f310c0b4670 2024-03-25T18:33:09.660 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:33:09.658+0000 7f3102ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:33:09.660 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:33:09.658+0000 7f3102ffd640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-25T18:33:09.660 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:33:09.658+0000 7f3102ffd640 10 monclient: dump: 2024-03-25T18:33:09.660 INFO:teuthology.misc.health.smithi001.stderr:epoch 1 2024-03-25T18:33:09.661 INFO:teuthology.misc.health.smithi001.stderr:fsid c8ea72cf-9e99-4383-9b52-6c23d365fdfc 2024-03-25T18:33:09.661 INFO:teuthology.misc.health.smithi001.stderr:last_changed 2024-03-25T18:25:50.510519+0000 2024-03-25T18:33:09.661 INFO:teuthology.misc.health.smithi001.stderr:created 2024-03-25T18:25:50.510519+0000 2024-03-25T18:33:09.661 INFO:teuthology.misc.health.smithi001.stderr:min_mon_release 19 (squid) 2024-03-25T18:33:09.661 INFO:teuthology.misc.health.smithi001.stderr:election_strategy: 3 2024-03-25T18:33:09.661 INFO:teuthology.misc.health.smithi001.stderr:0: v1:172.21.15.1:6789/0 mon.a 2024-03-25T18:33:09.661 INFO:teuthology.misc.health.smithi001.stderr:1: v1:172.21.15.115:6789/0 mon.b 2024-03-25T18:33:09.661 INFO:teuthology.misc.health.smithi001.stderr:2: v1:172.21.15.1:6790/0 mon.c 2024-03-25T18:33:09.661 INFO:teuthology.misc.health.smithi001.stderr: 2024-03-25T18:33:09.662 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:33:09.659+0000 7f3102ffd640 1 -- 172.21.15.1:0/688258845 <== mon.1 v1:172.21.15.115:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 21787975 0 0) 0x7f3108004e10 con 0x7f310c0b4670 2024-03-25T18:33:09.662 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:33:09.660+0000 7f3111983640 10 monclient: get_auth_request con 0x7f30f006df70 auth_method 0 2024-03-25T18:33:09.662 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:33:09.660+0000 7f3102ffd640 10 client.4513.objecter ms_handle_connect 0x7f30f006df70 2024-03-25T18:33:09.666 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:33:09.664+0000 7f3102ffd640 1 -- 172.21.15.1:0/688258845 <== mon.1 v1:172.21.15.115:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 996254230 0 0) 0x7f3108054890 con 0x7f310c0b4670 2024-03-25T18:33:09.666 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:33:09.664+0000 7f3102ffd640 10 client.4513.objecter ms_dispatch 0x7f310c000b90 osd_map(10..10 src has 1..10) v4 2024-03-25T18:33:09.666 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:33:09.664+0000 7f3102ffd640 3 client.4513.objecter handle_osd_map got epochs [10,10] > 0 2024-03-25T18:33:09.666 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:33:09.664+0000 7f3102ffd640 3 client.4513.objecter handle_osd_map decoding full epoch 10 2024-03-25T18:33:09.666 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:33:09.664+0000 7f3102ffd640 20 client.4513.objecter dump_active .. 0 homeless 2024-03-25T18:33:09.666 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:33:09.664+0000 7f3102ffd640 1 -- 172.21.15.1:0/688258845 <== mon.1 v1:172.21.15.115:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f31080503a0 con 0x7f310c0b4670 2024-03-25T18:33:09.667 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:33:09.664+0000 7f3102ffd640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:33:09.667 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:33:09.664+0000 7f3102ffd640 10 monclient: _finish_command 1 = system:0 2024-03-25T18:33:09.922 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:33:09.920+0000 7f31143e8640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:33:09.922 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:33:09.920+0000 7f31143e8640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:33:09.922 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:33:09.920+0000 7f31143e8640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.115:6789/0 2024-03-25T18:33:09.922 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:33:09.920+0000 7f31143e8640 1 -- 172.21.15.1:0/688258845 --> v1:172.21.15.115:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f310c0688c0 con 0x7f310c0b4670 2024-03-25T18:33:09.940 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:33:09.938+0000 7f3102ffd640 1 -- 172.21.15.1:0/688258845 <== mon.1 v1:172.21.15.115:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 4092213348) 0x7f31080522d0 con 0x7f310c0b4670 2024-03-25T18:33:09.940 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:33:09.938+0000 7f3102ffd640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:33:09.940 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:33:09.938+0000 7f3102ffd640 10 monclient: _finish_command 2 = system:0 2024-03-25T18:33:09.940 INFO:teuthology.misc.health.smithi001.stdout: 2024-03-25T18:33:09.940 INFO:teuthology.misc.health.smithi001.stdout:{"epoch":10,"fsid":"c8ea72cf-9e99-4383-9b52-6c23d365fdfc","created":"2024-03-25T18:26:38.856638+0000","modified":"2024-03-25T18:26:44.224208+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T18:26:44.224208+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"b9f34922-3a84-49be-8426-807c1eceaefc","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"42bf306f-55e3-4634-9874-8b4b73ac6644","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"292cb23a-18a9-4207-a7cc-4d5727feaec0","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"45e0e9ab-898d-4673-9590-b4117bc5e84a","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"831a5a09-73ab-4a61-9872-088e76bdbc66","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"88cd903d-3b55-4840-9396-6e3cfe005282","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"89db7355-bc9d-46f4-8ab0-e8d9e744e026","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"cac47bee-0914-4f08-896f-348838882866","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T18:33:09.943 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:33:09.940+0000 7f31143e8640 10 librados: watch_flush enter 2024-03-25T18:33:09.943 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:33:09.940+0000 7f31143e8640 10 librados: watch_flush exit 2024-03-25T18:33:09.943 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:33:09.940+0000 7f31143e8640 20 client.4513.objecter shutdown clearing up homeless session... 2024-03-25T18:33:09.944 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:33:09.940+0000 7f31143e8640 10 client.4513.objecter successfully canceled tick 2024-03-25T18:33:09.944 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:33:09.941+0000 7f31143e8640 1 -- 172.21.15.1:0/688258845 >> v1:172.21.15.115:6800/4237379176 conn(0x7f30f006df70 legacy=0x7f30f0070430 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:33:09.944 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:33:09.941+0000 7f31143e8640 10 monclient: shutdown 2024-03-25T18:33:09.944 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:33:09.941+0000 7f31143e8640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:33:09.944 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:33:09.941+0000 7f31143e8640 1 -- 172.21.15.1:0/688258845 >> v1:172.21.15.115:6789/0 conn(0x7f310c0b4670 legacy=0x7f310c13a0f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:33:09.944 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:33:09.941+0000 7f31143e8640 1 -- 172.21.15.1:0/688258845 shutdown_connections 2024-03-25T18:33:09.944 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:33:09.941+0000 7f31143e8640 1 -- 172.21.15.1:0/688258845 >> 172.21.15.1:0/688258845 conn(0x7f310c0617a0 msgr2=0x7f310c0b71c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:33:09.944 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:33:09.941+0000 7f31143e8640 1 -- 172.21.15.1:0/688258845 shutdown_connections 2024-03-25T18:33:09.944 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:33:09.941+0000 7f31143e8640 1 -- 172.21.15.1:0/688258845 wait complete. 2024-03-25T18:33:09.944 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:33:09.941+0000 7f31143e8640 1 librados: shutdown 2024-03-25T18:33:09.955 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T18:33:16.761 DEBUG:teuthology.orchestra.run.smithi001:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T18:33:16.877 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:33:16.876+0000 7f62d996b640 10 monclient: get_monmap_and_config 2024-03-25T18:33:16.877 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:33:16.876+0000 7f62d996b640 10 monclient: build_initial_monmap 2024-03-25T18:33:16.877 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:33:16.876+0000 7f62d996b640 10 monclient: monmap: 2024-03-25T18:33:16.877 INFO:teuthology.misc.health.smithi001.stderr:epoch 0 2024-03-25T18:33:16.877 INFO:teuthology.misc.health.smithi001.stderr:fsid c8ea72cf-9e99-4383-9b52-6c23d365fdfc 2024-03-25T18:33:16.877 INFO:teuthology.misc.health.smithi001.stderr:last_changed 2024-03-25T18:33:16.876789+0000 2024-03-25T18:33:16.877 INFO:teuthology.misc.health.smithi001.stderr:created 2024-03-25T18:33:16.876789+0000 2024-03-25T18:33:16.877 INFO:teuthology.misc.health.smithi001.stderr:min_mon_release 0 (unknown) 2024-03-25T18:33:16.877 INFO:teuthology.misc.health.smithi001.stderr:election_strategy: 3 2024-03-25T18:33:16.877 INFO:teuthology.misc.health.smithi001.stderr:0: v1:172.21.15.1:6789/0 mon.noname-a 2024-03-25T18:33:16.878 INFO:teuthology.misc.health.smithi001.stderr:1: v1:172.21.15.115:6789/0 mon.noname-c 2024-03-25T18:33:16.878 INFO:teuthology.misc.health.smithi001.stderr:2: v1:172.21.15.1:6790/0 mon.noname-b 2024-03-25T18:33:16.878 INFO:teuthology.misc.health.smithi001.stderr: 2024-03-25T18:33:16.879 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:33:16.876+0000 7f62d996b640 1 Processor -- start 2024-03-25T18:33:16.879 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:33:16.877+0000 7f62d996b640 1 -- start start 2024-03-25T18:33:16.879 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:33:16.877+0000 7f62d996b640 10 monclient: init 2024-03-25T18:33:16.879 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:33:16.877+0000 7f62d996b640 10 monclient: _reopen_session rank -1 2024-03-25T18:33:16.879 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:33:16.877+0000 7f62d996b640 10 monclient: _add_conns ranks=[2,0,1] 2024-03-25T18:33:16.880 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:33:16.877+0000 7f62d996b640 10 monclient(hunting): picked mon.noname-b con 0x7f62d40b4ff0 addr v1:172.21.15.1:6790/0 2024-03-25T18:33:16.880 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:33:16.877+0000 7f62d996b640 10 monclient(hunting): picked mon.noname-a con 0x7f62d40b4670 addr v1:172.21.15.1:6789/0 2024-03-25T18:33:16.880 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:33:16.877+0000 7f62d996b640 10 monclient(hunting): picked mon.noname-c con 0x7f62d413c480 addr v1:172.21.15.115:6789/0 2024-03-25T18:33:16.880 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:33:16.877+0000 7f62d996b640 1 -- --> v1:172.21.15.1:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f62d40ab0e0 con 0x7f62d40b4670 2024-03-25T18:33:16.880 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:33:16.878+0000 7f62d996b640 1 -- --> v1:172.21.15.115:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f62d40ab3c0 con 0x7f62d413c480 2024-03-25T18:33:16.880 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:33:16.878+0000 7f62d996b640 1 -- --> v1:172.21.15.1:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f62d40a9f30 con 0x7f62d40b4ff0 2024-03-25T18:33:16.880 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:33:16.878+0000 7f62d996b640 10 monclient(hunting): _renew_subs 2024-03-25T18:33:16.880 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:33:16.878+0000 7f62d996b640 10 monclient(hunting): authenticate will time out at 7877.347806s 2024-03-25T18:33:16.889 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:33:16.889+0000 7f62d37fe640 1 --1- >> v1:172.21.15.115:6789/0 conn(0x7f62d413c480 0x7f62d414dc00 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.115:6789/0 says I am v1:172.21.15.1:40700/0 (socket says 172.21.15.1:40700) 2024-03-25T18:33:16.890 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:33:16.889+0000 7f62d37fe640 1 -- 172.21.15.1:0/287032531 learned_addr learned my addr 172.21.15.1:0/287032531 (peer_addr_for_me v1:172.21.15.1:0/0) 2024-03-25T18:33:16.920 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:33:16.919+0000 7f62d1ffb640 1 -- 172.21.15.1:0/287032531 <== mon.0 v1:172.21.15.1:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3920751396 0 0) 0x7f62d40ab0e0 con 0x7f62d40b4670 2024-03-25T18:33:16.920 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:33:16.919+0000 7f62d1ffb640 10 monclient(hunting): _negotiate 2024-03-25T18:33:16.920 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:33:16.919+0000 7f62d1ffb640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:33:16.921 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:33:16.919+0000 7f62d1ffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:33:16.921 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:33:16.919+0000 7f62d1ffb640 10 monclient(hunting): my global_id is 4503 2024-03-25T18:33:16.921 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:33:16.920+0000 7f62d1ffb640 1 -- 172.21.15.1:0/287032531 --> v1:172.21.15.1:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f62bc003520 con 0x7f62d40b4670 2024-03-25T18:33:17.000 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:33:16.999+0000 7f62d1ffb640 1 -- 172.21.15.1:0/287032531 <== mon.0 v1:172.21.15.1:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1429471032 0 0) 0x7f62bc003520 con 0x7f62d40b4670 2024-03-25T18:33:17.001 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:33:16.999+0000 7f62d1ffb640 1 -- 172.21.15.1:0/287032531 >> v1:172.21.15.1:6790/0 conn(0x7f62d40b4ff0 legacy=0x7f62d413bd70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:33:17.001 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:33:16.999+0000 7f62d1ffb640 1 -- 172.21.15.1:0/287032531 >> v1:172.21.15.115:6789/0 conn(0x7f62d413c480 legacy=0x7f62d414dc00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:33:17.001 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:33:16.999+0000 7f62d1ffb640 10 monclient: _finish_hunting 0 2024-03-25T18:33:17.001 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:33:16.999+0000 7f62d1ffb640 1 monclient: found mon.noname-a 2024-03-25T18:33:17.001 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:33:16.999+0000 7f62d1ffb640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:33:17.001 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:33:16.999+0000 7f62d1ffb640 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.1:6789/0 2024-03-25T18:33:17.001 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:33:16.999+0000 7f62d1ffb640 1 -- 172.21.15.1:0/287032531 --> v1:172.21.15.1:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f62d40a9b70 con 0x7f62d40b4670 2024-03-25T18:33:17.001 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:33:16.999+0000 7f62d1ffb640 10 monclient: _finish_auth 0 2024-03-25T18:33:17.001 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:33:16.999+0000 7f62d1ffb640 10 monclient: _check_auth_tickets 2024-03-25T18:33:17.001 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:33:16.999+0000 7f62d1ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:33:17.001 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:33:16.999+0000 7f62d1ffb640 1 -- 172.21.15.1:0/287032531 <== mon.0 v1:172.21.15.1:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1586965025 0 0) 0x7f62c8001690 con 0x7f62d40b4670 2024-03-25T18:33:17.001 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:33:16.999+0000 7f62d1ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:33:17.001 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:33:16.999+0000 7f62d1ffb640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-03-25T18:33:17.001 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:33:16.999+0000 7f62d1ffb640 10 monclient: dump: 2024-03-25T18:33:17.002 INFO:teuthology.misc.health.smithi001.stderr:epoch 1 2024-03-25T18:33:17.002 INFO:teuthology.misc.health.smithi001.stderr:fsid c8ea72cf-9e99-4383-9b52-6c23d365fdfc 2024-03-25T18:33:17.002 INFO:teuthology.misc.health.smithi001.stderr:last_changed 2024-03-25T18:25:50.510519+0000 2024-03-25T18:33:17.002 INFO:teuthology.misc.health.smithi001.stderr:created 2024-03-25T18:25:50.510519+0000 2024-03-25T18:33:17.002 INFO:teuthology.misc.health.smithi001.stderr:min_mon_release 19 (squid) 2024-03-25T18:33:17.002 INFO:teuthology.misc.health.smithi001.stderr:election_strategy: 3 2024-03-25T18:33:17.002 INFO:teuthology.misc.health.smithi001.stderr:0: v1:172.21.15.1:6789/0 mon.a 2024-03-25T18:33:17.002 INFO:teuthology.misc.health.smithi001.stderr:1: v1:172.21.15.115:6789/0 mon.b 2024-03-25T18:33:17.002 INFO:teuthology.misc.health.smithi001.stderr:2: v1:172.21.15.1:6790/0 mon.c 2024-03-25T18:33:17.002 INFO:teuthology.misc.health.smithi001.stderr: 2024-03-25T18:33:17.002 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:33:16.999+0000 7f62d996b640 5 monclient: authenticate success, global_id 4503 2024-03-25T18:33:17.002 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:33:16.999+0000 7f62d996b640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T18:33:17.033 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:33:17.031+0000 7f62d1ffb640 1 -- 172.21.15.1:0/287032531 <== mon.0 v1:172.21.15.1:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f62c80017f0 con 0x7f62d40b4670 2024-03-25T18:33:17.033 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:33:17.031+0000 7f62d1ffb640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:33:17.033 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:33:17.031+0000 7f62d1ffb640 1 -- 172.21.15.1:0/287032531 <== mon.0 v1:172.21.15.1:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1586965025 0 0) 0x7f62c8001c00 con 0x7f62d40b4670 2024-03-25T18:33:17.033 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:33:17.031+0000 7f62d996b640 10 monclient: get_monmap_and_config success 2024-03-25T18:33:17.033 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:33:17.031+0000 7f62d996b640 4 set_mon_vals no callback set 2024-03-25T18:33:17.033 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:33:17.031+0000 7f62d1ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:33:17.034 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:33:17.031+0000 7f62d1ffb640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-25T18:33:17.034 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:33:17.031+0000 7f62d1ffb640 10 monclient: dump: 2024-03-25T18:33:17.034 INFO:teuthology.misc.health.smithi001.stderr:epoch 1 2024-03-25T18:33:17.034 INFO:teuthology.misc.health.smithi001.stderr:fsid c8ea72cf-9e99-4383-9b52-6c23d365fdfc 2024-03-25T18:33:17.034 INFO:teuthology.misc.health.smithi001.stderr:last_changed 2024-03-25T18:25:50.510519+0000 2024-03-25T18:33:17.034 INFO:teuthology.misc.health.smithi001.stderr:created 2024-03-25T18:25:50.510519+0000 2024-03-25T18:33:17.034 INFO:teuthology.misc.health.smithi001.stderr:min_mon_release 19 (squid) 2024-03-25T18:33:17.034 INFO:teuthology.misc.health.smithi001.stderr:election_strategy: 3 2024-03-25T18:33:17.034 INFO:teuthology.misc.health.smithi001.stderr:0: v1:172.21.15.1:6789/0 mon.a 2024-03-25T18:33:17.034 INFO:teuthology.misc.health.smithi001.stderr:1: v1:172.21.15.115:6789/0 mon.b 2024-03-25T18:33:17.034 INFO:teuthology.misc.health.smithi001.stderr:2: v1:172.21.15.1:6790/0 mon.c 2024-03-25T18:33:17.034 INFO:teuthology.misc.health.smithi001.stderr: 2024-03-25T18:33:17.034 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:33:17.032+0000 7f62d996b640 10 monclient: shutdown 2024-03-25T18:33:17.034 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:33:17.032+0000 7f62d996b640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:33:17.035 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:33:17.032+0000 7f62d996b640 1 -- 172.21.15.1:0/287032531 >> v1:172.21.15.1:6789/0 conn(0x7f62d40b4670 legacy=0x7f62d41425c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:33:17.035 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:33:17.032+0000 7f62d996b640 1 -- 172.21.15.1:0/287032531 shutdown_connections 2024-03-25T18:33:17.035 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:33:17.032+0000 7f62d996b640 1 -- 172.21.15.1:0/287032531 >> 172.21.15.1:0/287032531 conn(0x7f62d40617a0 msgr2=0x7f62d4061bb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:33:17.035 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:33:17.032+0000 7f62d996b640 1 -- 172.21.15.1:0/287032531 shutdown_connections 2024-03-25T18:33:17.035 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:33:17.032+0000 7f62d996b640 1 -- 172.21.15.1:0/287032531 wait complete. 2024-03-25T18:33:17.036 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:33:17.033+0000 7f62d996b640 10 monclient: build_initial_monmap 2024-03-25T18:33:17.036 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:33:17.033+0000 7f62d996b640 10 monclient: monmap: 2024-03-25T18:33:17.036 INFO:teuthology.misc.health.smithi001.stderr:epoch 0 2024-03-25T18:33:17.037 INFO:teuthology.misc.health.smithi001.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T18:33:17.037 INFO:teuthology.misc.health.smithi001.stderr:last_changed 0.000000 2024-03-25T18:33:17.037 INFO:teuthology.misc.health.smithi001.stderr:created 0.000000 2024-03-25T18:33:17.037 INFO:teuthology.misc.health.smithi001.stderr:min_mon_release 0 (unknown) 2024-03-25T18:33:17.037 INFO:teuthology.misc.health.smithi001.stderr:election_strategy: 1 2024-03-25T18:33:17.037 INFO:teuthology.misc.health.smithi001.stderr:0: v1:172.21.15.1:6789/0 mon.noname-a 2024-03-25T18:33:17.037 INFO:teuthology.misc.health.smithi001.stderr:1: v1:172.21.15.115:6789/0 mon.noname-b 2024-03-25T18:33:17.037 INFO:teuthology.misc.health.smithi001.stderr:2: v1:172.21.15.1:6790/0 mon.noname-c 2024-03-25T18:33:17.037 INFO:teuthology.misc.health.smithi001.stderr: 2024-03-25T18:33:17.037 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:33:17.033+0000 7f62d996b640 1 librados: starting msgr at 2024-03-25T18:33:17.037 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:33:17.033+0000 7f62d996b640 1 librados: starting objecter 2024-03-25T18:33:17.037 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:33:17.033+0000 7f62d996b640 1 Processor -- start 2024-03-25T18:33:17.037 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:33:17.033+0000 7f62d996b640 1 -- start start 2024-03-25T18:33:17.037 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:33:17.033+0000 7f62d996b640 1 librados: setting wanted keys 2024-03-25T18:33:17.037 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:33:17.033+0000 7f62d996b640 1 librados: calling monclient init 2024-03-25T18:33:17.038 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:33:17.033+0000 7f62d996b640 10 monclient: init 2024-03-25T18:33:17.038 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:33:17.034+0000 7f62d996b640 10 monclient: _reopen_session rank -1 2024-03-25T18:33:17.038 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:33:17.034+0000 7f62d996b640 10 monclient: _add_conns ranks=[1,2,0] 2024-03-25T18:33:17.038 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:33:17.034+0000 7f62d996b640 10 monclient(hunting): picked mon.noname-b con 0x7f62d40b4670 addr v1:172.21.15.115:6789/0 2024-03-25T18:33:17.038 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:33:17.034+0000 7f62d996b640 10 monclient(hunting): picked mon.noname-c con 0x7f62d40b4ff0 addr v1:172.21.15.1:6790/0 2024-03-25T18:33:17.038 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:33:17.034+0000 7f62d996b640 10 monclient(hunting): picked mon.noname-a con 0x7f62d413c480 addr v1:172.21.15.1:6789/0 2024-03-25T18:33:17.038 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:33:17.034+0000 7f62d996b640 1 -- --> v1:172.21.15.1:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f62d4144790 con 0x7f62d413c480 2024-03-25T18:33:17.038 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:33:17.034+0000 7f62d996b640 1 -- --> v1:172.21.15.115:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f62d4144a70 con 0x7f62d40b4670 2024-03-25T18:33:17.038 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:33:17.034+0000 7f62d996b640 1 -- --> v1:172.21.15.1:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f62d4144d50 con 0x7f62d40b4ff0 2024-03-25T18:33:17.038 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:33:17.034+0000 7f62d996b640 10 monclient(hunting): _renew_subs 2024-03-25T18:33:17.038 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:33:17.034+0000 7f62d996b640 10 monclient(hunting): authenticate will time out at 7877.504352s 2024-03-25T18:33:17.042 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:33:17.041+0000 7f62d27fc640 1 --1- >> v1:172.21.15.1:6790/0 conn(0x7f62d40b4ff0 0x7f62d4145ed0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.1:6790/0 says I am v1:172.21.15.1:33858/0 (socket says 172.21.15.1:33858) 2024-03-25T18:33:17.042 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:33:17.041+0000 7f62d27fc640 1 -- 172.21.15.1:0/3195963245 learned_addr learned my addr 172.21.15.1:0/3195963245 (peer_addr_for_me v1:172.21.15.1:0/0) 2024-03-25T18:33:17.043 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:33:17.042+0000 7f62bbfff640 10 client.?.objecter ms_handle_connect 0x7f62d40b4ff0 2024-03-25T18:33:17.043 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:33:17.042+0000 7f62bbfff640 10 client.?.objecter resend_mon_ops 2024-03-25T18:33:17.049 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:33:17.048+0000 7f62bbfff640 10 client.?.objecter ms_handle_connect 0x7f62d413c480 2024-03-25T18:33:17.049 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:33:17.048+0000 7f62bbfff640 10 client.?.objecter resend_mon_ops 2024-03-25T18:33:17.049 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:33:17.049+0000 7f62bbfff640 10 client.?.objecter ms_handle_connect 0x7f62d40b4670 2024-03-25T18:33:17.049 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:33:17.049+0000 7f62bbfff640 10 client.?.objecter resend_mon_ops 2024-03-25T18:33:17.106 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:33:17.105+0000 7f62bbfff640 1 -- 172.21.15.1:0/3195963245 <== mon.0 v1:172.21.15.1:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1546045067 0 0) 0x7f62d4144790 con 0x7f62d413c480 2024-03-25T18:33:17.106 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:33:17.105+0000 7f62bbfff640 10 monclient(hunting): _negotiate 2024-03-25T18:33:17.106 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:33:17.105+0000 7f62bbfff640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:33:17.106 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:33:17.105+0000 7f62bbfff640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:33:17.107 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:33:17.105+0000 7f62bbfff640 10 monclient(hunting): my global_id is 4506 2024-03-25T18:33:17.107 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:33:17.105+0000 7f62bbfff640 1 -- 172.21.15.1:0/3195963245 --> v1:172.21.15.1:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f62c0004eb0 con 0x7f62d413c480 2024-03-25T18:33:17.167 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:33:17.166+0000 7f62bbfff640 1 -- 172.21.15.1:0/3195963245 <== mon.1 v1:172.21.15.115:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3102153783 0 0) 0x7f62d4144a70 con 0x7f62d40b4670 2024-03-25T18:33:17.167 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:33:17.166+0000 7f62bbfff640 10 monclient(hunting): _negotiate 2024-03-25T18:33:17.167 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:33:17.166+0000 7f62bbfff640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:33:17.167 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:33:17.166+0000 7f62bbfff640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:33:17.168 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:33:17.166+0000 7f62bbfff640 10 monclient(hunting): my global_id is 4519 2024-03-25T18:33:17.168 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:33:17.166+0000 7f62bbfff640 1 -- 172.21.15.1:0/3195963245 --> v1:172.21.15.115:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f62d4144790 con 0x7f62d40b4670 2024-03-25T18:33:17.179 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:33:17.177+0000 7f62bbfff640 1 -- 172.21.15.1:0/3195963245 <== mon.1 v1:172.21.15.115:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 4125211638 0 0) 0x7f62d4144790 con 0x7f62d40b4670 2024-03-25T18:33:17.179 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:33:17.178+0000 7f62bbfff640 1 -- 172.21.15.1:0/3195963245 --> v1:172.21.15.115:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f62d4144a70 con 0x7f62d40b4670 2024-03-25T18:33:17.179 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:33:17.178+0000 7f62bbfff640 1 -- 172.21.15.1:0/3195963245 <== mon.1 v1:172.21.15.115:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1586965025 0 0) 0x7f62c40018a0 con 0x7f62d40b4670 2024-03-25T18:33:17.179 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:33:17.178+0000 7f62bbfff640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:33:17.179 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:33:17.178+0000 7f62bbfff640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-03-25T18:33:17.179 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:33:17.178+0000 7f62bbfff640 10 monclient(hunting): dump: 2024-03-25T18:33:17.179 INFO:teuthology.misc.health.smithi001.stderr:epoch 1 2024-03-25T18:33:17.179 INFO:teuthology.misc.health.smithi001.stderr:fsid c8ea72cf-9e99-4383-9b52-6c23d365fdfc 2024-03-25T18:33:17.179 INFO:teuthology.misc.health.smithi001.stderr:last_changed 2024-03-25T18:25:50.510519+0000 2024-03-25T18:33:17.179 INFO:teuthology.misc.health.smithi001.stderr:created 2024-03-25T18:25:50.510519+0000 2024-03-25T18:33:17.179 INFO:teuthology.misc.health.smithi001.stderr:min_mon_release 19 (squid) 2024-03-25T18:33:17.179 INFO:teuthology.misc.health.smithi001.stderr:election_strategy: 3 2024-03-25T18:33:17.179 INFO:teuthology.misc.health.smithi001.stderr:0: v1:172.21.15.1:6789/0 mon.a 2024-03-25T18:33:17.179 INFO:teuthology.misc.health.smithi001.stderr:1: v1:172.21.15.115:6789/0 mon.b 2024-03-25T18:33:17.180 INFO:teuthology.misc.health.smithi001.stderr:2: v1:172.21.15.1:6790/0 mon.c 2024-03-25T18:33:17.180 INFO:teuthology.misc.health.smithi001.stderr: 2024-03-25T18:33:17.180 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:33:17.178+0000 7f62bbfff640 10 monclient(hunting): _finish_auth 0 2024-03-25T18:33:17.188 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:33:17.187+0000 7f62bbfff640 1 -- 172.21.15.1:0/3195963245 <== mon.0 v1:172.21.15.1:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2848969914 0 0) 0x7f62c0004eb0 con 0x7f62d413c480 2024-03-25T18:33:17.188 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:33:17.187+0000 7f62bbfff640 1 -- 172.21.15.1:0/3195963245 --> v1:172.21.15.1:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f62d4144790 con 0x7f62d413c480 2024-03-25T18:33:17.189 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:33:17.187+0000 7f62bbfff640 1 -- 172.21.15.1:0/3195963245 <== mon.0 v1:172.21.15.1:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1586965025 0 0) 0x7f62cc001b00 con 0x7f62d413c480 2024-03-25T18:33:17.189 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:33:17.187+0000 7f62bbfff640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:33:17.189 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:33:17.187+0000 7f62bbfff640 10 monclient(hunting): got monmap 1 from mon.a (according to old e1) 2024-03-25T18:33:17.189 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:33:17.187+0000 7f62bbfff640 10 monclient(hunting): dump: 2024-03-25T18:33:17.189 INFO:teuthology.misc.health.smithi001.stderr:epoch 1 2024-03-25T18:33:17.189 INFO:teuthology.misc.health.smithi001.stderr:fsid c8ea72cf-9e99-4383-9b52-6c23d365fdfc 2024-03-25T18:33:17.189 INFO:teuthology.misc.health.smithi001.stderr:last_changed 2024-03-25T18:25:50.510519+0000 2024-03-25T18:33:17.189 INFO:teuthology.misc.health.smithi001.stderr:created 2024-03-25T18:25:50.510519+0000 2024-03-25T18:33:17.189 INFO:teuthology.misc.health.smithi001.stderr:min_mon_release 19 (squid) 2024-03-25T18:33:17.189 INFO:teuthology.misc.health.smithi001.stderr:election_strategy: 3 2024-03-25T18:33:17.189 INFO:teuthology.misc.health.smithi001.stderr:0: v1:172.21.15.1:6789/0 mon.a 2024-03-25T18:33:17.189 INFO:teuthology.misc.health.smithi001.stderr:1: v1:172.21.15.115:6789/0 mon.b 2024-03-25T18:33:17.190 INFO:teuthology.misc.health.smithi001.stderr:2: v1:172.21.15.1:6790/0 mon.c 2024-03-25T18:33:17.190 INFO:teuthology.misc.health.smithi001.stderr: 2024-03-25T18:33:17.203 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:33:17.202+0000 7f62bbfff640 1 -- 172.21.15.1:0/3195963245 <== mon.0 v1:172.21.15.1:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 1067967064 0 0) 0x7f62d4144790 con 0x7f62d413c480 2024-03-25T18:33:17.203 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:33:17.202+0000 7f62bbfff640 1 -- 172.21.15.1:0/3195963245 >> v1:172.21.15.1:6790/0 conn(0x7f62d40b4ff0 legacy=0x7f62d4145ed0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:33:17.203 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:33:17.202+0000 7f62bbfff640 1 -- 172.21.15.1:0/3195963245 >> v1:172.21.15.115:6789/0 conn(0x7f62d40b4670 legacy=0x7f62d41457c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:33:17.203 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:33:17.202+0000 7f62bbfff640 10 monclient: _finish_hunting 0 2024-03-25T18:33:17.204 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:33:17.202+0000 7f62bbfff640 1 monclient: found mon.a 2024-03-25T18:33:17.204 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:33:17.202+0000 7f62bbfff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:33:17.204 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:33:17.202+0000 7f62bbfff640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.1:6789/0 2024-03-25T18:33:17.204 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:33:17.202+0000 7f62bbfff640 1 -- 172.21.15.1:0/3195963245 --> v1:172.21.15.1:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f62d4145030 con 0x7f62d413c480 2024-03-25T18:33:17.204 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:33:17.202+0000 7f62bbfff640 10 monclient: _finish_auth 0 2024-03-25T18:33:17.204 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:33:17.202+0000 7f62bbfff640 10 monclient: _check_auth_tickets 2024-03-25T18:33:17.204 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:33:17.202+0000 7f62bbfff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:33:17.204 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:33:17.202+0000 7f62d996b640 5 monclient: authenticate success, global_id 4506 2024-03-25T18:33:17.204 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:33:17.202+0000 7f62d996b640 10 monclient: _renew_subs 2024-03-25T18:33:17.204 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:33:17.202+0000 7f62d996b640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.1:6789/0 2024-03-25T18:33:17.204 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:33:17.202+0000 7f62d996b640 1 -- 172.21.15.1:0/3195963245 --> v1:172.21.15.1:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f62d41435a0 con 0x7f62d413c480 2024-03-25T18:33:17.204 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:33:17.202+0000 7f62d996b640 10 client.4506.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T18:33:17.204 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:33:17.202+0000 7f62d996b640 10 monclient: _renew_subs 2024-03-25T18:33:17.204 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:33:17.202+0000 7f62d996b640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.1:6789/0 2024-03-25T18:33:17.204 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:33:17.202+0000 7f62d996b640 1 -- 172.21.15.1:0/3195963245 --> v1:172.21.15.1:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f62d413dbe0 con 0x7f62d413c480 2024-03-25T18:33:17.205 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:33:17.202+0000 7f62d996b640 1 librados: init done 2024-03-25T18:33:17.205 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:33:17.203+0000 7f62d996b640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T18:33:17.205 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:33:17.203+0000 7f62d996b640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:33:17.205 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:33:17.203+0000 7f62d996b640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.1:6789/0 2024-03-25T18:33:17.205 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:33:17.203+0000 7f62d996b640 1 -- 172.21.15.1:0/3195963245 --> v1:172.21.15.1:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f62d40688c0 con 0x7f62d413c480 2024-03-25T18:33:17.561 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:33:17.560+0000 7f62bbfff640 1 -- 172.21.15.1:0/3195963245 <== mon.0 v1:172.21.15.1:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f62cc002010 con 0x7f62d413c480 2024-03-25T18:33:17.561 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:33:17.560+0000 7f62bbfff640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:33:17.561 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:33:17.560+0000 7f62bbfff640 1 -- 172.21.15.1:0/3195963245 <== mon.0 v1:172.21.15.1:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1586965025 0 0) 0x7f62cc004d00 con 0x7f62d413c480 2024-03-25T18:33:17.561 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:33:17.560+0000 7f62bbfff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:33:17.561 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:33:17.560+0000 7f62bbfff640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-25T18:33:17.561 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:33:17.560+0000 7f62bbfff640 10 monclient: dump: 2024-03-25T18:33:17.561 INFO:teuthology.misc.health.smithi001.stderr:epoch 1 2024-03-25T18:33:17.562 INFO:teuthology.misc.health.smithi001.stderr:fsid c8ea72cf-9e99-4383-9b52-6c23d365fdfc 2024-03-25T18:33:17.562 INFO:teuthology.misc.health.smithi001.stderr:last_changed 2024-03-25T18:25:50.510519+0000 2024-03-25T18:33:17.562 INFO:teuthology.misc.health.smithi001.stderr:created 2024-03-25T18:25:50.510519+0000 2024-03-25T18:33:17.562 INFO:teuthology.misc.health.smithi001.stderr:min_mon_release 19 (squid) 2024-03-25T18:33:17.562 INFO:teuthology.misc.health.smithi001.stderr:election_strategy: 3 2024-03-25T18:33:17.562 INFO:teuthology.misc.health.smithi001.stderr:0: v1:172.21.15.1:6789/0 mon.a 2024-03-25T18:33:17.562 INFO:teuthology.misc.health.smithi001.stderr:1: v1:172.21.15.115:6789/0 mon.b 2024-03-25T18:33:17.562 INFO:teuthology.misc.health.smithi001.stderr:2: v1:172.21.15.1:6790/0 mon.c 2024-03-25T18:33:17.562 INFO:teuthology.misc.health.smithi001.stderr: 2024-03-25T18:33:17.562 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:33:17.560+0000 7f62d17fa640 4 set_mon_vals no callback set 2024-03-25T18:33:17.563 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:33:17.561+0000 7f62bbfff640 1 -- 172.21.15.1:0/3195963245 <== mon.0 v1:172.21.15.1:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 21787975 0 0) 0x7f62cc01ad10 con 0x7f62d413c480 2024-03-25T18:33:17.563 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:33:17.562+0000 7f62d2ffd640 10 monclient: get_auth_request con 0x7f62c006de90 auth_method 0 2024-03-25T18:33:17.563 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:33:17.562+0000 7f62bbfff640 10 client.4506.objecter ms_handle_connect 0x7f62c006de90 2024-03-25T18:33:17.568 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:33:17.567+0000 7f62bbfff640 1 -- 172.21.15.1:0/3195963245 <== mon.0 v1:172.21.15.1:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 996254230 0 0) 0x7f62cc054a20 con 0x7f62d413c480 2024-03-25T18:33:17.568 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:33:17.567+0000 7f62bbfff640 10 client.4506.objecter ms_dispatch 0x7f62d4000b90 osd_map(10..10 src has 1..10) v4 2024-03-25T18:33:17.568 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:33:17.567+0000 7f62bbfff640 3 client.4506.objecter handle_osd_map got epochs [10,10] > 0 2024-03-25T18:33:17.568 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:33:17.567+0000 7f62bbfff640 3 client.4506.objecter handle_osd_map decoding full epoch 10 2024-03-25T18:33:17.568 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:33:17.567+0000 7f62bbfff640 20 client.4506.objecter dump_active .. 0 homeless 2024-03-25T18:33:17.569 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:33:17.567+0000 7f62bbfff640 1 -- 172.21.15.1:0/3195963245 <== mon.0 v1:172.21.15.1:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f62cc050360 con 0x7f62d413c480 2024-03-25T18:33:17.569 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:33:17.567+0000 7f62bbfff640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:33:17.569 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:33:17.567+0000 7f62bbfff640 10 monclient: _finish_command 1 = system:0 2024-03-25T18:33:17.827 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:33:17.826+0000 7f62d996b640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:33:17.827 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:33:17.826+0000 7f62d996b640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:33:17.828 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:33:17.826+0000 7f62d996b640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.1:6789/0 2024-03-25T18:33:17.828 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:33:17.826+0000 7f62d996b640 1 -- 172.21.15.1:0/3195963245 --> v1:172.21.15.1:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f62d406dbd0 con 0x7f62d413c480 2024-03-25T18:33:17.863 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:33:17.862+0000 7f62bbfff640 1 -- 172.21.15.1:0/3195963245 <== mon.0 v1:172.21.15.1:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 4092213348) 0x7f62cc052460 con 0x7f62d413c480 2024-03-25T18:33:17.863 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:33:17.862+0000 7f62bbfff640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:33:17.863 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:33:17.862+0000 7f62bbfff640 10 monclient: _finish_command 2 = system:0 2024-03-25T18:33:17.863 INFO:teuthology.misc.health.smithi001.stdout: 2024-03-25T18:33:17.863 INFO:teuthology.misc.health.smithi001.stdout:{"epoch":10,"fsid":"c8ea72cf-9e99-4383-9b52-6c23d365fdfc","created":"2024-03-25T18:26:38.856638+0000","modified":"2024-03-25T18:26:44.224208+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T18:26:44.224208+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"b9f34922-3a84-49be-8426-807c1eceaefc","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"42bf306f-55e3-4634-9874-8b4b73ac6644","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"292cb23a-18a9-4207-a7cc-4d5727feaec0","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"45e0e9ab-898d-4673-9590-b4117bc5e84a","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"831a5a09-73ab-4a61-9872-088e76bdbc66","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"88cd903d-3b55-4840-9396-6e3cfe005282","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"89db7355-bc9d-46f4-8ab0-e8d9e744e026","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"cac47bee-0914-4f08-896f-348838882866","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T18:33:17.866 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:33:17.864+0000 7f62d996b640 10 librados: watch_flush enter 2024-03-25T18:33:17.866 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:33:17.864+0000 7f62d996b640 10 librados: watch_flush exit 2024-03-25T18:33:17.866 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:33:17.864+0000 7f62d996b640 20 client.4506.objecter shutdown clearing up homeless session... 2024-03-25T18:33:17.866 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:33:17.864+0000 7f62d996b640 10 client.4506.objecter successfully canceled tick 2024-03-25T18:33:17.866 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:33:17.864+0000 7f62d996b640 1 -- 172.21.15.1:0/3195963245 >> v1:172.21.15.115:6800/4237379176 conn(0x7f62c006de90 legacy=0x7f62c0070350 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:33:17.866 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:33:17.864+0000 7f62d996b640 10 monclient: shutdown 2024-03-25T18:33:17.866 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:33:17.864+0000 7f62d996b640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:33:17.866 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:33:17.864+0000 7f62d996b640 1 -- 172.21.15.1:0/3195963245 >> v1:172.21.15.1:6789/0 conn(0x7f62d413c480 legacy=0x7f62d41e2170 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:33:17.867 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:33:17.865+0000 7f62d996b640 1 -- 172.21.15.1:0/3195963245 shutdown_connections 2024-03-25T18:33:17.867 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:33:17.865+0000 7f62d996b640 1 -- 172.21.15.1:0/3195963245 >> 172.21.15.1:0/3195963245 conn(0x7f62d40617a0 msgr2=0x7f62d4140760 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:33:17.867 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:33:17.865+0000 7f62d996b640 1 -- 172.21.15.1:0/3195963245 shutdown_connections 2024-03-25T18:33:17.867 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:33:17.865+0000 7f62d996b640 1 -- 172.21.15.1:0/3195963245 wait complete. 2024-03-25T18:33:17.867 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:33:17.865+0000 7f62d996b640 1 librados: shutdown 2024-03-25T18:33:17.878 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T18:33:22.202 INFO:tasks.ceph.mon.a.smithi001.stderr:==00:00:07:00.760 34876== Warning: unimplemented fcntl command: 1036 2024-03-25T18:33:22.215 INFO:tasks.ceph.mon.a.smithi001.stderr:==00:00:07:00.774 34876== Warning: unimplemented fcntl command: 1036 2024-03-25T18:33:22.259 INFO:tasks.ceph.mon.b.smithi115.stderr:==00:00:07:00.743 34836== Warning: unimplemented fcntl command: 1036 2024-03-25T18:33:22.268 INFO:tasks.ceph.mon.a.smithi001.stderr:==00:00:07:00.826 34876== Warning: unimplemented fcntl command: 1036 2024-03-25T18:33:22.272 INFO:tasks.ceph.mon.b.smithi115.stderr:==00:00:07:00.755 34836== Warning: unimplemented fcntl command: 1036 2024-03-25T18:33:22.281 INFO:tasks.ceph.mon.c.smithi001.stderr:==00:00:07:00.839 34877== Warning: unimplemented fcntl command: 1036 2024-03-25T18:33:22.293 INFO:tasks.ceph.mon.c.smithi001.stderr:==00:00:07:00.851 34877== Warning: unimplemented fcntl command: 1036 2024-03-25T18:33:22.298 INFO:tasks.ceph.mon.b.smithi115.stderr:==00:00:07:00.781 34836== Warning: unimplemented fcntl command: 1036 2024-03-25T18:33:22.317 INFO:tasks.ceph.mon.c.smithi001.stderr:==00:00:07:00.875 34877== Warning: unimplemented fcntl command: 1036 2024-03-25T18:33:24.684 DEBUG:teuthology.orchestra.run.smithi001:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T18:33:24.783 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:33:24.782+0000 7f8b4c44e640 10 monclient: get_monmap_and_config 2024-03-25T18:33:24.783 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:33:24.782+0000 7f8b4c44e640 10 monclient: build_initial_monmap 2024-03-25T18:33:24.783 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:33:24.782+0000 7f8b4c44e640 10 monclient: monmap: 2024-03-25T18:33:24.784 INFO:teuthology.misc.health.smithi001.stderr:epoch 0 2024-03-25T18:33:24.784 INFO:teuthology.misc.health.smithi001.stderr:fsid c8ea72cf-9e99-4383-9b52-6c23d365fdfc 2024-03-25T18:33:24.784 INFO:teuthology.misc.health.smithi001.stderr:last_changed 2024-03-25T18:33:24.783332+0000 2024-03-25T18:33:24.784 INFO:teuthology.misc.health.smithi001.stderr:created 2024-03-25T18:33:24.783332+0000 2024-03-25T18:33:24.784 INFO:teuthology.misc.health.smithi001.stderr:min_mon_release 0 (unknown) 2024-03-25T18:33:24.784 INFO:teuthology.misc.health.smithi001.stderr:election_strategy: 3 2024-03-25T18:33:24.784 INFO:teuthology.misc.health.smithi001.stderr:0: v1:172.21.15.1:6789/0 mon.noname-a 2024-03-25T18:33:24.784 INFO:teuthology.misc.health.smithi001.stderr:1: v1:172.21.15.115:6789/0 mon.noname-c 2024-03-25T18:33:24.784 INFO:teuthology.misc.health.smithi001.stderr:2: v1:172.21.15.1:6790/0 mon.noname-b 2024-03-25T18:33:24.784 INFO:teuthology.misc.health.smithi001.stderr: 2024-03-25T18:33:24.785 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:33:24.783+0000 7f8b4c44e640 1 Processor -- start 2024-03-25T18:33:24.786 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:33:24.783+0000 7f8b4c44e640 1 -- start start 2024-03-25T18:33:24.786 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:33:24.783+0000 7f8b4c44e640 10 monclient: init 2024-03-25T18:33:24.786 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:33:24.784+0000 7f8b4c44e640 10 monclient: _reopen_session rank -1 2024-03-25T18:33:24.786 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:33:24.784+0000 7f8b4c44e640 10 monclient: _add_conns ranks=[1,2,0] 2024-03-25T18:33:24.786 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:33:24.784+0000 7f8b4c44e640 10 monclient(hunting): picked mon.noname-c con 0x7f8b440b4ff0 addr v1:172.21.15.115:6789/0 2024-03-25T18:33:24.786 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:33:24.784+0000 7f8b4c44e640 10 monclient(hunting): picked mon.noname-b con 0x7f8b440b4670 addr v1:172.21.15.1:6790/0 2024-03-25T18:33:24.786 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:33:24.784+0000 7f8b4c44e640 10 monclient(hunting): picked mon.noname-a con 0x7f8b4413c480 addr v1:172.21.15.1:6789/0 2024-03-25T18:33:24.786 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:33:24.784+0000 7f8b4c44e640 1 -- --> v1:172.21.15.1:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8b440ab0e0 con 0x7f8b4413c480 2024-03-25T18:33:24.786 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:33:24.784+0000 7f8b4c44e640 1 -- --> v1:172.21.15.115:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8b440ab3c0 con 0x7f8b440b4ff0 2024-03-25T18:33:24.786 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:33:24.784+0000 7f8b4c44e640 1 -- --> v1:172.21.15.1:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8b440a9f30 con 0x7f8b440b4670 2024-03-25T18:33:24.786 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:33:24.784+0000 7f8b4c44e640 10 monclient(hunting): _renew_subs 2024-03-25T18:33:24.786 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:33:24.784+0000 7f8b4c44e640 10 monclient(hunting): authenticate will time out at 7885.254283s 2024-03-25T18:33:24.793 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:33:24.792+0000 7f8b4a9eb640 1 --1- >> v1:172.21.15.1:6789/0 conn(0x7f8b4413c480 0x7f8b4414dc00 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.1:6789/0 says I am v1:172.21.15.1:33084/0 (socket says 172.21.15.1:33084) 2024-03-25T18:33:24.793 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:33:24.792+0000 7f8b4a9eb640 1 -- 172.21.15.1:0/4059079670 learned_addr learned my addr 172.21.15.1:0/4059079670 (peer_addr_for_me v1:172.21.15.1:0/0) 2024-03-25T18:33:24.807 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:33:24.806+0000 7f8b491e8640 1 -- 172.21.15.1:0/4059079670 <== mon.0 v1:172.21.15.1:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2425010998 0 0) 0x7f8b440ab0e0 con 0x7f8b4413c480 2024-03-25T18:33:24.807 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:33:24.806+0000 7f8b491e8640 10 monclient(hunting): _negotiate 2024-03-25T18:33:24.807 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:33:24.806+0000 7f8b491e8640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:33:24.807 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:33:24.806+0000 7f8b491e8640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:33:24.807 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:33:24.806+0000 7f8b491e8640 10 monclient(hunting): my global_id is 4509 2024-03-25T18:33:24.808 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:33:24.806+0000 7f8b491e8640 1 -- 172.21.15.1:0/4059079670 --> v1:172.21.15.1:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f8b30003520 con 0x7f8b4413c480 2024-03-25T18:33:24.822 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:33:24.821+0000 7f8b491e8640 1 -- 172.21.15.1:0/4059079670 <== mon.0 v1:172.21.15.1:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 585823286 0 0) 0x7f8b30003520 con 0x7f8b4413c480 2024-03-25T18:33:24.823 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:33:24.821+0000 7f8b491e8640 1 -- 172.21.15.1:0/4059079670 >> v1:172.21.15.1:6790/0 conn(0x7f8b440b4670 legacy=0x7f8b441425c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:33:24.823 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:33:24.821+0000 7f8b491e8640 1 -- 172.21.15.1:0/4059079670 >> v1:172.21.15.115:6789/0 conn(0x7f8b440b4ff0 legacy=0x7f8b4413bd70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:33:24.823 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:33:24.821+0000 7f8b491e8640 10 monclient: _finish_hunting 0 2024-03-25T18:33:24.823 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:33:24.821+0000 7f8b491e8640 1 monclient: found mon.noname-a 2024-03-25T18:33:24.823 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:33:24.821+0000 7f8b491e8640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:33:24.823 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:33:24.821+0000 7f8b491e8640 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.1:6789/0 2024-03-25T18:33:24.823 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:33:24.821+0000 7f8b491e8640 1 -- 172.21.15.1:0/4059079670 --> v1:172.21.15.1:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8b440a9b70 con 0x7f8b4413c480 2024-03-25T18:33:24.823 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:33:24.821+0000 7f8b491e8640 10 monclient: _finish_auth 0 2024-03-25T18:33:24.823 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:33:24.821+0000 7f8b491e8640 10 monclient: _check_auth_tickets 2024-03-25T18:33:24.823 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:33:24.821+0000 7f8b491e8640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:33:24.823 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:33:24.821+0000 7f8b491e8640 1 -- 172.21.15.1:0/4059079670 <== mon.0 v1:172.21.15.1:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1586965025 0 0) 0x7f8b3c001a10 con 0x7f8b4413c480 2024-03-25T18:33:24.823 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:33:24.821+0000 7f8b4c44e640 5 monclient: authenticate success, global_id 4509 2024-03-25T18:33:24.823 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:33:24.821+0000 7f8b4c44e640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T18:33:24.823 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:33:24.821+0000 7f8b491e8640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:33:24.824 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:33:24.821+0000 7f8b491e8640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-03-25T18:33:24.824 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:33:24.821+0000 7f8b491e8640 10 monclient: dump: 2024-03-25T18:33:24.824 INFO:teuthology.misc.health.smithi001.stderr:epoch 1 2024-03-25T18:33:24.824 INFO:teuthology.misc.health.smithi001.stderr:fsid c8ea72cf-9e99-4383-9b52-6c23d365fdfc 2024-03-25T18:33:24.824 INFO:teuthology.misc.health.smithi001.stderr:last_changed 2024-03-25T18:25:50.510519+0000 2024-03-25T18:33:24.824 INFO:teuthology.misc.health.smithi001.stderr:created 2024-03-25T18:25:50.510519+0000 2024-03-25T18:33:24.824 INFO:teuthology.misc.health.smithi001.stderr:min_mon_release 19 (squid) 2024-03-25T18:33:24.824 INFO:teuthology.misc.health.smithi001.stderr:election_strategy: 3 2024-03-25T18:33:24.824 INFO:teuthology.misc.health.smithi001.stderr:0: v1:172.21.15.1:6789/0 mon.a 2024-03-25T18:33:24.824 INFO:teuthology.misc.health.smithi001.stderr:1: v1:172.21.15.115:6789/0 mon.b 2024-03-25T18:33:24.824 INFO:teuthology.misc.health.smithi001.stderr:2: v1:172.21.15.1:6790/0 mon.c 2024-03-25T18:33:24.824 INFO:teuthology.misc.health.smithi001.stderr: 2024-03-25T18:33:24.824 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:33:24.821+0000 7f8b4c44e640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T18:33:24.854 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:33:24.853+0000 7f8b491e8640 1 -- 172.21.15.1:0/4059079670 <== mon.0 v1:172.21.15.1:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f8b3c001b70 con 0x7f8b4413c480 2024-03-25T18:33:24.854 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:33:24.853+0000 7f8b491e8640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:33:24.854 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:33:24.853+0000 7f8b491e8640 1 -- 172.21.15.1:0/4059079670 <== mon.0 v1:172.21.15.1:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1586965025 0 0) 0x7f8b3c001f80 con 0x7f8b4413c480 2024-03-25T18:33:24.854 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:33:24.853+0000 7f8b4c44e640 10 monclient: get_monmap_and_config success 2024-03-25T18:33:24.854 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:33:24.853+0000 7f8b4c44e640 4 set_mon_vals no callback set 2024-03-25T18:33:24.854 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:33:24.853+0000 7f8b491e8640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:33:24.854 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:33:24.853+0000 7f8b491e8640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-25T18:33:24.854 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:33:24.853+0000 7f8b491e8640 10 monclient: dump: 2024-03-25T18:33:24.854 INFO:teuthology.misc.health.smithi001.stderr:epoch 1 2024-03-25T18:33:24.854 INFO:teuthology.misc.health.smithi001.stderr:fsid c8ea72cf-9e99-4383-9b52-6c23d365fdfc 2024-03-25T18:33:24.854 INFO:teuthology.misc.health.smithi001.stderr:last_changed 2024-03-25T18:25:50.510519+0000 2024-03-25T18:33:24.854 INFO:teuthology.misc.health.smithi001.stderr:created 2024-03-25T18:25:50.510519+0000 2024-03-25T18:33:24.855 INFO:teuthology.misc.health.smithi001.stderr:min_mon_release 19 (squid) 2024-03-25T18:33:24.855 INFO:teuthology.misc.health.smithi001.stderr:election_strategy: 3 2024-03-25T18:33:24.855 INFO:teuthology.misc.health.smithi001.stderr:0: v1:172.21.15.1:6789/0 mon.a 2024-03-25T18:33:24.855 INFO:teuthology.misc.health.smithi001.stderr:1: v1:172.21.15.115:6789/0 mon.b 2024-03-25T18:33:24.855 INFO:teuthology.misc.health.smithi001.stderr:2: v1:172.21.15.1:6790/0 mon.c 2024-03-25T18:33:24.855 INFO:teuthology.misc.health.smithi001.stderr: 2024-03-25T18:33:24.857 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:33:24.854+0000 7f8b4c44e640 10 monclient: shutdown 2024-03-25T18:33:24.857 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:33:24.854+0000 7f8b4c44e640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:33:24.857 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:33:24.854+0000 7f8b4c44e640 1 -- 172.21.15.1:0/4059079670 >> v1:172.21.15.1:6789/0 conn(0x7f8b4413c480 legacy=0x7f8b4414dc00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:33:24.857 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:33:24.854+0000 7f8b4c44e640 1 -- 172.21.15.1:0/4059079670 shutdown_connections 2024-03-25T18:33:24.857 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:33:24.854+0000 7f8b4c44e640 1 -- 172.21.15.1:0/4059079670 >> 172.21.15.1:0/4059079670 conn(0x7f8b440617a0 msgr2=0x7f8b44061bb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:33:24.857 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:33:24.854+0000 7f8b4c44e640 1 -- 172.21.15.1:0/4059079670 shutdown_connections 2024-03-25T18:33:24.857 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:33:24.854+0000 7f8b4c44e640 1 -- 172.21.15.1:0/4059079670 wait complete. 2024-03-25T18:33:24.857 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:33:24.855+0000 7f8b4c44e640 10 monclient: build_initial_monmap 2024-03-25T18:33:24.857 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:33:24.855+0000 7f8b4c44e640 10 monclient: monmap: 2024-03-25T18:33:24.857 INFO:teuthology.misc.health.smithi001.stderr:epoch 0 2024-03-25T18:33:24.857 INFO:teuthology.misc.health.smithi001.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T18:33:24.858 INFO:teuthology.misc.health.smithi001.stderr:last_changed 0.000000 2024-03-25T18:33:24.858 INFO:teuthology.misc.health.smithi001.stderr:created 0.000000 2024-03-25T18:33:24.858 INFO:teuthology.misc.health.smithi001.stderr:min_mon_release 0 (unknown) 2024-03-25T18:33:24.858 INFO:teuthology.misc.health.smithi001.stderr:election_strategy: 1 2024-03-25T18:33:24.858 INFO:teuthology.misc.health.smithi001.stderr:0: v1:172.21.15.1:6789/0 mon.noname-a 2024-03-25T18:33:24.858 INFO:teuthology.misc.health.smithi001.stderr:1: v1:172.21.15.115:6789/0 mon.noname-b 2024-03-25T18:33:24.858 INFO:teuthology.misc.health.smithi001.stderr:2: v1:172.21.15.1:6790/0 mon.noname-c 2024-03-25T18:33:24.858 INFO:teuthology.misc.health.smithi001.stderr: 2024-03-25T18:33:24.858 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:33:24.855+0000 7f8b4c44e640 1 librados: starting msgr at 2024-03-25T18:33:24.858 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:33:24.855+0000 7f8b4c44e640 1 librados: starting objecter 2024-03-25T18:33:24.858 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:33:24.855+0000 7f8b4c44e640 1 Processor -- start 2024-03-25T18:33:24.858 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:33:24.855+0000 7f8b4c44e640 1 -- start start 2024-03-25T18:33:24.858 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:33:24.855+0000 7f8b4c44e640 1 librados: setting wanted keys 2024-03-25T18:33:24.858 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:33:24.855+0000 7f8b4c44e640 1 librados: calling monclient init 2024-03-25T18:33:24.859 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:33:24.855+0000 7f8b4c44e640 10 monclient: init 2024-03-25T18:33:24.859 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:33:24.856+0000 7f8b4c44e640 10 monclient: _reopen_session rank -1 2024-03-25T18:33:24.859 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:33:24.856+0000 7f8b4c44e640 10 monclient: _add_conns ranks=[2,0,1] 2024-03-25T18:33:24.859 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:33:24.856+0000 7f8b4c44e640 10 monclient(hunting): picked mon.noname-c con 0x7f8b440b4670 addr v1:172.21.15.1:6790/0 2024-03-25T18:33:24.859 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:33:24.856+0000 7f8b4c44e640 10 monclient(hunting): picked mon.noname-a con 0x7f8b440b4ff0 addr v1:172.21.15.1:6789/0 2024-03-25T18:33:24.859 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:33:24.856+0000 7f8b4c44e640 10 monclient(hunting): picked mon.noname-b con 0x7f8b4413c480 addr v1:172.21.15.115:6789/0 2024-03-25T18:33:24.859 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:33:24.856+0000 7f8b4c44e640 1 -- --> v1:172.21.15.1:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8b440abf50 con 0x7f8b440b4ff0 2024-03-25T18:33:24.859 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:33:24.856+0000 7f8b4c44e640 1 -- --> v1:172.21.15.115:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8b440ac180 con 0x7f8b4413c480 2024-03-25T18:33:24.859 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:33:24.856+0000 7f8b4c44e640 1 -- --> v1:172.21.15.1:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8b440ac460 con 0x7f8b440b4670 2024-03-25T18:33:24.859 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:33:24.856+0000 7f8b4c44e640 10 monclient(hunting): _renew_subs 2024-03-25T18:33:24.859 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:33:24.856+0000 7f8b4c44e640 10 monclient(hunting): authenticate will time out at 7885.326163s 2024-03-25T18:33:24.866 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:33:24.865+0000 7f8b4a9eb640 1 --1- >> v1:172.21.15.115:6789/0 conn(0x7f8b4413c480 0x7f8b440b0ed0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.115:6789/0 says I am v1:172.21.15.1:37668/0 (socket says 172.21.15.1:37668) 2024-03-25T18:33:24.866 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:33:24.865+0000 7f8b4a9eb640 1 -- 172.21.15.1:0/3385340250 learned_addr learned my addr 172.21.15.1:0/3385340250 (peer_addr_for_me v1:172.21.15.1:0/0) 2024-03-25T18:33:24.868 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:33:24.867+0000 7f8b3affd640 10 client.?.objecter ms_handle_connect 0x7f8b4413c480 2024-03-25T18:33:24.868 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:33:24.867+0000 7f8b3affd640 10 client.?.objecter resend_mon_ops 2024-03-25T18:33:24.880 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:33:24.879+0000 7f8b499e9640 1 -- 172.21.15.1:0/3385340250 >> v1:172.21.15.1:6789/0 conn(0x7f8b440b4ff0 legacy=0x7f8b441383f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 17 2024-03-25T18:33:24.880 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:33:24.879+0000 7f8b499e9640 1 -- 172.21.15.1:0/3385340250 >> v1:172.21.15.1:6789/0 conn(0x7f8b440b4ff0 legacy=0x7f8b441383f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-03-25T18:33:24.880 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:33:24.879+0000 7f8b499e9640 1 --1- 172.21.15.1:0/3385340250 >> v1:172.21.15.1:6789/0 conn(0x7f8b440b4ff0 0x7f8b441383f0 :-1 s=CONNECTING_SEND_CONNECT_MSG pgs=0 cs=0 l=1).handle_connect_reply_1 read connect reply failed 2024-03-25T18:33:24.880 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:33:24.879+0000 7f8b499e9640 1 --1- 172.21.15.1:0/3385340250 >> v1:172.21.15.1:6789/0 conn(0x7f8b440b4ff0 0x7f8b441383f0 :-1 s=CONNECTING_SEND_CONNECT_MSG pgs=0 cs=0 l=1).fault on lossy channel, failing 2024-03-25T18:33:24.880 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:33:24.879+0000 7f8b3affd640 10 monclient(hunting): ms_handle_reset hunted mon v1:172.21.15.1:6789/0 2024-03-25T18:33:24.881 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:33:24.880+0000 7f8b3affd640 1 -- 172.21.15.1:0/3385340250 <== mon.1 v1:172.21.15.115:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3086121585 0 0) 0x7f8b440ac180 con 0x7f8b4413c480 2024-03-25T18:33:24.881 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:33:24.880+0000 7f8b3affd640 10 monclient(hunting): _negotiate 2024-03-25T18:33:24.881 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:33:24.880+0000 7f8b3affd640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:33:24.881 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:33:24.880+0000 7f8b3affd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:33:24.881 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:33:24.880+0000 7f8b3affd640 10 monclient(hunting): my global_id is 4525 2024-03-25T18:33:24.881 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:33:24.880+0000 7f8b3affd640 1 -- 172.21.15.1:0/3385340250 --> v1:172.21.15.115:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f8b28004eb0 con 0x7f8b4413c480 2024-03-25T18:33:24.882 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:33:24.881+0000 7f8b3affd640 10 client.?.objecter ms_handle_connect 0x7f8b440b4670 2024-03-25T18:33:24.882 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:33:24.881+0000 7f8b3affd640 10 client.?.objecter resend_mon_ops 2024-03-25T18:33:24.897 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:33:24.896+0000 7f8b3affd640 1 -- 172.21.15.1:0/3385340250 <== mon.2 v1:172.21.15.1:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3887686433 0 0) 0x7f8b440ac460 con 0x7f8b440b4670 2024-03-25T18:33:24.898 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:33:24.896+0000 7f8b3affd640 10 monclient(hunting): _negotiate 2024-03-25T18:33:24.898 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:33:24.896+0000 7f8b3affd640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:33:24.898 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:33:24.896+0000 7f8b3affd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:33:24.898 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:33:24.896+0000 7f8b3affd640 10 monclient(hunting): my global_id is 4526 2024-03-25T18:33:24.898 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:33:24.896+0000 7f8b3affd640 1 -- 172.21.15.1:0/3385340250 --> v1:172.21.15.1:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f8b440ac180 con 0x7f8b440b4670 2024-03-25T18:33:24.898 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:33:24.896+0000 7f8b3affd640 1 -- 172.21.15.1:0/3385340250 <== mon.1 v1:172.21.15.115:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2046442960 0 0) 0x7f8b28004eb0 con 0x7f8b4413c480 2024-03-25T18:33:24.898 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:33:24.896+0000 7f8b3affd640 1 -- 172.21.15.1:0/3385340250 --> v1:172.21.15.115:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f8b440ac460 con 0x7f8b4413c480 2024-03-25T18:33:24.898 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:33:24.896+0000 7f8b3affd640 1 -- 172.21.15.1:0/3385340250 <== mon.1 v1:172.21.15.115:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1586965025 0 0) 0x7f8b3c001ec0 con 0x7f8b4413c480 2024-03-25T18:33:24.898 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:33:24.896+0000 7f8b3affd640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:33:24.898 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:33:24.896+0000 7f8b3affd640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-03-25T18:33:24.898 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:33:24.896+0000 7f8b3affd640 10 monclient(hunting): dump: 2024-03-25T18:33:24.898 INFO:teuthology.misc.health.smithi001.stderr:epoch 1 2024-03-25T18:33:24.898 INFO:teuthology.misc.health.smithi001.stderr:fsid c8ea72cf-9e99-4383-9b52-6c23d365fdfc 2024-03-25T18:33:24.898 INFO:teuthology.misc.health.smithi001.stderr:last_changed 2024-03-25T18:25:50.510519+0000 2024-03-25T18:33:24.898 INFO:teuthology.misc.health.smithi001.stderr:created 2024-03-25T18:25:50.510519+0000 2024-03-25T18:33:24.899 INFO:teuthology.misc.health.smithi001.stderr:min_mon_release 19 (squid) 2024-03-25T18:33:24.899 INFO:teuthology.misc.health.smithi001.stderr:election_strategy: 3 2024-03-25T18:33:24.899 INFO:teuthology.misc.health.smithi001.stderr:0: v1:172.21.15.1:6789/0 mon.a 2024-03-25T18:33:24.899 INFO:teuthology.misc.health.smithi001.stderr:1: v1:172.21.15.115:6789/0 mon.b 2024-03-25T18:33:24.899 INFO:teuthology.misc.health.smithi001.stderr:2: v1:172.21.15.1:6790/0 mon.c 2024-03-25T18:33:24.899 INFO:teuthology.misc.health.smithi001.stderr: 2024-03-25T18:33:24.899 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:33:24.896+0000 7f8b3affd640 10 monclient(hunting): _finish_auth 0 2024-03-25T18:33:24.907 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:33:24.905+0000 7f8b3affd640 1 -- 172.21.15.1:0/3385340250 <== mon.1 v1:172.21.15.115:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 692196152 0 0) 0x7f8b440ac460 con 0x7f8b4413c480 2024-03-25T18:33:24.907 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:33:24.905+0000 7f8b3affd640 1 -- 172.21.15.1:0/3385340250 >> v1:172.21.15.1:6790/0 conn(0x7f8b440b4670 legacy=0x7f8b4413b610 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:33:24.907 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:33:24.905+0000 7f8b3affd640 1 -- 172.21.15.1:0/3385340250 >> v1:172.21.15.1:6789/0 conn(0x7f8b440b4ff0 legacy=0x7f8b441383f0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-03-25T18:33:24.907 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:33:24.905+0000 7f8b3affd640 10 monclient: _finish_hunting 0 2024-03-25T18:33:24.907 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:33:24.905+0000 7f8b3affd640 1 monclient: found mon.b 2024-03-25T18:33:24.907 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:33:24.905+0000 7f8b3affd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:33:24.907 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:33:24.905+0000 7f8b3affd640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.115:6789/0 2024-03-25T18:33:24.907 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:33:24.905+0000 7f8b3affd640 1 -- 172.21.15.1:0/3385340250 --> v1:172.21.15.115:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8b44138b00 con 0x7f8b4413c480 2024-03-25T18:33:24.908 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:33:24.905+0000 7f8b3affd640 10 monclient: _finish_auth 0 2024-03-25T18:33:24.908 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:33:24.905+0000 7f8b3affd640 10 monclient: _check_auth_tickets 2024-03-25T18:33:24.908 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:33:24.905+0000 7f8b3affd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:33:24.908 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:33:24.905+0000 7f8b4c44e640 5 monclient: authenticate success, global_id 4525 2024-03-25T18:33:24.908 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:33:24.905+0000 7f8b4c44e640 10 monclient: _renew_subs 2024-03-25T18:33:24.908 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:33:24.905+0000 7f8b4c44e640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.115:6789/0 2024-03-25T18:33:24.908 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:33:24.905+0000 7f8b4c44e640 1 -- 172.21.15.1:0/3385340250 --> v1:172.21.15.115:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8b4413c000 con 0x7f8b4413c480 2024-03-25T18:33:24.908 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:33:24.905+0000 7f8b4c44e640 10 client.4525.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T18:33:24.908 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:33:24.905+0000 7f8b4c44e640 10 monclient: _renew_subs 2024-03-25T18:33:24.908 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:33:24.905+0000 7f8b4c44e640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.115:6789/0 2024-03-25T18:33:24.908 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:33:24.905+0000 7f8b4c44e640 1 -- 172.21.15.1:0/3385340250 --> v1:172.21.15.115:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f8b440b17c0 con 0x7f8b4413c480 2024-03-25T18:33:24.908 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:33:24.906+0000 7f8b4c44e640 1 librados: init done 2024-03-25T18:33:24.909 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:33:24.907+0000 7f8b4c44e640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T18:33:24.909 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:33:24.907+0000 7f8b4c44e640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:33:24.909 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:33:24.907+0000 7f8b4c44e640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.115:6789/0 2024-03-25T18:33:24.909 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:33:24.907+0000 7f8b4c44e640 1 -- 172.21.15.1:0/3385340250 --> v1:172.21.15.115:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f8b44138d50 con 0x7f8b4413c480 2024-03-25T18:33:25.185 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:33:25.184+0000 7f8b3affd640 1 -- 172.21.15.1:0/3385340250 <== mon.1 v1:172.21.15.115:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f8b3c0020e0 con 0x7f8b4413c480 2024-03-25T18:33:25.185 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:33:25.184+0000 7f8b3affd640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:33:25.185 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:33:25.184+0000 7f8b3affd640 1 -- 172.21.15.1:0/3385340250 <== mon.1 v1:172.21.15.115:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1586965025 0 0) 0x7f8b3c004e10 con 0x7f8b4413c480 2024-03-25T18:33:25.185 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:33:25.184+0000 7f8b3affd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:33:25.185 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:33:25.184+0000 7f8b3affd640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-25T18:33:25.186 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:33:25.184+0000 7f8b3affd640 10 monclient: dump: 2024-03-25T18:33:25.186 INFO:teuthology.misc.health.smithi001.stderr:epoch 1 2024-03-25T18:33:25.186 INFO:teuthology.misc.health.smithi001.stderr:fsid c8ea72cf-9e99-4383-9b52-6c23d365fdfc 2024-03-25T18:33:25.186 INFO:teuthology.misc.health.smithi001.stderr:last_changed 2024-03-25T18:25:50.510519+0000 2024-03-25T18:33:25.186 INFO:teuthology.misc.health.smithi001.stderr:created 2024-03-25T18:25:50.510519+0000 2024-03-25T18:33:25.186 INFO:teuthology.misc.health.smithi001.stderr:min_mon_release 19 (squid) 2024-03-25T18:33:25.186 INFO:teuthology.misc.health.smithi001.stderr:election_strategy: 3 2024-03-25T18:33:25.186 INFO:teuthology.misc.health.smithi001.stderr:0: v1:172.21.15.1:6789/0 mon.a 2024-03-25T18:33:25.186 INFO:teuthology.misc.health.smithi001.stderr:1: v1:172.21.15.115:6789/0 mon.b 2024-03-25T18:33:25.186 INFO:teuthology.misc.health.smithi001.stderr:2: v1:172.21.15.1:6790/0 mon.c 2024-03-25T18:33:25.186 INFO:teuthology.misc.health.smithi001.stderr: 2024-03-25T18:33:25.186 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:33:25.184+0000 7f8b489e7640 4 set_mon_vals no callback set 2024-03-25T18:33:25.187 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:33:25.185+0000 7f8b3affd640 1 -- 172.21.15.1:0/3385340250 <== mon.1 v1:172.21.15.115:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 21787975 0 0) 0x7f8b3c01ade0 con 0x7f8b4413c480 2024-03-25T18:33:25.187 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:33:25.186+0000 7f8b4a1ea640 10 monclient: get_auth_request con 0x7f8b2806dcb0 auth_method 0 2024-03-25T18:33:25.187 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:33:25.186+0000 7f8b3affd640 10 client.4525.objecter ms_handle_connect 0x7f8b2806dcb0 2024-03-25T18:33:25.190 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:33:25.190+0000 7f8b3affd640 1 -- 172.21.15.1:0/3385340250 <== mon.1 v1:172.21.15.115:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 996254230 0 0) 0x7f8b3c054990 con 0x7f8b4413c480 2024-03-25T18:33:25.191 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:33:25.190+0000 7f8b3affd640 10 client.4525.objecter ms_dispatch 0x7f8b44000b90 osd_map(10..10 src has 1..10) v4 2024-03-25T18:33:25.191 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:33:25.190+0000 7f8b3affd640 3 client.4525.objecter handle_osd_map got epochs [10,10] > 0 2024-03-25T18:33:25.191 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:33:25.190+0000 7f8b3affd640 3 client.4525.objecter handle_osd_map decoding full epoch 10 2024-03-25T18:33:25.191 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:33:25.190+0000 7f8b3affd640 20 client.4525.objecter dump_active .. 0 homeless 2024-03-25T18:33:25.191 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:33:25.190+0000 7f8b3affd640 1 -- 172.21.15.1:0/3385340250 <== mon.1 v1:172.21.15.115:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f8b3c0502d0 con 0x7f8b4413c480 2024-03-25T18:33:25.191 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:33:25.190+0000 7f8b3affd640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:33:25.191 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:33:25.190+0000 7f8b3affd640 10 monclient: _finish_command 1 = system:0 2024-03-25T18:33:25.447 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:33:25.446+0000 7f8b4c44e640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:33:25.447 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:33:25.446+0000 7f8b4c44e640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:33:25.447 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:33:25.446+0000 7f8b4c44e640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.115:6789/0 2024-03-25T18:33:25.447 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:33:25.446+0000 7f8b4c44e640 1 -- 172.21.15.1:0/3385340250 --> v1:172.21.15.115:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f8b4406dbd0 con 0x7f8b4413c480 2024-03-25T18:33:25.472 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:33:25.471+0000 7f8b3affd640 1 -- 172.21.15.1:0/3385340250 <== mon.1 v1:172.21.15.115:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 4092213348) 0x7f8b3c0523d0 con 0x7f8b4413c480 2024-03-25T18:33:25.472 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:33:25.471+0000 7f8b3affd640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:33:25.472 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:33:25.471+0000 7f8b3affd640 10 monclient: _finish_command 2 = system:0 2024-03-25T18:33:25.472 INFO:teuthology.misc.health.smithi001.stdout: 2024-03-25T18:33:25.472 INFO:teuthology.misc.health.smithi001.stdout:{"epoch":10,"fsid":"c8ea72cf-9e99-4383-9b52-6c23d365fdfc","created":"2024-03-25T18:26:38.856638+0000","modified":"2024-03-25T18:26:44.224208+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T18:26:44.224208+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"b9f34922-3a84-49be-8426-807c1eceaefc","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"42bf306f-55e3-4634-9874-8b4b73ac6644","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"292cb23a-18a9-4207-a7cc-4d5727feaec0","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"45e0e9ab-898d-4673-9590-b4117bc5e84a","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"831a5a09-73ab-4a61-9872-088e76bdbc66","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"88cd903d-3b55-4840-9396-6e3cfe005282","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"89db7355-bc9d-46f4-8ab0-e8d9e744e026","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"cac47bee-0914-4f08-896f-348838882866","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T18:33:25.475 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:33:25.474+0000 7f8b4c44e640 10 librados: watch_flush enter 2024-03-25T18:33:25.475 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:33:25.474+0000 7f8b4c44e640 10 librados: watch_flush exit 2024-03-25T18:33:25.476 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:33:25.474+0000 7f8b4c44e640 20 client.4525.objecter shutdown clearing up homeless session... 2024-03-25T18:33:25.476 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:33:25.474+0000 7f8b4c44e640 10 client.4525.objecter successfully canceled tick 2024-03-25T18:33:25.476 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:33:25.474+0000 7f8b4c44e640 1 -- 172.21.15.1:0/3385340250 >> v1:172.21.15.115:6800/4237379176 conn(0x7f8b2806dcb0 legacy=0x7f8b28070170 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:33:25.476 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:33:25.474+0000 7f8b4c44e640 10 monclient: shutdown 2024-03-25T18:33:25.476 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:33:25.474+0000 7f8b4c44e640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:33:25.476 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:33:25.474+0000 7f8b4c44e640 1 -- 172.21.15.1:0/3385340250 >> v1:172.21.15.115:6789/0 conn(0x7f8b4413c480 legacy=0x7f8b440b0ed0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:33:25.476 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:33:25.474+0000 7f8b4c44e640 1 -- 172.21.15.1:0/3385340250 shutdown_connections 2024-03-25T18:33:25.476 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:33:25.474+0000 7f8b4c44e640 1 -- 172.21.15.1:0/3385340250 >> 172.21.15.1:0/3385340250 conn(0x7f8b440617a0 msgr2=0x7f8b44145540 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:33:25.476 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:33:25.474+0000 7f8b4c44e640 1 -- 172.21.15.1:0/3385340250 shutdown_connections 2024-03-25T18:33:25.476 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:33:25.474+0000 7f8b4c44e640 1 -- 172.21.15.1:0/3385340250 wait complete. 2024-03-25T18:33:25.476 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:33:25.475+0000 7f8b4c44e640 1 librados: shutdown 2024-03-25T18:33:25.487 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T18:33:32.293 DEBUG:teuthology.orchestra.run.smithi001:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T18:33:32.410 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:33:32.409+0000 7fac0b282640 10 monclient: get_monmap_and_config 2024-03-25T18:33:32.410 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:33:32.409+0000 7fac0b282640 10 monclient: build_initial_monmap 2024-03-25T18:33:32.410 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:33:32.409+0000 7fac0b282640 10 monclient: monmap: 2024-03-25T18:33:32.410 INFO:teuthology.misc.health.smithi001.stderr:epoch 0 2024-03-25T18:33:32.410 INFO:teuthology.misc.health.smithi001.stderr:fsid c8ea72cf-9e99-4383-9b52-6c23d365fdfc 2024-03-25T18:33:32.410 INFO:teuthology.misc.health.smithi001.stderr:last_changed 2024-03-25T18:33:32.409751+0000 2024-03-25T18:33:32.410 INFO:teuthology.misc.health.smithi001.stderr:created 2024-03-25T18:33:32.409751+0000 2024-03-25T18:33:32.410 INFO:teuthology.misc.health.smithi001.stderr:min_mon_release 0 (unknown) 2024-03-25T18:33:32.410 INFO:teuthology.misc.health.smithi001.stderr:election_strategy: 3 2024-03-25T18:33:32.410 INFO:teuthology.misc.health.smithi001.stderr:0: v1:172.21.15.1:6789/0 mon.noname-a 2024-03-25T18:33:32.411 INFO:teuthology.misc.health.smithi001.stderr:1: v1:172.21.15.115:6789/0 mon.noname-c 2024-03-25T18:33:32.411 INFO:teuthology.misc.health.smithi001.stderr:2: v1:172.21.15.1:6790/0 mon.noname-b 2024-03-25T18:33:32.411 INFO:teuthology.misc.health.smithi001.stderr: 2024-03-25T18:33:32.412 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:33:32.409+0000 7fac0b282640 1 Processor -- start 2024-03-25T18:33:32.412 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:33:32.410+0000 7fac0b282640 1 -- start start 2024-03-25T18:33:32.412 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:33:32.410+0000 7fac0b282640 10 monclient: init 2024-03-25T18:33:32.412 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:33:32.411+0000 7fac0b282640 10 monclient: _reopen_session rank -1 2024-03-25T18:33:32.412 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:33:32.411+0000 7fac0b282640 10 monclient: _add_conns ranks=[2,1,0] 2024-03-25T18:33:32.412 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:33:32.411+0000 7fac0b282640 10 monclient(hunting): picked mon.noname-b con 0x7fac040b4e70 addr v1:172.21.15.1:6790/0 2024-03-25T18:33:32.412 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:33:32.411+0000 7fac0b282640 10 monclient(hunting): picked mon.noname-c con 0x7fac040b44b0 addr v1:172.21.15.115:6789/0 2024-03-25T18:33:32.412 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:33:32.411+0000 7fac0b282640 10 monclient(hunting): picked mon.noname-a con 0x7fac040b1b00 addr v1:172.21.15.1:6789/0 2024-03-25T18:33:32.412 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:33:32.411+0000 7fac0b282640 1 -- --> v1:172.21.15.1:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fac040a96f0 con 0x7fac040b1b00 2024-03-25T18:33:32.412 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:33:32.411+0000 7fac0b282640 1 -- --> v1:172.21.15.115:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fac040ab0e0 con 0x7fac040b44b0 2024-03-25T18:33:32.413 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:33:32.411+0000 7fac0b282640 1 -- --> v1:172.21.15.1:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fac04145f70 con 0x7fac040b4e70 2024-03-25T18:33:32.413 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:33:32.411+0000 7fac0b282640 10 monclient(hunting): _renew_subs 2024-03-25T18:33:32.413 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:33:32.411+0000 7fac0b282640 10 monclient(hunting): authenticate will time out at 7892.880882s 2024-03-25T18:33:32.435 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:33:32.434+0000 7fac0881d640 1 --1- >> v1:172.21.15.115:6789/0 conn(0x7fac040b44b0 0x7fac04145860 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.115:6789/0 says I am v1:172.21.15.1:49512/0 (socket says 172.21.15.1:49512) 2024-03-25T18:33:32.435 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:33:32.434+0000 7fac0881d640 1 -- 172.21.15.1:0/72849706 learned_addr learned my addr 172.21.15.1:0/72849706 (peer_addr_for_me v1:172.21.15.1:0/0) 2024-03-25T18:33:32.450 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:33:32.449+0000 7fabfbfff640 1 -- 172.21.15.1:0/72849706 <== mon.1 v1:172.21.15.115:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 593213688 0 0) 0x7fac040ab0e0 con 0x7fac040b44b0 2024-03-25T18:33:32.450 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:33:32.449+0000 7fabfbfff640 10 monclient(hunting): _negotiate 2024-03-25T18:33:32.450 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:33:32.449+0000 7fabfbfff640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:33:32.451 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:33:32.449+0000 7fabfbfff640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:33:32.451 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:33:32.449+0000 7fabfbfff640 10 monclient(hunting): my global_id is 4528 2024-03-25T18:33:32.451 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:33:32.449+0000 7fabfbfff640 1 -- 172.21.15.1:0/72849706 --> v1:172.21.15.115:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fabf0003520 con 0x7fac040b44b0 2024-03-25T18:33:32.453 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:33:32.452+0000 7fabfbfff640 1 -- 172.21.15.1:0/72849706 <== mon.0 v1:172.21.15.1:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 183926988 0 0) 0x7fac040a96f0 con 0x7fac040b1b00 2024-03-25T18:33:32.453 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:33:32.452+0000 7fabfbfff640 10 monclient(hunting): _negotiate 2024-03-25T18:33:32.454 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:33:32.452+0000 7fabfbfff640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:33:32.454 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:33:32.452+0000 7fabfbfff640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:33:32.454 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:33:32.452+0000 7fabfbfff640 10 monclient(hunting): my global_id is 4512 2024-03-25T18:33:32.454 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:33:32.452+0000 7fabfbfff640 1 -- 172.21.15.1:0/72849706 --> v1:172.21.15.1:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fac040ab0e0 con 0x7fac040b1b00 2024-03-25T18:33:32.454 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:33:32.453+0000 7fabfbfff640 1 -- 172.21.15.1:0/72849706 <== mon.2 v1:172.21.15.1:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3751116460 0 0) 0x7fac04145f70 con 0x7fac040b4e70 2024-03-25T18:33:32.454 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:33:32.453+0000 7fabfbfff640 10 monclient(hunting): _negotiate 2024-03-25T18:33:32.454 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:33:32.453+0000 7fabfbfff640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:33:32.454 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:33:32.453+0000 7fabfbfff640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:33:32.454 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:33:32.453+0000 7fabfbfff640 10 monclient(hunting): my global_id is 4529 2024-03-25T18:33:32.454 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:33:32.453+0000 7fabfbfff640 1 -- 172.21.15.1:0/72849706 --> v1:172.21.15.1:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fac040a96f0 con 0x7fac040b4e70 2024-03-25T18:33:32.468 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:33:32.467+0000 7fabfbfff640 1 -- 172.21.15.1:0/72849706 <== mon.1 v1:172.21.15.115:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2346230908 0 0) 0x7fabf0003520 con 0x7fac040b44b0 2024-03-25T18:33:32.469 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:33:32.467+0000 7fabfbfff640 1 -- 172.21.15.1:0/72849706 >> v1:172.21.15.1:6790/0 conn(0x7fac040b4e70 legacy=0x7fac0413f940 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:33:32.469 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:33:32.467+0000 7fabfbfff640 1 -- 172.21.15.1:0/72849706 >> v1:172.21.15.1:6789/0 conn(0x7fac040b1b00 legacy=0x7fac0414dc00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:33:32.469 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:33:32.467+0000 7fabfbfff640 10 monclient: _finish_hunting 0 2024-03-25T18:33:32.469 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:33:32.467+0000 7fabfbfff640 1 monclient: found mon.noname-c 2024-03-25T18:33:32.469 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:33:32.467+0000 7fabfbfff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:33:32.469 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:33:32.467+0000 7fabfbfff640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.115:6789/0 2024-03-25T18:33:32.469 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:33:32.467+0000 7fabfbfff640 1 -- 172.21.15.1:0/72849706 --> v1:172.21.15.115:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fac040a9b80 con 0x7fac040b44b0 2024-03-25T18:33:32.469 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:33:32.467+0000 7fabfbfff640 10 monclient: _finish_auth 0 2024-03-25T18:33:32.469 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:33:32.467+0000 7fabfbfff640 10 monclient: _check_auth_tickets 2024-03-25T18:33:32.469 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:33:32.467+0000 7fabfbfff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:33:32.469 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:33:32.467+0000 7fabfbfff640 1 -- 172.21.15.1:0/72849706 <== mon.1 v1:172.21.15.115:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1586965025 0 0) 0x7fabf4001690 con 0x7fac040b44b0 2024-03-25T18:33:32.469 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:33:32.467+0000 7fabfbfff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:33:32.469 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:33:32.467+0000 7fabfbfff640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-03-25T18:33:32.469 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:33:32.467+0000 7fabfbfff640 10 monclient: dump: 2024-03-25T18:33:32.469 INFO:teuthology.misc.health.smithi001.stderr:epoch 1 2024-03-25T18:33:32.470 INFO:teuthology.misc.health.smithi001.stderr:fsid c8ea72cf-9e99-4383-9b52-6c23d365fdfc 2024-03-25T18:33:32.470 INFO:teuthology.misc.health.smithi001.stderr:last_changed 2024-03-25T18:25:50.510519+0000 2024-03-25T18:33:32.470 INFO:teuthology.misc.health.smithi001.stderr:created 2024-03-25T18:25:50.510519+0000 2024-03-25T18:33:32.470 INFO:teuthology.misc.health.smithi001.stderr:min_mon_release 19 (squid) 2024-03-25T18:33:32.470 INFO:teuthology.misc.health.smithi001.stderr:election_strategy: 3 2024-03-25T18:33:32.470 INFO:teuthology.misc.health.smithi001.stderr:0: v1:172.21.15.1:6789/0 mon.a 2024-03-25T18:33:32.470 INFO:teuthology.misc.health.smithi001.stderr:1: v1:172.21.15.115:6789/0 mon.b 2024-03-25T18:33:32.470 INFO:teuthology.misc.health.smithi001.stderr:2: v1:172.21.15.1:6790/0 mon.c 2024-03-25T18:33:32.470 INFO:teuthology.misc.health.smithi001.stderr: 2024-03-25T18:33:32.470 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:33:32.467+0000 7fac0b282640 5 monclient: authenticate success, global_id 4528 2024-03-25T18:33:32.470 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:33:32.467+0000 7fac0b282640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T18:33:32.494 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:33:32.493+0000 7fabfbfff640 1 -- 172.21.15.1:0/72849706 <== mon.1 v1:172.21.15.115:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fabf40017f0 con 0x7fac040b44b0 2024-03-25T18:33:32.494 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:33:32.493+0000 7fabfbfff640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:33:32.494 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:33:32.493+0000 7fabfbfff640 1 -- 172.21.15.1:0/72849706 <== mon.1 v1:172.21.15.115:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1586965025 0 0) 0x7fabf4001c00 con 0x7fac040b44b0 2024-03-25T18:33:32.494 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:33:32.493+0000 7fabfbfff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:33:32.495 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:33:32.493+0000 7fabfbfff640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-25T18:33:32.495 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:33:32.493+0000 7fabfbfff640 10 monclient: dump: 2024-03-25T18:33:32.495 INFO:teuthology.misc.health.smithi001.stderr:epoch 1 2024-03-25T18:33:32.495 INFO:teuthology.misc.health.smithi001.stderr:fsid c8ea72cf-9e99-4383-9b52-6c23d365fdfc 2024-03-25T18:33:32.495 INFO:teuthology.misc.health.smithi001.stderr:last_changed 2024-03-25T18:25:50.510519+0000 2024-03-25T18:33:32.495 INFO:teuthology.misc.health.smithi001.stderr:created 2024-03-25T18:25:50.510519+0000 2024-03-25T18:33:32.495 INFO:teuthology.misc.health.smithi001.stderr:min_mon_release 19 (squid) 2024-03-25T18:33:32.495 INFO:teuthology.misc.health.smithi001.stderr:election_strategy: 3 2024-03-25T18:33:32.495 INFO:teuthology.misc.health.smithi001.stderr:0: v1:172.21.15.1:6789/0 mon.a 2024-03-25T18:33:32.495 INFO:teuthology.misc.health.smithi001.stderr:1: v1:172.21.15.115:6789/0 mon.b 2024-03-25T18:33:32.495 INFO:teuthology.misc.health.smithi001.stderr:2: v1:172.21.15.1:6790/0 mon.c 2024-03-25T18:33:32.495 INFO:teuthology.misc.health.smithi001.stderr: 2024-03-25T18:33:32.495 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:33:32.493+0000 7fac0b282640 10 monclient: get_monmap_and_config success 2024-03-25T18:33:32.495 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:33:32.493+0000 7fac0b282640 4 set_mon_vals no callback set 2024-03-25T18:33:32.497 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:33:32.494+0000 7fac0b282640 10 monclient: shutdown 2024-03-25T18:33:32.497 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:33:32.494+0000 7fac0b282640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:33:32.498 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:33:32.494+0000 7fac0b282640 1 -- 172.21.15.1:0/72849706 >> v1:172.21.15.115:6789/0 conn(0x7fac040b44b0 legacy=0x7fac04145860 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:33:32.498 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:33:32.494+0000 7fac0b282640 1 -- 172.21.15.1:0/72849706 shutdown_connections 2024-03-25T18:33:32.498 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:33:32.494+0000 7fac0b282640 1 -- 172.21.15.1:0/72849706 >> 172.21.15.1:0/72849706 conn(0x7fac040617a0 msgr2=0x7fac04061bb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:33:32.498 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:33:32.494+0000 7fac0b282640 1 -- 172.21.15.1:0/72849706 shutdown_connections 2024-03-25T18:33:32.498 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:33:32.494+0000 7fac0b282640 1 -- 172.21.15.1:0/72849706 wait complete. 2024-03-25T18:33:32.498 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:33:32.495+0000 7fac0b282640 10 monclient: build_initial_monmap 2024-03-25T18:33:32.498 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:33:32.495+0000 7fac0b282640 10 monclient: monmap: 2024-03-25T18:33:32.498 INFO:teuthology.misc.health.smithi001.stderr:epoch 0 2024-03-25T18:33:32.498 INFO:teuthology.misc.health.smithi001.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T18:33:32.498 INFO:teuthology.misc.health.smithi001.stderr:last_changed 0.000000 2024-03-25T18:33:32.498 INFO:teuthology.misc.health.smithi001.stderr:created 0.000000 2024-03-25T18:33:32.498 INFO:teuthology.misc.health.smithi001.stderr:min_mon_release 0 (unknown) 2024-03-25T18:33:32.498 INFO:teuthology.misc.health.smithi001.stderr:election_strategy: 1 2024-03-25T18:33:32.498 INFO:teuthology.misc.health.smithi001.stderr:0: v1:172.21.15.1:6789/0 mon.noname-a 2024-03-25T18:33:32.499 INFO:teuthology.misc.health.smithi001.stderr:1: v1:172.21.15.115:6789/0 mon.noname-b 2024-03-25T18:33:32.499 INFO:teuthology.misc.health.smithi001.stderr:2: v1:172.21.15.1:6790/0 mon.noname-c 2024-03-25T18:33:32.499 INFO:teuthology.misc.health.smithi001.stderr: 2024-03-25T18:33:32.499 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:33:32.495+0000 7fac0b282640 1 librados: starting msgr at 2024-03-25T18:33:32.499 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:33:32.495+0000 7fac0b282640 1 librados: starting objecter 2024-03-25T18:33:32.499 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:33:32.496+0000 7fac0b282640 1 Processor -- start 2024-03-25T18:33:32.499 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:33:32.496+0000 7fac0b282640 1 -- start start 2024-03-25T18:33:32.499 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:33:32.496+0000 7fac0b282640 1 librados: setting wanted keys 2024-03-25T18:33:32.499 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:33:32.496+0000 7fac0b282640 1 librados: calling monclient init 2024-03-25T18:33:32.499 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:33:32.496+0000 7fac0b282640 10 monclient: init 2024-03-25T18:33:32.499 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:33:32.496+0000 7fac0b282640 10 monclient: _reopen_session rank -1 2024-03-25T18:33:32.499 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:33:32.496+0000 7fac0b282640 10 monclient: _add_conns ranks=[2,1,0] 2024-03-25T18:33:32.499 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:33:32.496+0000 7fac0b282640 10 monclient(hunting): picked mon.noname-c con 0x7fac040b1b00 addr v1:172.21.15.1:6790/0 2024-03-25T18:33:32.499 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:33:32.496+0000 7fac0b282640 10 monclient(hunting): picked mon.noname-b con 0x7fac040b44b0 addr v1:172.21.15.115:6789/0 2024-03-25T18:33:32.499 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:33:32.496+0000 7fac0b282640 10 monclient(hunting): picked mon.noname-a con 0x7fac040b4e70 addr v1:172.21.15.1:6789/0 2024-03-25T18:33:32.499 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:33:32.496+0000 7fac0b282640 1 -- --> v1:172.21.15.1:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fac04144e20 con 0x7fac040b4e70 2024-03-25T18:33:32.499 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:33:32.496+0000 7fac0b282640 1 -- --> v1:172.21.15.115:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fac041452b0 con 0x7fac040b44b0 2024-03-25T18:33:32.500 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:33:32.496+0000 7fac0b282640 1 -- --> v1:172.21.15.1:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fac0413c150 con 0x7fac040b1b00 2024-03-25T18:33:32.500 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:33:32.496+0000 7fac0b282640 10 monclient(hunting): _renew_subs 2024-03-25T18:33:32.500 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:33:32.496+0000 7fac0b282640 10 monclient(hunting): authenticate will time out at 7892.966512s 2024-03-25T18:33:32.507 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:33:32.507+0000 7fac0901e640 1 --1- >> v1:172.21.15.1:6790/0 conn(0x7fac040b1b00 0x7fac0413a600 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.1:6790/0 says I am v1:172.21.15.1:41172/0 (socket says 172.21.15.1:41172) 2024-03-25T18:33:32.508 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:33:32.507+0000 7fac0901e640 1 -- 172.21.15.1:0/1612919699 learned_addr learned my addr 172.21.15.1:0/1612919699 (peer_addr_for_me v1:172.21.15.1:0/0) 2024-03-25T18:33:32.509 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:33:32.508+0000 7fabf9ffb640 10 client.?.objecter ms_handle_connect 0x7fac040b1b00 2024-03-25T18:33:32.509 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:33:32.508+0000 7fabf9ffb640 10 client.?.objecter resend_mon_ops 2024-03-25T18:33:32.512 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:33:32.511+0000 7fabf9ffb640 10 client.?.objecter ms_handle_connect 0x7fac040b44b0 2024-03-25T18:33:32.512 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:33:32.511+0000 7fabf9ffb640 10 client.?.objecter resend_mon_ops 2024-03-25T18:33:32.519 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:33:32.519+0000 7fabf9ffb640 1 -- 172.21.15.1:0/1612919699 <== mon.2 v1:172.21.15.1:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2142954326 0 0) 0x7fac0413c150 con 0x7fac040b1b00 2024-03-25T18:33:32.520 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:33:32.519+0000 7fabf9ffb640 10 monclient(hunting): _negotiate 2024-03-25T18:33:32.520 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:33:32.519+0000 7fabf9ffb640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:33:32.520 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:33:32.519+0000 7fabf9ffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:33:32.520 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:33:32.519+0000 7fabf9ffb640 10 monclient(hunting): my global_id is 4532 2024-03-25T18:33:32.520 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:33:32.519+0000 7fabf9ffb640 1 -- 172.21.15.1:0/1612919699 --> v1:172.21.15.1:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fabe8004eb0 con 0x7fac040b1b00 2024-03-25T18:33:32.526 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:33:32.526+0000 7fabf9ffb640 1 -- 172.21.15.1:0/1612919699 <== mon.1 v1:172.21.15.115:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2661366487 0 0) 0x7fac041452b0 con 0x7fac040b44b0 2024-03-25T18:33:32.527 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:33:32.526+0000 7fabf9ffb640 10 monclient(hunting): _negotiate 2024-03-25T18:33:32.527 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:33:32.526+0000 7fabf9ffb640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:33:32.527 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:33:32.526+0000 7fabf9ffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:33:32.527 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:33:32.526+0000 7fabf9ffb640 10 monclient(hunting): my global_id is 4531 2024-03-25T18:33:32.527 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:33:32.526+0000 7fabf9ffb640 1 -- 172.21.15.1:0/1612919699 --> v1:172.21.15.115:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fac0413c150 con 0x7fac040b44b0 2024-03-25T18:33:32.527 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:33:32.526+0000 7fabf9ffb640 10 client.?.objecter ms_handle_connect 0x7fac040b4e70 2024-03-25T18:33:32.527 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:33:32.526+0000 7fabf9ffb640 10 client.?.objecter resend_mon_ops 2024-03-25T18:33:32.533 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:33:32.532+0000 7fabf9ffb640 1 -- 172.21.15.1:0/1612919699 <== mon.2 v1:172.21.15.1:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 4182371580 0 0) 0x7fabe8004eb0 con 0x7fac040b1b00 2024-03-25T18:33:32.533 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:33:32.532+0000 7fabf9ffb640 1 -- 172.21.15.1:0/1612919699 --> v1:172.21.15.1:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fac041452b0 con 0x7fac040b1b00 2024-03-25T18:33:32.533 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:33:32.532+0000 7fabf9ffb640 1 -- 172.21.15.1:0/1612919699 <== mon.2 v1:172.21.15.1:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1586965025 0 0) 0x7fac00001970 con 0x7fac040b1b00 2024-03-25T18:33:32.534 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:33:32.532+0000 7fabf9ffb640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:33:32.534 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:33:32.532+0000 7fabf9ffb640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-03-25T18:33:32.534 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:33:32.532+0000 7fabf9ffb640 10 monclient(hunting): dump: 2024-03-25T18:33:32.534 INFO:teuthology.misc.health.smithi001.stderr:epoch 1 2024-03-25T18:33:32.534 INFO:teuthology.misc.health.smithi001.stderr:fsid c8ea72cf-9e99-4383-9b52-6c23d365fdfc 2024-03-25T18:33:32.534 INFO:teuthology.misc.health.smithi001.stderr:last_changed 2024-03-25T18:25:50.510519+0000 2024-03-25T18:33:32.534 INFO:teuthology.misc.health.smithi001.stderr:created 2024-03-25T18:25:50.510519+0000 2024-03-25T18:33:32.534 INFO:teuthology.misc.health.smithi001.stderr:min_mon_release 19 (squid) 2024-03-25T18:33:32.534 INFO:teuthology.misc.health.smithi001.stderr:election_strategy: 3 2024-03-25T18:33:32.534 INFO:teuthology.misc.health.smithi001.stderr:0: v1:172.21.15.1:6789/0 mon.a 2024-03-25T18:33:32.534 INFO:teuthology.misc.health.smithi001.stderr:1: v1:172.21.15.115:6789/0 mon.b 2024-03-25T18:33:32.534 INFO:teuthology.misc.health.smithi001.stderr:2: v1:172.21.15.1:6790/0 mon.c 2024-03-25T18:33:32.534 INFO:teuthology.misc.health.smithi001.stderr: 2024-03-25T18:33:32.534 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:33:32.532+0000 7fabf9ffb640 10 monclient(hunting): _finish_auth 0 2024-03-25T18:33:32.537 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:33:32.536+0000 7fabf9ffb640 1 -- 172.21.15.1:0/1612919699 <== mon.0 v1:172.21.15.1:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4030654538 0 0) 0x7fac04144e20 con 0x7fac040b4e70 2024-03-25T18:33:32.537 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:33:32.536+0000 7fabf9ffb640 10 monclient(hunting): _negotiate 2024-03-25T18:33:32.537 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:33:32.536+0000 7fabf9ffb640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:33:32.537 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:33:32.536+0000 7fabf9ffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:33:32.537 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:33:32.536+0000 7fabf9ffb640 10 monclient(hunting): my global_id is 4515 2024-03-25T18:33:32.537 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:33:32.536+0000 7fabf9ffb640 1 -- 172.21.15.1:0/1612919699 --> v1:172.21.15.1:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fabe8004eb0 con 0x7fac040b4e70 2024-03-25T18:33:32.538 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:33:32.537+0000 7fabf9ffb640 1 -- 172.21.15.1:0/1612919699 <== mon.2 v1:172.21.15.1:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 1442884710 0 0) 0x7fac041452b0 con 0x7fac040b1b00 2024-03-25T18:33:32.538 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:33:32.537+0000 7fabf9ffb640 1 -- 172.21.15.1:0/1612919699 >> v1:172.21.15.115:6789/0 conn(0x7fac040b44b0 legacy=0x7fac0413ae40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:33:32.539 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:33:32.537+0000 7fabf9ffb640 1 -- 172.21.15.1:0/1612919699 >> v1:172.21.15.1:6789/0 conn(0x7fac040b4e70 legacy=0x7fac041e1f10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:33:32.539 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:33:32.537+0000 7fabf9ffb640 10 monclient: _finish_hunting 0 2024-03-25T18:33:32.539 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:33:32.537+0000 7fabf9ffb640 1 monclient: found mon.c 2024-03-25T18:33:32.539 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:33:32.537+0000 7fabf9ffb640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:33:32.539 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:33:32.537+0000 7fabf9ffb640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.1:6790/0 2024-03-25T18:33:32.539 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:33:32.537+0000 7fabf9ffb640 1 -- 172.21.15.1:0/1612919699 --> v1:172.21.15.1:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fac0413c430 con 0x7fac040b1b00 2024-03-25T18:33:32.539 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:33:32.537+0000 7fabf9ffb640 10 monclient: _finish_auth 0 2024-03-25T18:33:32.539 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:33:32.537+0000 7fabf9ffb640 10 monclient: _check_auth_tickets 2024-03-25T18:33:32.539 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:33:32.537+0000 7fabf9ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:33:32.539 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:33:32.537+0000 7fac0b282640 5 monclient: authenticate success, global_id 4532 2024-03-25T18:33:32.539 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:33:32.537+0000 7fac0b282640 10 monclient: _renew_subs 2024-03-25T18:33:32.539 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:33:32.537+0000 7fac0b282640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.1:6790/0 2024-03-25T18:33:32.539 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:33:32.537+0000 7fac0b282640 1 -- 172.21.15.1:0/1612919699 --> v1:172.21.15.1:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fac0413c710 con 0x7fac040b1b00 2024-03-25T18:33:32.539 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:33:32.537+0000 7fac0b282640 10 client.4532.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T18:33:32.539 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:33:32.537+0000 7fac0b282640 10 monclient: _renew_subs 2024-03-25T18:33:32.539 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:33:32.537+0000 7fac0b282640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.1:6790/0 2024-03-25T18:33:32.540 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:33:32.537+0000 7fac0b282640 1 -- 172.21.15.1:0/1612919699 --> v1:172.21.15.1:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7fac0413b550 con 0x7fac040b1b00 2024-03-25T18:33:32.540 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:33:32.537+0000 7fac0b282640 1 librados: init done 2024-03-25T18:33:32.540 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:33:32.538+0000 7fac0b282640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T18:33:32.540 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:33:32.538+0000 7fac0b282640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:33:32.540 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:33:32.538+0000 7fac0b282640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.1:6790/0 2024-03-25T18:33:32.540 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:33:32.538+0000 7fac0b282640 1 -- 172.21.15.1:0/1612919699 --> v1:172.21.15.1:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fac0404f730 con 0x7fac040b1b00 2024-03-25T18:33:32.567 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:33:32.566+0000 7fabf9ffb640 1 -- 172.21.15.1:0/1612919699 <== mon.2 v1:172.21.15.1:6790/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fac00003610 con 0x7fac040b1b00 2024-03-25T18:33:32.567 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:33:32.566+0000 7fabf9ffb640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:33:32.567 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:33:32.566+0000 7fabf9ffb640 1 -- 172.21.15.1:0/1612919699 <== mon.2 v1:172.21.15.1:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1586965025 0 0) 0x7fac00003a20 con 0x7fac040b1b00 2024-03-25T18:33:32.567 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:33:32.566+0000 7fabf9ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:33:32.567 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:33:32.566+0000 7fabf9ffb640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-25T18:33:32.567 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:33:32.566+0000 7fabf9ffb640 10 monclient: dump: 2024-03-25T18:33:32.567 INFO:teuthology.misc.health.smithi001.stderr:epoch 1 2024-03-25T18:33:32.567 INFO:teuthology.misc.health.smithi001.stderr:fsid c8ea72cf-9e99-4383-9b52-6c23d365fdfc 2024-03-25T18:33:32.567 INFO:teuthology.misc.health.smithi001.stderr:last_changed 2024-03-25T18:25:50.510519+0000 2024-03-25T18:33:32.568 INFO:teuthology.misc.health.smithi001.stderr:created 2024-03-25T18:25:50.510519+0000 2024-03-25T18:33:32.568 INFO:teuthology.misc.health.smithi001.stderr:min_mon_release 19 (squid) 2024-03-25T18:33:32.568 INFO:teuthology.misc.health.smithi001.stderr:election_strategy: 3 2024-03-25T18:33:32.568 INFO:teuthology.misc.health.smithi001.stderr:0: v1:172.21.15.1:6789/0 mon.a 2024-03-25T18:33:32.568 INFO:teuthology.misc.health.smithi001.stderr:1: v1:172.21.15.115:6789/0 mon.b 2024-03-25T18:33:32.568 INFO:teuthology.misc.health.smithi001.stderr:2: v1:172.21.15.1:6790/0 mon.c 2024-03-25T18:33:32.568 INFO:teuthology.misc.health.smithi001.stderr: 2024-03-25T18:33:32.568 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:33:32.566+0000 7fabfb7fe640 4 set_mon_vals no callback set 2024-03-25T18:33:32.569 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:33:32.567+0000 7fabf9ffb640 1 -- 172.21.15.1:0/1612919699 <== mon.2 v1:172.21.15.1:6790/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 21787975 0 0) 0x7fac00003c40 con 0x7fac040b1b00 2024-03-25T18:33:32.569 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:33:32.567+0000 7fac0881d640 10 monclient: get_auth_request con 0x7fabe806de00 auth_method 0 2024-03-25T18:33:32.569 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:33:32.568+0000 7fabf9ffb640 10 client.4532.objecter ms_handle_connect 0x7fabe806de00 2024-03-25T18:33:32.571 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:33:32.570+0000 7fabf9ffb640 1 -- 172.21.15.1:0/1612919699 <== mon.2 v1:172.21.15.1:6790/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 996254230 0 0) 0x7fac00054820 con 0x7fac040b1b00 2024-03-25T18:33:32.571 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:33:32.570+0000 7fabf9ffb640 10 client.4532.objecter ms_dispatch 0x7fac04000b90 osd_map(10..10 src has 1..10) v4 2024-03-25T18:33:32.571 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:33:32.570+0000 7fabf9ffb640 3 client.4532.objecter handle_osd_map got epochs [10,10] > 0 2024-03-25T18:33:32.572 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:33:32.570+0000 7fabf9ffb640 3 client.4532.objecter handle_osd_map decoding full epoch 10 2024-03-25T18:33:32.572 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:33:32.570+0000 7fabf9ffb640 20 client.4532.objecter dump_active .. 0 homeless 2024-03-25T18:33:32.860 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:33:32.859+0000 7fabf9ffb640 1 -- 172.21.15.1:0/1612919699 <== mon.2 v1:172.21.15.1:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7fac00050330 con 0x7fac040b1b00 2024-03-25T18:33:32.860 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:33:32.859+0000 7fabf9ffb640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:33:32.860 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:33:32.859+0000 7fabf9ffb640 10 monclient: _finish_command 1 = system:0 2024-03-25T18:33:33.089 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:33:33.088+0000 7fac0b282640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:33:33.089 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:33:33.088+0000 7fac0b282640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:33:33.089 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:33:33.088+0000 7fac0b282640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.1:6790/0 2024-03-25T18:33:33.089 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:33:33.088+0000 7fac0b282640 1 -- 172.21.15.1:0/1612919699 --> v1:172.21.15.1:6790/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fac040aa090 con 0x7fac040b1b00 2024-03-25T18:33:33.222 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:33:33.221+0000 7fabf9ffb640 1 -- 172.21.15.1:0/1612919699 <== mon.2 v1:172.21.15.1:6790/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 4092213348) 0x7fac00052260 con 0x7fac040b1b00 2024-03-25T18:33:33.222 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:33:33.221+0000 7fabf9ffb640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:33:33.222 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:33:33.221+0000 7fabf9ffb640 10 monclient: _finish_command 2 = system:0 2024-03-25T18:33:33.222 INFO:teuthology.misc.health.smithi001.stdout: 2024-03-25T18:33:33.222 INFO:teuthology.misc.health.smithi001.stdout:{"epoch":10,"fsid":"c8ea72cf-9e99-4383-9b52-6c23d365fdfc","created":"2024-03-25T18:26:38.856638+0000","modified":"2024-03-25T18:26:44.224208+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T18:26:44.224208+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"b9f34922-3a84-49be-8426-807c1eceaefc","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"42bf306f-55e3-4634-9874-8b4b73ac6644","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"292cb23a-18a9-4207-a7cc-4d5727feaec0","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"45e0e9ab-898d-4673-9590-b4117bc5e84a","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"831a5a09-73ab-4a61-9872-088e76bdbc66","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"88cd903d-3b55-4840-9396-6e3cfe005282","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"89db7355-bc9d-46f4-8ab0-e8d9e744e026","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"cac47bee-0914-4f08-896f-348838882866","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T18:33:33.224 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:33:33.223+0000 7fac0b282640 10 librados: watch_flush enter 2024-03-25T18:33:33.224 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:33:33.223+0000 7fac0b282640 10 librados: watch_flush exit 2024-03-25T18:33:33.224 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:33:33.223+0000 7fac0b282640 20 client.4532.objecter shutdown clearing up homeless session... 2024-03-25T18:33:33.224 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:33:33.223+0000 7fac0b282640 10 client.4532.objecter successfully canceled tick 2024-03-25T18:33:33.224 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:33:33.223+0000 7fac0b282640 1 -- 172.21.15.1:0/1612919699 >> v1:172.21.15.115:6800/4237379176 conn(0x7fabe806de00 legacy=0x7fabe80702c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:33:33.224 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:33:33.223+0000 7fac0b282640 10 monclient: shutdown 2024-03-25T18:33:33.224 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:33:33.223+0000 7fac0b282640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:33:33.225 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:33:33.223+0000 7fac0b282640 1 -- 172.21.15.1:0/1612919699 >> v1:172.21.15.1:6790/0 conn(0x7fac040b1b00 legacy=0x7fac0413a600 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:33:33.225 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:33:33.223+0000 7fac0b282640 1 -- 172.21.15.1:0/1612919699 shutdown_connections 2024-03-25T18:33:33.225 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:33:33.223+0000 7fac0b282640 1 -- 172.21.15.1:0/1612919699 >> 172.21.15.1:0/1612919699 conn(0x7fac040617a0 msgr2=0x7fac040b6720 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:33:33.225 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:33:33.223+0000 7fac0b282640 1 -- 172.21.15.1:0/1612919699 shutdown_connections 2024-03-25T18:33:33.225 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:33:33.223+0000 7fac0b282640 1 -- 172.21.15.1:0/1612919699 wait complete. 2024-03-25T18:33:33.225 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:33:33.223+0000 7fac0b282640 1 librados: shutdown 2024-03-25T18:33:33.235 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T18:33:40.042 DEBUG:teuthology.orchestra.run.smithi001:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T18:33:40.161 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:33:40.160+0000 7f2b5df1c640 10 monclient: get_monmap_and_config 2024-03-25T18:33:40.162 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:33:40.160+0000 7f2b5df1c640 10 monclient: build_initial_monmap 2024-03-25T18:33:40.162 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:33:40.161+0000 7f2b5df1c640 10 monclient: monmap: 2024-03-25T18:33:40.162 INFO:teuthology.misc.health.smithi001.stderr:epoch 0 2024-03-25T18:33:40.162 INFO:teuthology.misc.health.smithi001.stderr:fsid c8ea72cf-9e99-4383-9b52-6c23d365fdfc 2024-03-25T18:33:40.162 INFO:teuthology.misc.health.smithi001.stderr:last_changed 2024-03-25T18:33:40.161553+0000 2024-03-25T18:33:40.162 INFO:teuthology.misc.health.smithi001.stderr:created 2024-03-25T18:33:40.161553+0000 2024-03-25T18:33:40.162 INFO:teuthology.misc.health.smithi001.stderr:min_mon_release 0 (unknown) 2024-03-25T18:33:40.162 INFO:teuthology.misc.health.smithi001.stderr:election_strategy: 3 2024-03-25T18:33:40.162 INFO:teuthology.misc.health.smithi001.stderr:0: v1:172.21.15.1:6789/0 mon.noname-a 2024-03-25T18:33:40.162 INFO:teuthology.misc.health.smithi001.stderr:1: v1:172.21.15.115:6789/0 mon.noname-c 2024-03-25T18:33:40.162 INFO:teuthology.misc.health.smithi001.stderr:2: v1:172.21.15.1:6790/0 mon.noname-b 2024-03-25T18:33:40.162 INFO:teuthology.misc.health.smithi001.stderr: 2024-03-25T18:33:40.163 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:33:40.161+0000 7f2b5df1c640 1 Processor -- start 2024-03-25T18:33:40.164 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:33:40.161+0000 7f2b5df1c640 1 -- start start 2024-03-25T18:33:40.164 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:33:40.161+0000 7f2b5df1c640 10 monclient: init 2024-03-25T18:33:40.164 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:33:40.162+0000 7f2b5df1c640 10 monclient: _reopen_session rank -1 2024-03-25T18:33:40.164 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:33:40.162+0000 7f2b5df1c640 10 monclient: _add_conns ranks=[1,2,0] 2024-03-25T18:33:40.164 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:33:40.162+0000 7f2b5df1c640 10 monclient(hunting): picked mon.noname-c con 0x7f2b580b4ff0 addr v1:172.21.15.115:6789/0 2024-03-25T18:33:40.164 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:33:40.162+0000 7f2b5df1c640 10 monclient(hunting): picked mon.noname-b con 0x7f2b580b4670 addr v1:172.21.15.1:6790/0 2024-03-25T18:33:40.164 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:33:40.162+0000 7f2b5df1c640 10 monclient(hunting): picked mon.noname-a con 0x7f2b5813c480 addr v1:172.21.15.1:6789/0 2024-03-25T18:33:40.164 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:33:40.162+0000 7f2b5df1c640 1 -- --> v1:172.21.15.1:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2b580ab0e0 con 0x7f2b5813c480 2024-03-25T18:33:40.164 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:33:40.162+0000 7f2b5df1c640 1 -- --> v1:172.21.15.115:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2b580ab3c0 con 0x7f2b580b4ff0 2024-03-25T18:33:40.164 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:33:40.162+0000 7f2b5df1c640 1 -- --> v1:172.21.15.1:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2b580a9f30 con 0x7f2b580b4670 2024-03-25T18:33:40.164 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:33:40.162+0000 7f2b5df1c640 10 monclient(hunting): _renew_subs 2024-03-25T18:33:40.164 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:33:40.162+0000 7f2b5df1c640 10 monclient(hunting): authenticate will time out at 7900.632292s 2024-03-25T18:33:40.170 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:33:40.170+0000 7f2b56ffd640 1 --1- >> v1:172.21.15.1:6790/0 conn(0x7f2b580b4670 0x7f2b581425c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.1:6790/0 says I am v1:172.21.15.1:45866/0 (socket says 172.21.15.1:45866) 2024-03-25T18:33:40.171 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:33:40.170+0000 7f2b56ffd640 1 -- 172.21.15.1:0/2166846248 learned_addr learned my addr 172.21.15.1:0/2166846248 (peer_addr_for_me v1:172.21.15.1:0/0) 2024-03-25T18:33:40.181 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:33:40.180+0000 7f2b567fc640 1 -- 172.21.15.1:0/2166846248 <== mon.2 v1:172.21.15.1:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1232271865 0 0) 0x7f2b580a9f30 con 0x7f2b580b4670 2024-03-25T18:33:40.181 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:33:40.180+0000 7f2b567fc640 10 monclient(hunting): _negotiate 2024-03-25T18:33:40.181 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:33:40.180+0000 7f2b567fc640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:33:40.181 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:33:40.180+0000 7f2b567fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:33:40.181 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:33:40.180+0000 7f2b567fc640 10 monclient(hunting): my global_id is 4535 2024-03-25T18:33:40.181 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:33:40.180+0000 7f2b567fc640 1 -- 172.21.15.1:0/2166846248 --> v1:172.21.15.1:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f2b40003520 con 0x7f2b580b4670 2024-03-25T18:33:40.195 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:33:40.194+0000 7f2b567fc640 1 -- 172.21.15.1:0/2166846248 <== mon.2 v1:172.21.15.1:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 4114446835 0 0) 0x7f2b40003520 con 0x7f2b580b4670 2024-03-25T18:33:40.195 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:33:40.194+0000 7f2b567fc640 1 -- 172.21.15.1:0/2166846248 >> v1:172.21.15.115:6789/0 conn(0x7f2b580b4ff0 legacy=0x7f2b5813bd70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-25T18:33:40.195 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:33:40.194+0000 7f2b567fc640 1 -- 172.21.15.1:0/2166846248 >> v1:172.21.15.1:6789/0 conn(0x7f2b5813c480 legacy=0x7f2b5814dc00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:33:40.195 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:33:40.194+0000 7f2b567fc640 10 monclient: _finish_hunting 0 2024-03-25T18:33:40.195 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:33:40.194+0000 7f2b567fc640 1 monclient: found mon.noname-b 2024-03-25T18:33:40.196 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:33:40.194+0000 7f2b567fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:33:40.196 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:33:40.194+0000 7f2b567fc640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.1:6790/0 2024-03-25T18:33:40.196 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:33:40.194+0000 7f2b567fc640 1 -- 172.21.15.1:0/2166846248 --> v1:172.21.15.1:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2b580a9b70 con 0x7f2b580b4670 2024-03-25T18:33:40.196 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:33:40.194+0000 7f2b567fc640 10 monclient: _finish_auth 0 2024-03-25T18:33:40.196 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:33:40.194+0000 7f2b567fc640 10 monclient: _check_auth_tickets 2024-03-25T18:33:40.196 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:33:40.194+0000 7f2b567fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:33:40.196 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:33:40.194+0000 7f2b5df1c640 5 monclient: authenticate success, global_id 4535 2024-03-25T18:33:40.196 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:33:40.194+0000 7f2b5df1c640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T18:33:40.201 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:33:40.200+0000 7f2b567fc640 1 -- 172.21.15.1:0/2166846248 <== mon.2 v1:172.21.15.1:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1586965025 0 0) 0x7f2b4c001690 con 0x7f2b580b4670 2024-03-25T18:33:40.201 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:33:40.200+0000 7f2b567fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:33:40.201 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:33:40.200+0000 7f2b567fc640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-03-25T18:33:40.201 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:33:40.200+0000 7f2b567fc640 10 monclient: dump: 2024-03-25T18:33:40.201 INFO:teuthology.misc.health.smithi001.stderr:epoch 1 2024-03-25T18:33:40.201 INFO:teuthology.misc.health.smithi001.stderr:fsid c8ea72cf-9e99-4383-9b52-6c23d365fdfc 2024-03-25T18:33:40.201 INFO:teuthology.misc.health.smithi001.stderr:last_changed 2024-03-25T18:25:50.510519+0000 2024-03-25T18:33:40.201 INFO:teuthology.misc.health.smithi001.stderr:created 2024-03-25T18:25:50.510519+0000 2024-03-25T18:33:40.201 INFO:teuthology.misc.health.smithi001.stderr:min_mon_release 19 (squid) 2024-03-25T18:33:40.201 INFO:teuthology.misc.health.smithi001.stderr:election_strategy: 3 2024-03-25T18:33:40.201 INFO:teuthology.misc.health.smithi001.stderr:0: v1:172.21.15.1:6789/0 mon.a 2024-03-25T18:33:40.201 INFO:teuthology.misc.health.smithi001.stderr:1: v1:172.21.15.115:6789/0 mon.b 2024-03-25T18:33:40.201 INFO:teuthology.misc.health.smithi001.stderr:2: v1:172.21.15.1:6790/0 mon.c 2024-03-25T18:33:40.201 INFO:teuthology.misc.health.smithi001.stderr: 2024-03-25T18:33:40.201 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:33:40.200+0000 7f2b5df1c640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T18:33:40.203 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:33:40.203+0000 7f2b567fc640 1 -- 172.21.15.1:0/2166846248 <== mon.2 v1:172.21.15.1:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f2b4c001920 con 0x7f2b580b4670 2024-03-25T18:33:40.204 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:33:40.203+0000 7f2b567fc640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:33:40.204 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:33:40.203+0000 7f2b567fc640 1 -- 172.21.15.1:0/2166846248 <== mon.2 v1:172.21.15.1:6790/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1586965025 0 0) 0x7f2b4c001cb0 con 0x7f2b580b4670 2024-03-25T18:33:40.204 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:33:40.203+0000 7f2b5df1c640 10 monclient: get_monmap_and_config success 2024-03-25T18:33:40.204 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:33:40.203+0000 7f2b5df1c640 4 set_mon_vals no callback set 2024-03-25T18:33:40.204 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:33:40.203+0000 7f2b567fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:33:40.204 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:33:40.203+0000 7f2b567fc640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-25T18:33:40.204 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:33:40.203+0000 7f2b567fc640 10 monclient: dump: 2024-03-25T18:33:40.204 INFO:teuthology.misc.health.smithi001.stderr:epoch 1 2024-03-25T18:33:40.204 INFO:teuthology.misc.health.smithi001.stderr:fsid c8ea72cf-9e99-4383-9b52-6c23d365fdfc 2024-03-25T18:33:40.204 INFO:teuthology.misc.health.smithi001.stderr:last_changed 2024-03-25T18:25:50.510519+0000 2024-03-25T18:33:40.204 INFO:teuthology.misc.health.smithi001.stderr:created 2024-03-25T18:25:50.510519+0000 2024-03-25T18:33:40.204 INFO:teuthology.misc.health.smithi001.stderr:min_mon_release 19 (squid) 2024-03-25T18:33:40.204 INFO:teuthology.misc.health.smithi001.stderr:election_strategy: 3 2024-03-25T18:33:40.205 INFO:teuthology.misc.health.smithi001.stderr:0: v1:172.21.15.1:6789/0 mon.a 2024-03-25T18:33:40.205 INFO:teuthology.misc.health.smithi001.stderr:1: v1:172.21.15.115:6789/0 mon.b 2024-03-25T18:33:40.205 INFO:teuthology.misc.health.smithi001.stderr:2: v1:172.21.15.1:6790/0 mon.c 2024-03-25T18:33:40.205 INFO:teuthology.misc.health.smithi001.stderr: 2024-03-25T18:33:40.206 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:33:40.204+0000 7f2b5df1c640 10 monclient: shutdown 2024-03-25T18:33:40.206 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:33:40.204+0000 7f2b5df1c640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:33:40.206 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:33:40.204+0000 7f2b5df1c640 1 -- 172.21.15.1:0/2166846248 >> v1:172.21.15.1:6790/0 conn(0x7f2b580b4670 legacy=0x7f2b581425c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:33:40.206 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:33:40.204+0000 7f2b5df1c640 1 -- 172.21.15.1:0/2166846248 shutdown_connections 2024-03-25T18:33:40.206 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:33:40.204+0000 7f2b5df1c640 1 -- 172.21.15.1:0/2166846248 >> 172.21.15.1:0/2166846248 conn(0x7f2b580617a0 msgr2=0x7f2b58061bb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:33:40.206 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:33:40.204+0000 7f2b5df1c640 1 -- 172.21.15.1:0/2166846248 shutdown_connections 2024-03-25T18:33:40.207 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:33:40.204+0000 7f2b5df1c640 1 -- 172.21.15.1:0/2166846248 wait complete. 2024-03-25T18:33:40.207 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:33:40.204+0000 7f2b5df1c640 10 monclient: build_initial_monmap 2024-03-25T18:33:40.207 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:33:40.204+0000 7f2b5df1c640 10 monclient: monmap: 2024-03-25T18:33:40.207 INFO:teuthology.misc.health.smithi001.stderr:epoch 0 2024-03-25T18:33:40.207 INFO:teuthology.misc.health.smithi001.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T18:33:40.207 INFO:teuthology.misc.health.smithi001.stderr:last_changed 0.000000 2024-03-25T18:33:40.207 INFO:teuthology.misc.health.smithi001.stderr:created 0.000000 2024-03-25T18:33:40.207 INFO:teuthology.misc.health.smithi001.stderr:min_mon_release 0 (unknown) 2024-03-25T18:33:40.207 INFO:teuthology.misc.health.smithi001.stderr:election_strategy: 1 2024-03-25T18:33:40.207 INFO:teuthology.misc.health.smithi001.stderr:0: v1:172.21.15.1:6789/0 mon.noname-a 2024-03-25T18:33:40.207 INFO:teuthology.misc.health.smithi001.stderr:1: v1:172.21.15.115:6789/0 mon.noname-b 2024-03-25T18:33:40.207 INFO:teuthology.misc.health.smithi001.stderr:2: v1:172.21.15.1:6790/0 mon.noname-c 2024-03-25T18:33:40.207 INFO:teuthology.misc.health.smithi001.stderr: 2024-03-25T18:33:40.207 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:33:40.205+0000 7f2b5df1c640 1 librados: starting msgr at 2024-03-25T18:33:40.207 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:33:40.205+0000 7f2b5df1c640 1 librados: starting objecter 2024-03-25T18:33:40.208 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:33:40.205+0000 7f2b5df1c640 1 Processor -- start 2024-03-25T18:33:40.208 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:33:40.205+0000 7f2b5df1c640 1 -- start start 2024-03-25T18:33:40.208 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:33:40.205+0000 7f2b5df1c640 1 librados: setting wanted keys 2024-03-25T18:33:40.208 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:33:40.205+0000 7f2b5df1c640 1 librados: calling monclient init 2024-03-25T18:33:40.208 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:33:40.205+0000 7f2b5df1c640 10 monclient: init 2024-03-25T18:33:40.209 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:33:40.206+0000 7f2b5df1c640 10 monclient: _reopen_session rank -1 2024-03-25T18:33:40.209 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:33:40.206+0000 7f2b5df1c640 10 monclient: _add_conns ranks=[1,0,2] 2024-03-25T18:33:40.209 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:33:40.206+0000 7f2b5df1c640 10 monclient(hunting): picked mon.noname-b con 0x7f2b580b4670 addr v1:172.21.15.115:6789/0 2024-03-25T18:33:40.209 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:33:40.206+0000 7f2b5df1c640 10 monclient(hunting): picked mon.noname-a con 0x7f2b580b4ff0 addr v1:172.21.15.1:6789/0 2024-03-25T18:33:40.209 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:33:40.206+0000 7f2b5df1c640 10 monclient(hunting): picked mon.noname-c con 0x7f2b5813c480 addr v1:172.21.15.1:6790/0 2024-03-25T18:33:40.209 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:33:40.206+0000 7f2b5df1c640 1 -- --> v1:172.21.15.1:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2b5813e4d0 con 0x7f2b580b4ff0 2024-03-25T18:33:40.209 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:33:40.206+0000 7f2b5df1c640 1 -- --> v1:172.21.15.115:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2b581446b0 con 0x7f2b580b4670 2024-03-25T18:33:40.209 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:33:40.206+0000 7f2b5df1c640 1 -- --> v1:172.21.15.1:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2b58144990 con 0x7f2b5813c480 2024-03-25T18:33:40.209 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:33:40.206+0000 7f2b5df1c640 10 monclient(hunting): _renew_subs 2024-03-25T18:33:40.209 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:33:40.206+0000 7f2b5df1c640 10 monclient(hunting): authenticate will time out at 7900.676206s 2024-03-25T18:33:40.220 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:33:40.219+0000 7f2b57fff640 1 --1- >> v1:172.21.15.1:6790/0 conn(0x7f2b5813c480 0x7f2b581e2380 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.1:6790/0 says I am v1:172.21.15.1:45902/0 (socket says 172.21.15.1:45902) 2024-03-25T18:33:40.220 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:33:40.219+0000 7f2b57fff640 1 -- 172.21.15.1:0/1741793181 learned_addr learned my addr 172.21.15.1:0/1741793181 (peer_addr_for_me v1:172.21.15.1:0/0) 2024-03-25T18:33:40.221 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:33:40.220+0000 7f2b3ffff640 10 client.?.objecter ms_handle_connect 0x7f2b5813c480 2024-03-25T18:33:40.221 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:33:40.220+0000 7f2b3ffff640 10 client.?.objecter resend_mon_ops 2024-03-25T18:33:40.229 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:33:40.228+0000 7f2b3ffff640 1 -- 172.21.15.1:0/1741793181 <== mon.2 v1:172.21.15.1:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1749405025 0 0) 0x7f2b58144990 con 0x7f2b5813c480 2024-03-25T18:33:40.229 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:33:40.228+0000 7f2b3ffff640 10 monclient(hunting): _negotiate 2024-03-25T18:33:40.229 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:33:40.228+0000 7f2b3ffff640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:33:40.229 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:33:40.228+0000 7f2b3ffff640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:33:40.229 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:33:40.228+0000 7f2b3ffff640 10 monclient(hunting): my global_id is 4538 2024-03-25T18:33:40.229 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:33:40.228+0000 7f2b3ffff640 1 -- 172.21.15.1:0/1741793181 --> v1:172.21.15.1:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f2b44004eb0 con 0x7f2b5813c480 2024-03-25T18:33:40.230 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:33:40.229+0000 7f2b3ffff640 10 client.?.objecter ms_handle_connect 0x7f2b580b4670 2024-03-25T18:33:40.230 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:33:40.229+0000 7f2b3ffff640 10 client.?.objecter resend_mon_ops 2024-03-25T18:33:40.231 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:33:40.230+0000 7f2b3ffff640 10 client.?.objecter ms_handle_connect 0x7f2b580b4ff0 2024-03-25T18:33:40.231 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:33:40.230+0000 7f2b3ffff640 10 client.?.objecter resend_mon_ops 2024-03-25T18:33:40.239 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:33:40.238+0000 7f2b3ffff640 1 -- 172.21.15.1:0/1741793181 <== mon.1 v1:172.21.15.115:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4078594885 0 0) 0x7f2b581446b0 con 0x7f2b580b4670 2024-03-25T18:33:40.239 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:33:40.238+0000 7f2b3ffff640 10 monclient(hunting): _negotiate 2024-03-25T18:33:40.239 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:33:40.238+0000 7f2b3ffff640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:33:40.239 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:33:40.238+0000 7f2b3ffff640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:33:40.240 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:33:40.238+0000 7f2b3ffff640 10 monclient(hunting): my global_id is 4534 2024-03-25T18:33:40.240 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:33:40.238+0000 7f2b3ffff640 1 -- 172.21.15.1:0/1741793181 --> v1:172.21.15.115:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f2b58144990 con 0x7f2b580b4670 2024-03-25T18:33:40.244 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:33:40.243+0000 7f2b3ffff640 1 -- 172.21.15.1:0/1741793181 <== mon.2 v1:172.21.15.1:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3366319735 0 0) 0x7f2b44004eb0 con 0x7f2b5813c480 2024-03-25T18:33:40.244 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:33:40.243+0000 7f2b3ffff640 1 -- 172.21.15.1:0/1741793181 --> v1:172.21.15.1:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f2b581446b0 con 0x7f2b5813c480 2024-03-25T18:33:40.244 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:33:40.243+0000 7f2b3ffff640 1 -- 172.21.15.1:0/1741793181 <== mon.2 v1:172.21.15.1:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1586965025 0 0) 0x7f2b50001c90 con 0x7f2b5813c480 2024-03-25T18:33:40.245 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:33:40.243+0000 7f2b3ffff640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:33:40.245 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:33:40.243+0000 7f2b3ffff640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-03-25T18:33:40.245 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:33:40.243+0000 7f2b3ffff640 10 monclient(hunting): dump: 2024-03-25T18:33:40.245 INFO:teuthology.misc.health.smithi001.stderr:epoch 1 2024-03-25T18:33:40.245 INFO:teuthology.misc.health.smithi001.stderr:fsid c8ea72cf-9e99-4383-9b52-6c23d365fdfc 2024-03-25T18:33:40.245 INFO:teuthology.misc.health.smithi001.stderr:last_changed 2024-03-25T18:25:50.510519+0000 2024-03-25T18:33:40.245 INFO:teuthology.misc.health.smithi001.stderr:created 2024-03-25T18:25:50.510519+0000 2024-03-25T18:33:40.245 INFO:teuthology.misc.health.smithi001.stderr:min_mon_release 19 (squid) 2024-03-25T18:33:40.245 INFO:teuthology.misc.health.smithi001.stderr:election_strategy: 3 2024-03-25T18:33:40.245 INFO:teuthology.misc.health.smithi001.stderr:0: v1:172.21.15.1:6789/0 mon.a 2024-03-25T18:33:40.245 INFO:teuthology.misc.health.smithi001.stderr:1: v1:172.21.15.115:6789/0 mon.b 2024-03-25T18:33:40.245 INFO:teuthology.misc.health.smithi001.stderr:2: v1:172.21.15.1:6790/0 mon.c 2024-03-25T18:33:40.245 INFO:teuthology.misc.health.smithi001.stderr: 2024-03-25T18:33:40.245 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:33:40.243+0000 7f2b3ffff640 10 monclient(hunting): _finish_auth 0 2024-03-25T18:33:40.247 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:33:40.246+0000 7f2b3ffff640 1 -- 172.21.15.1:0/1741793181 <== mon.0 v1:172.21.15.1:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1356584911 0 0) 0x7f2b580a9b70 con 0x7f2b580b4ff0 2024-03-25T18:33:40.247 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:33:40.246+0000 7f2b3ffff640 10 monclient(hunting): _negotiate 2024-03-25T18:33:40.247 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:33:40.246+0000 7f2b3ffff640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:33:40.247 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:33:40.246+0000 7f2b3ffff640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:33:40.247 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:33:40.246+0000 7f2b3ffff640 10 monclient(hunting): my global_id is 4521 2024-03-25T18:33:40.247 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:33:40.246+0000 7f2b3ffff640 1 -- 172.21.15.1:0/1741793181 --> v1:172.21.15.1:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f2b44004eb0 con 0x7f2b580b4ff0 2024-03-25T18:33:40.254 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:33:40.252+0000 7f2b3ffff640 1 -- 172.21.15.1:0/1741793181 <== mon.2 v1:172.21.15.1:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 17121134 0 0) 0x7f2b581446b0 con 0x7f2b5813c480 2024-03-25T18:33:40.254 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:33:40.253+0000 7f2b3ffff640 1 -- 172.21.15.1:0/1741793181 >> v1:172.21.15.115:6789/0 conn(0x7f2b580b4670 legacy=0x7f2b5813ceb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:33:40.254 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:33:40.253+0000 7f2b3ffff640 1 -- 172.21.15.1:0/1741793181 >> v1:172.21.15.1:6789/0 conn(0x7f2b580b4ff0 legacy=0x7f2b5813d5c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:33:40.254 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:33:40.253+0000 7f2b3ffff640 10 monclient: _finish_hunting 0 2024-03-25T18:33:40.254 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:33:40.253+0000 7f2b3ffff640 1 monclient: found mon.c 2024-03-25T18:33:40.254 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:33:40.253+0000 7f2b3ffff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:33:40.254 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:33:40.253+0000 7f2b3ffff640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.1:6790/0 2024-03-25T18:33:40.255 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:33:40.253+0000 7f2b3ffff640 1 -- 172.21.15.1:0/1741793181 --> v1:172.21.15.1:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2b58146660 con 0x7f2b5813c480 2024-03-25T18:33:40.255 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:33:40.253+0000 7f2b3ffff640 10 monclient: _finish_auth 0 2024-03-25T18:33:40.255 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:33:40.253+0000 7f2b3ffff640 10 monclient: _check_auth_tickets 2024-03-25T18:33:40.255 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:33:40.253+0000 7f2b3ffff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:33:40.255 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:33:40.253+0000 7f2b5df1c640 5 monclient: authenticate success, global_id 4538 2024-03-25T18:33:40.255 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:33:40.253+0000 7f2b5df1c640 10 monclient: _renew_subs 2024-03-25T18:33:40.255 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:33:40.253+0000 7f2b5df1c640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.1:6790/0 2024-03-25T18:33:40.255 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:33:40.253+0000 7f2b5df1c640 1 -- 172.21.15.1:0/1741793181 --> v1:172.21.15.1:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2b5813f750 con 0x7f2b5813c480 2024-03-25T18:33:40.255 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:33:40.253+0000 7f2b5df1c640 10 client.4538.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T18:33:40.255 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:33:40.253+0000 7f2b5df1c640 10 monclient: _renew_subs 2024-03-25T18:33:40.255 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:33:40.253+0000 7f2b5df1c640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.1:6790/0 2024-03-25T18:33:40.255 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:33:40.253+0000 7f2b5df1c640 1 -- 172.21.15.1:0/1741793181 --> v1:172.21.15.1:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f2b581439f0 con 0x7f2b5813c480 2024-03-25T18:33:40.255 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:33:40.253+0000 7f2b5df1c640 1 librados: init done 2024-03-25T18:33:40.256 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:33:40.254+0000 7f2b5df1c640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T18:33:40.256 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:33:40.254+0000 7f2b5df1c640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:33:40.256 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:33:40.254+0000 7f2b5df1c640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.1:6790/0 2024-03-25T18:33:40.256 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:33:40.254+0000 7f2b5df1c640 1 -- 172.21.15.1:0/1741793181 --> v1:172.21.15.1:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f2b5806dbd0 con 0x7f2b5813c480 2024-03-25T18:33:40.568 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:33:40.567+0000 7f2b3ffff640 1 -- 172.21.15.1:0/1741793181 <== mon.2 v1:172.21.15.1:6790/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f2b50003990 con 0x7f2b5813c480 2024-03-25T18:33:40.568 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:33:40.567+0000 7f2b3ffff640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:33:40.568 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:33:40.567+0000 7f2b3ffff640 1 -- 172.21.15.1:0/1741793181 <== mon.2 v1:172.21.15.1:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1586965025 0 0) 0x7f2b50003d60 con 0x7f2b5813c480 2024-03-25T18:33:40.568 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:33:40.567+0000 7f2b3ffff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:33:40.568 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:33:40.567+0000 7f2b3ffff640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-25T18:33:40.568 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:33:40.567+0000 7f2b3ffff640 10 monclient: dump: 2024-03-25T18:33:40.568 INFO:teuthology.misc.health.smithi001.stderr:epoch 1 2024-03-25T18:33:40.568 INFO:teuthology.misc.health.smithi001.stderr:fsid c8ea72cf-9e99-4383-9b52-6c23d365fdfc 2024-03-25T18:33:40.568 INFO:teuthology.misc.health.smithi001.stderr:last_changed 2024-03-25T18:25:50.510519+0000 2024-03-25T18:33:40.569 INFO:teuthology.misc.health.smithi001.stderr:created 2024-03-25T18:25:50.510519+0000 2024-03-25T18:33:40.569 INFO:teuthology.misc.health.smithi001.stderr:min_mon_release 19 (squid) 2024-03-25T18:33:40.569 INFO:teuthology.misc.health.smithi001.stderr:election_strategy: 3 2024-03-25T18:33:40.569 INFO:teuthology.misc.health.smithi001.stderr:0: v1:172.21.15.1:6789/0 mon.a 2024-03-25T18:33:40.569 INFO:teuthology.misc.health.smithi001.stderr:1: v1:172.21.15.115:6789/0 mon.b 2024-03-25T18:33:40.569 INFO:teuthology.misc.health.smithi001.stderr:2: v1:172.21.15.1:6790/0 mon.c 2024-03-25T18:33:40.569 INFO:teuthology.misc.health.smithi001.stderr: 2024-03-25T18:33:40.569 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:33:40.567+0000 7f2b55ffb640 4 set_mon_vals no callback set 2024-03-25T18:33:40.570 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:33:40.567+0000 7f2b3ffff640 1 -- 172.21.15.1:0/1741793181 <== mon.2 v1:172.21.15.1:6790/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 21787975 0 0) 0x7f2b50003f80 con 0x7f2b5813c480 2024-03-25T18:33:40.570 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:33:40.568+0000 7f2b577fe640 10 monclient: get_auth_request con 0x7f2b4406de00 auth_method 0 2024-03-25T18:33:40.570 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:33:40.568+0000 7f2b3ffff640 10 client.4538.objecter ms_handle_connect 0x7f2b4406de00 2024-03-25T18:33:40.574 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:33:40.573+0000 7f2b3ffff640 1 -- 172.21.15.1:0/1741793181 <== mon.2 v1:172.21.15.1:6790/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 996254230 0 0) 0x7f2b50054ac0 con 0x7f2b5813c480 2024-03-25T18:33:40.574 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:33:40.573+0000 7f2b3ffff640 10 client.4538.objecter ms_dispatch 0x7f2b58000b90 osd_map(10..10 src has 1..10) v4 2024-03-25T18:33:40.574 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:33:40.573+0000 7f2b3ffff640 3 client.4538.objecter handle_osd_map got epochs [10,10] > 0 2024-03-25T18:33:40.574 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:33:40.573+0000 7f2b3ffff640 3 client.4538.objecter handle_osd_map decoding full epoch 10 2024-03-25T18:33:40.574 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:33:40.573+0000 7f2b3ffff640 20 client.4538.objecter dump_active .. 0 homeless 2024-03-25T18:33:40.574 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:33:40.573+0000 7f2b3ffff640 1 -- 172.21.15.1:0/1741793181 <== mon.2 v1:172.21.15.1:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f2b500505d0 con 0x7f2b5813c480 2024-03-25T18:33:40.574 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:33:40.573+0000 7f2b3ffff640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:33:40.574 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:33:40.573+0000 7f2b3ffff640 10 monclient: _finish_command 1 = system:0 2024-03-25T18:33:40.826 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:33:40.825+0000 7f2b5df1c640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:33:40.827 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:33:40.825+0000 7f2b5df1c640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:33:40.827 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:33:40.825+0000 7f2b5df1c640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.1:6790/0 2024-03-25T18:33:40.827 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:33:40.825+0000 7f2b5df1c640 1 -- 172.21.15.1:0/1741793181 --> v1:172.21.15.1:6790/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f2b580688c0 con 0x7f2b5813c480 2024-03-25T18:33:40.868 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:33:40.867+0000 7f2b3ffff640 1 -- 172.21.15.1:0/1741793181 <== mon.2 v1:172.21.15.1:6790/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 4092213348) 0x7f2b50052500 con 0x7f2b5813c480 2024-03-25T18:33:40.868 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:33:40.867+0000 7f2b3ffff640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:33:40.868 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:33:40.867+0000 7f2b3ffff640 10 monclient: _finish_command 2 = system:0 2024-03-25T18:33:40.868 INFO:teuthology.misc.health.smithi001.stdout: 2024-03-25T18:33:40.869 INFO:teuthology.misc.health.smithi001.stdout:{"epoch":10,"fsid":"c8ea72cf-9e99-4383-9b52-6c23d365fdfc","created":"2024-03-25T18:26:38.856638+0000","modified":"2024-03-25T18:26:44.224208+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T18:26:44.224208+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"b9f34922-3a84-49be-8426-807c1eceaefc","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"42bf306f-55e3-4634-9874-8b4b73ac6644","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"292cb23a-18a9-4207-a7cc-4d5727feaec0","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"45e0e9ab-898d-4673-9590-b4117bc5e84a","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"831a5a09-73ab-4a61-9872-088e76bdbc66","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"88cd903d-3b55-4840-9396-6e3cfe005282","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"89db7355-bc9d-46f4-8ab0-e8d9e744e026","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"cac47bee-0914-4f08-896f-348838882866","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T18:33:40.871 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:33:40.869+0000 7f2b5df1c640 10 librados: watch_flush enter 2024-03-25T18:33:40.871 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:33:40.869+0000 7f2b5df1c640 10 librados: watch_flush exit 2024-03-25T18:33:40.871 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:33:40.870+0000 7f2b5df1c640 20 client.4538.objecter shutdown clearing up homeless session... 2024-03-25T18:33:40.871 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:33:40.870+0000 7f2b5df1c640 10 client.4538.objecter successfully canceled tick 2024-03-25T18:33:40.871 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:33:40.870+0000 7f2b5df1c640 1 -- 172.21.15.1:0/1741793181 >> v1:172.21.15.115:6800/4237379176 conn(0x7f2b4406de00 legacy=0x7f2b440702c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:33:40.872 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:33:40.870+0000 7f2b5df1c640 10 monclient: shutdown 2024-03-25T18:33:40.872 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:33:40.870+0000 7f2b5df1c640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:33:40.872 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:33:40.870+0000 7f2b5df1c640 1 -- 172.21.15.1:0/1741793181 >> v1:172.21.15.1:6790/0 conn(0x7f2b5813c480 legacy=0x7f2b581e2380 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:33:40.872 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:33:40.870+0000 7f2b5df1c640 1 -- 172.21.15.1:0/1741793181 shutdown_connections 2024-03-25T18:33:40.872 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:33:40.870+0000 7f2b5df1c640 1 -- 172.21.15.1:0/1741793181 >> 172.21.15.1:0/1741793181 conn(0x7f2b580617a0 msgr2=0x7f2b580b8690 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:33:40.872 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:33:40.870+0000 7f2b5df1c640 1 -- 172.21.15.1:0/1741793181 shutdown_connections 2024-03-25T18:33:40.872 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:33:40.870+0000 7f2b5df1c640 1 -- 172.21.15.1:0/1741793181 wait complete. 2024-03-25T18:33:40.872 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:33:40.870+0000 7f2b5df1c640 1 librados: shutdown 2024-03-25T18:33:40.884 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T18:33:42.332 INFO:tasks.ceph.mon.a.smithi001.stderr:==00:00:07:20.891 34876== Warning: unimplemented fcntl command: 1036 2024-03-25T18:33:42.348 INFO:tasks.ceph.mon.a.smithi001.stderr:==00:00:07:20.907 34876== Warning: unimplemented fcntl command: 1036 2024-03-25T18:33:42.366 INFO:tasks.ceph.mon.a.smithi001.stderr:==00:00:07:20.925 34876== Warning: unimplemented fcntl command: 1036 2024-03-25T18:33:42.393 INFO:tasks.ceph.mon.b.smithi115.stderr:==00:00:07:20.876 34836== Warning: unimplemented fcntl command: 1036 2024-03-25T18:33:42.401 INFO:tasks.ceph.mon.c.smithi001.stderr:==00:00:07:20.960 34877== Warning: unimplemented fcntl command: 1036 2024-03-25T18:33:42.408 INFO:tasks.ceph.mon.b.smithi115.stderr:==00:00:07:20.892 34836== Warning: unimplemented fcntl command: 1036 2024-03-25T18:33:42.416 INFO:tasks.ceph.mon.c.smithi001.stderr:==00:00:07:20.975 34877== Warning: unimplemented fcntl command: 1036 2024-03-25T18:33:42.442 INFO:tasks.ceph.mon.b.smithi115.stderr:==00:00:07:20.926 34836== Warning: unimplemented fcntl command: 1036 2024-03-25T18:33:42.450 INFO:tasks.ceph.mon.c.smithi001.stderr:==00:00:07:21.008 34877== Warning: unimplemented fcntl command: 1036 2024-03-25T18:33:47.691 DEBUG:teuthology.orchestra.run.smithi001:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T18:33:47.808 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:33:47.807+0000 7fe03a64e640 10 monclient: get_monmap_and_config 2024-03-25T18:33:47.809 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:33:47.807+0000 7fe03a64e640 10 monclient: build_initial_monmap 2024-03-25T18:33:47.809 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:33:47.807+0000 7fe03a64e640 10 monclient: monmap: 2024-03-25T18:33:47.809 INFO:teuthology.misc.health.smithi001.stderr:epoch 0 2024-03-25T18:33:47.809 INFO:teuthology.misc.health.smithi001.stderr:fsid c8ea72cf-9e99-4383-9b52-6c23d365fdfc 2024-03-25T18:33:47.809 INFO:teuthology.misc.health.smithi001.stderr:last_changed 2024-03-25T18:33:47.808338+0000 2024-03-25T18:33:47.809 INFO:teuthology.misc.health.smithi001.stderr:created 2024-03-25T18:33:47.808338+0000 2024-03-25T18:33:47.809 INFO:teuthology.misc.health.smithi001.stderr:min_mon_release 0 (unknown) 2024-03-25T18:33:47.809 INFO:teuthology.misc.health.smithi001.stderr:election_strategy: 3 2024-03-25T18:33:47.809 INFO:teuthology.misc.health.smithi001.stderr:0: v1:172.21.15.1:6789/0 mon.noname-a 2024-03-25T18:33:47.809 INFO:teuthology.misc.health.smithi001.stderr:1: v1:172.21.15.115:6789/0 mon.noname-c 2024-03-25T18:33:47.809 INFO:teuthology.misc.health.smithi001.stderr:2: v1:172.21.15.1:6790/0 mon.noname-b 2024-03-25T18:33:47.809 INFO:teuthology.misc.health.smithi001.stderr: 2024-03-25T18:33:47.811 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:33:47.808+0000 7fe03a64e640 1 Processor -- start 2024-03-25T18:33:47.811 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:33:47.808+0000 7fe03a64e640 1 -- start start 2024-03-25T18:33:47.811 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:33:47.808+0000 7fe03a64e640 10 monclient: init 2024-03-25T18:33:47.811 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:33:47.809+0000 7fe03a64e640 10 monclient: _reopen_session rank -1 2024-03-25T18:33:47.811 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:33:47.809+0000 7fe03a64e640 10 monclient: _add_conns ranks=[2,1,0] 2024-03-25T18:33:47.811 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:33:47.809+0000 7fe03a64e640 10 monclient(hunting): picked mon.noname-b con 0x7fe0340b4ff0 addr v1:172.21.15.1:6790/0 2024-03-25T18:33:47.811 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:33:47.809+0000 7fe03a64e640 10 monclient(hunting): picked mon.noname-c con 0x7fe0340b4670 addr v1:172.21.15.115:6789/0 2024-03-25T18:33:47.811 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:33:47.809+0000 7fe03a64e640 10 monclient(hunting): picked mon.noname-a con 0x7fe03413c480 addr v1:172.21.15.1:6789/0 2024-03-25T18:33:47.811 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:33:47.809+0000 7fe03a64e640 1 -- --> v1:172.21.15.1:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe0340ab0e0 con 0x7fe03413c480 2024-03-25T18:33:47.811 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:33:47.809+0000 7fe03a64e640 1 -- --> v1:172.21.15.115:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe0340ab3c0 con 0x7fe0340b4670 2024-03-25T18:33:47.811 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:33:47.809+0000 7fe03a64e640 1 -- --> v1:172.21.15.1:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe0340a9f30 con 0x7fe0340b4ff0 2024-03-25T18:33:47.811 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:33:47.809+0000 7fe03a64e640 10 monclient(hunting): _renew_subs 2024-03-25T18:33:47.811 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:33:47.809+0000 7fe03a64e640 10 monclient(hunting): authenticate will time out at 7908.279650s 2024-03-25T18:33:47.831 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:33:47.830+0000 7fe0337fe640 1 --1- >> v1:172.21.15.115:6789/0 conn(0x7fe0340b4670 0x7fe0341425c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.115:6789/0 says I am v1:172.21.15.1:47042/0 (socket says 172.21.15.1:47042) 2024-03-25T18:33:47.831 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:33:47.831+0000 7fe0337fe640 1 -- 172.21.15.1:0/3273451961 learned_addr learned my addr 172.21.15.1:0/3273451961 (peer_addr_for_me v1:172.21.15.1:0/0) 2024-03-25T18:33:47.852 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:33:47.852+0000 7fe032ffd640 1 -- 172.21.15.1:0/3273451961 <== mon.0 v1:172.21.15.1:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1414494932 0 0) 0x7fe0340ab0e0 con 0x7fe03413c480 2024-03-25T18:33:47.853 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:33:47.852+0000 7fe032ffd640 10 monclient(hunting): _negotiate 2024-03-25T18:33:47.853 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:33:47.852+0000 7fe032ffd640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:33:47.853 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:33:47.852+0000 7fe032ffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:33:47.853 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:33:47.852+0000 7fe032ffd640 10 monclient(hunting): my global_id is 4524 2024-03-25T18:33:47.853 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:33:47.852+0000 7fe032ffd640 1 -- 172.21.15.1:0/3273451961 --> v1:172.21.15.1:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fe01c003520 con 0x7fe03413c480 2024-03-25T18:33:47.863 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:33:47.862+0000 7fe032ffd640 1 -- 172.21.15.1:0/3273451961 <== mon.0 v1:172.21.15.1:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3564763660 0 0) 0x7fe01c003520 con 0x7fe03413c480 2024-03-25T18:33:47.863 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:33:47.862+0000 7fe032ffd640 1 -- 172.21.15.1:0/3273451961 >> v1:172.21.15.1:6790/0 conn(0x7fe0340b4ff0 legacy=0x7fe03413bd70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:33:47.863 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:33:47.862+0000 7fe032ffd640 1 -- 172.21.15.1:0/3273451961 >> v1:172.21.15.115:6789/0 conn(0x7fe0340b4670 legacy=0x7fe0341425c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:33:47.863 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:33:47.862+0000 7fe032ffd640 10 monclient: _finish_hunting 0 2024-03-25T18:33:47.863 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:33:47.862+0000 7fe032ffd640 1 monclient: found mon.noname-a 2024-03-25T18:33:47.863 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:33:47.862+0000 7fe032ffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:33:47.863 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:33:47.862+0000 7fe032ffd640 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.1:6789/0 2024-03-25T18:33:47.864 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:33:47.862+0000 7fe032ffd640 1 -- 172.21.15.1:0/3273451961 --> v1:172.21.15.1:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe0340a9b70 con 0x7fe03413c480 2024-03-25T18:33:47.864 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:33:47.862+0000 7fe032ffd640 10 monclient: _finish_auth 0 2024-03-25T18:33:47.864 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:33:47.862+0000 7fe032ffd640 10 monclient: _check_auth_tickets 2024-03-25T18:33:47.864 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:33:47.862+0000 7fe032ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:33:47.864 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:33:47.862+0000 7fe032ffd640 1 -- 172.21.15.1:0/3273451961 <== mon.0 v1:172.21.15.1:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1586965025 0 0) 0x7fe02c001a10 con 0x7fe03413c480 2024-03-25T18:33:47.864 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:33:47.862+0000 7fe03a64e640 5 monclient: authenticate success, global_id 4524 2024-03-25T18:33:47.864 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:33:47.862+0000 7fe03a64e640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T18:33:47.864 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:33:47.862+0000 7fe032ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:33:47.864 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:33:47.862+0000 7fe032ffd640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-03-25T18:33:47.864 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:33:47.862+0000 7fe032ffd640 10 monclient: dump: 2024-03-25T18:33:47.864 INFO:teuthology.misc.health.smithi001.stderr:epoch 1 2024-03-25T18:33:47.864 INFO:teuthology.misc.health.smithi001.stderr:fsid c8ea72cf-9e99-4383-9b52-6c23d365fdfc 2024-03-25T18:33:47.864 INFO:teuthology.misc.health.smithi001.stderr:last_changed 2024-03-25T18:25:50.510519+0000 2024-03-25T18:33:47.865 INFO:teuthology.misc.health.smithi001.stderr:created 2024-03-25T18:25:50.510519+0000 2024-03-25T18:33:47.865 INFO:teuthology.misc.health.smithi001.stderr:min_mon_release 19 (squid) 2024-03-25T18:33:47.865 INFO:teuthology.misc.health.smithi001.stderr:election_strategy: 3 2024-03-25T18:33:47.865 INFO:teuthology.misc.health.smithi001.stderr:0: v1:172.21.15.1:6789/0 mon.a 2024-03-25T18:33:47.865 INFO:teuthology.misc.health.smithi001.stderr:1: v1:172.21.15.115:6789/0 mon.b 2024-03-25T18:33:47.865 INFO:teuthology.misc.health.smithi001.stderr:2: v1:172.21.15.1:6790/0 mon.c 2024-03-25T18:33:47.865 INFO:teuthology.misc.health.smithi001.stderr: 2024-03-25T18:33:47.865 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:33:47.862+0000 7fe03a64e640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T18:33:47.927 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:33:47.926+0000 7fe032ffd640 1 -- 172.21.15.1:0/3273451961 <== mon.0 v1:172.21.15.1:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fe02c001b70 con 0x7fe03413c480 2024-03-25T18:33:47.927 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:33:47.926+0000 7fe032ffd640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:33:47.927 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:33:47.926+0000 7fe032ffd640 1 -- 172.21.15.1:0/3273451961 <== mon.0 v1:172.21.15.1:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1586965025 0 0) 0x7fe02c001fc0 con 0x7fe03413c480 2024-03-25T18:33:47.927 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:33:47.926+0000 7fe032ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:33:47.928 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:33:47.926+0000 7fe032ffd640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-25T18:33:47.928 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:33:47.926+0000 7fe032ffd640 10 monclient: dump: 2024-03-25T18:33:47.928 INFO:teuthology.misc.health.smithi001.stderr:epoch 1 2024-03-25T18:33:47.928 INFO:teuthology.misc.health.smithi001.stderr:fsid c8ea72cf-9e99-4383-9b52-6c23d365fdfc 2024-03-25T18:33:47.928 INFO:teuthology.misc.health.smithi001.stderr:last_changed 2024-03-25T18:25:50.510519+0000 2024-03-25T18:33:47.928 INFO:teuthology.misc.health.smithi001.stderr:created 2024-03-25T18:25:50.510519+0000 2024-03-25T18:33:47.928 INFO:teuthology.misc.health.smithi001.stderr:min_mon_release 19 (squid) 2024-03-25T18:33:47.928 INFO:teuthology.misc.health.smithi001.stderr:election_strategy: 3 2024-03-25T18:33:47.928 INFO:teuthology.misc.health.smithi001.stderr:0: v1:172.21.15.1:6789/0 mon.a 2024-03-25T18:33:47.928 INFO:teuthology.misc.health.smithi001.stderr:1: v1:172.21.15.115:6789/0 mon.b 2024-03-25T18:33:47.928 INFO:teuthology.misc.health.smithi001.stderr:2: v1:172.21.15.1:6790/0 mon.c 2024-03-25T18:33:47.928 INFO:teuthology.misc.health.smithi001.stderr: 2024-03-25T18:33:47.928 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:33:47.926+0000 7fe03a64e640 10 monclient: get_monmap_and_config success 2024-03-25T18:33:47.928 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:33:47.926+0000 7fe03a64e640 4 set_mon_vals no callback set 2024-03-25T18:33:47.930 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:33:47.927+0000 7fe03a64e640 10 monclient: shutdown 2024-03-25T18:33:47.931 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:33:47.927+0000 7fe03a64e640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:33:47.931 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:33:47.927+0000 7fe03a64e640 1 -- 172.21.15.1:0/3273451961 >> v1:172.21.15.1:6789/0 conn(0x7fe03413c480 legacy=0x7fe03414dc00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:33:47.931 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:33:47.927+0000 7fe03a64e640 1 -- 172.21.15.1:0/3273451961 shutdown_connections 2024-03-25T18:33:47.931 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:33:47.927+0000 7fe03a64e640 1 -- 172.21.15.1:0/3273451961 >> 172.21.15.1:0/3273451961 conn(0x7fe0340617a0 msgr2=0x7fe034061bb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:33:47.931 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:33:47.927+0000 7fe03a64e640 1 -- 172.21.15.1:0/3273451961 shutdown_connections 2024-03-25T18:33:47.931 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:33:47.927+0000 7fe03a64e640 1 -- 172.21.15.1:0/3273451961 wait complete. 2024-03-25T18:33:47.931 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:33:47.928+0000 7fe03a64e640 10 monclient: build_initial_monmap 2024-03-25T18:33:47.931 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:33:47.928+0000 7fe03a64e640 10 monclient: monmap: 2024-03-25T18:33:47.931 INFO:teuthology.misc.health.smithi001.stderr:epoch 0 2024-03-25T18:33:47.931 INFO:teuthology.misc.health.smithi001.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T18:33:47.931 INFO:teuthology.misc.health.smithi001.stderr:last_changed 0.000000 2024-03-25T18:33:47.931 INFO:teuthology.misc.health.smithi001.stderr:created 0.000000 2024-03-25T18:33:47.931 INFO:teuthology.misc.health.smithi001.stderr:min_mon_release 0 (unknown) 2024-03-25T18:33:47.931 INFO:teuthology.misc.health.smithi001.stderr:election_strategy: 1 2024-03-25T18:33:47.931 INFO:teuthology.misc.health.smithi001.stderr:0: v1:172.21.15.1:6789/0 mon.noname-a 2024-03-25T18:33:47.931 INFO:teuthology.misc.health.smithi001.stderr:1: v1:172.21.15.115:6789/0 mon.noname-b 2024-03-25T18:33:47.932 INFO:teuthology.misc.health.smithi001.stderr:2: v1:172.21.15.1:6790/0 mon.noname-c 2024-03-25T18:33:47.932 INFO:teuthology.misc.health.smithi001.stderr: 2024-03-25T18:33:47.932 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:33:47.928+0000 7fe03a64e640 1 librados: starting msgr at 2024-03-25T18:33:47.932 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:33:47.928+0000 7fe03a64e640 1 librados: starting objecter 2024-03-25T18:33:47.932 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:33:47.928+0000 7fe03a64e640 1 Processor -- start 2024-03-25T18:33:47.932 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:33:47.928+0000 7fe03a64e640 1 -- start start 2024-03-25T18:33:47.932 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:33:47.928+0000 7fe03a64e640 1 librados: setting wanted keys 2024-03-25T18:33:47.932 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:33:47.928+0000 7fe03a64e640 1 librados: calling monclient init 2024-03-25T18:33:47.932 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:33:47.928+0000 7fe03a64e640 10 monclient: init 2024-03-25T18:33:47.932 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:33:47.929+0000 7fe03a64e640 10 monclient: _reopen_session rank -1 2024-03-25T18:33:47.932 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:33:47.929+0000 7fe03a64e640 10 monclient: _add_conns ranks=[2,1,0] 2024-03-25T18:33:47.932 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:33:47.929+0000 7fe03a64e640 10 monclient(hunting): picked mon.noname-c con 0x7fe0340b4670 addr v1:172.21.15.1:6790/0 2024-03-25T18:33:47.932 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:33:47.929+0000 7fe03a64e640 10 monclient(hunting): picked mon.noname-b con 0x7fe0340b4ff0 addr v1:172.21.15.115:6789/0 2024-03-25T18:33:47.932 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:33:47.929+0000 7fe03a64e640 10 monclient(hunting): picked mon.noname-a con 0x7fe03413c480 addr v1:172.21.15.1:6789/0 2024-03-25T18:33:47.932 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:33:47.929+0000 7fe03a64e640 1 -- --> v1:172.21.15.1:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe0341dbf50 con 0x7fe03413c480 2024-03-25T18:33:47.932 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:33:47.929+0000 7fe03a64e640 1 -- --> v1:172.21.15.115:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe034144730 con 0x7fe0340b4ff0 2024-03-25T18:33:47.932 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:33:47.929+0000 7fe03a64e640 1 -- --> v1:172.21.15.1:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe034144a10 con 0x7fe0340b4670 2024-03-25T18:33:47.933 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:33:47.929+0000 7fe03a64e640 10 monclient(hunting): _renew_subs 2024-03-25T18:33:47.933 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:33:47.929+0000 7fe03a64e640 10 monclient(hunting): authenticate will time out at 7908.399201s 2024-03-25T18:33:47.938 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:33:47.937+0000 7fe0337fe640 1 --1- >> v1:172.21.15.115:6789/0 conn(0x7fe0340b4ff0 0x7fe0341db130 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.115:6789/0 says I am v1:172.21.15.1:47082/0 (socket says 172.21.15.1:47082) 2024-03-25T18:33:47.938 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:33:47.937+0000 7fe0337fe640 1 -- 172.21.15.1:0/1270629975 learned_addr learned my addr 172.21.15.1:0/1270629975 (peer_addr_for_me v1:172.21.15.1:0/0) 2024-03-25T18:33:47.945 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:33:47.944+0000 7fe030ff9640 10 client.?.objecter ms_handle_connect 0x7fe0340b4ff0 2024-03-25T18:33:47.945 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:33:47.944+0000 7fe030ff9640 10 client.?.objecter resend_mon_ops 2024-03-25T18:33:47.960 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:33:47.960+0000 7fe030ff9640 10 client.?.objecter ms_handle_connect 0x7fe03413c480 2024-03-25T18:33:47.960 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:33:47.960+0000 7fe030ff9640 10 client.?.objecter resend_mon_ops 2024-03-25T18:33:47.972 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:33:47.972+0000 7fe030ff9640 1 -- 172.21.15.1:0/1270629975 <== mon.1 v1:172.21.15.115:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3028685009 0 0) 0x7fe034144730 con 0x7fe0340b4ff0 2024-03-25T18:33:47.973 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:33:47.972+0000 7fe030ff9640 10 monclient(hunting): _negotiate 2024-03-25T18:33:47.973 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:33:47.972+0000 7fe030ff9640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:33:47.973 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:33:47.972+0000 7fe030ff9640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:33:47.973 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:33:47.972+0000 7fe030ff9640 10 monclient(hunting): my global_id is 4537 2024-03-25T18:33:47.973 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:33:47.972+0000 7fe030ff9640 1 -- 172.21.15.1:0/1270629975 --> v1:172.21.15.115:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fe020004eb0 con 0x7fe0340b4ff0 2024-03-25T18:33:47.982 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:33:47.981+0000 7fe030ff9640 1 -- 172.21.15.1:0/1270629975 <== mon.0 v1:172.21.15.1:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4170110303 0 0) 0x7fe0341dbf50 con 0x7fe03413c480 2024-03-25T18:33:47.982 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:33:47.981+0000 7fe030ff9640 10 monclient(hunting): _negotiate 2024-03-25T18:33:47.982 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:33:47.981+0000 7fe030ff9640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:33:47.982 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:33:47.981+0000 7fe030ff9640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:33:47.982 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:33:47.981+0000 7fe030ff9640 10 monclient(hunting): my global_id is 4527 2024-03-25T18:33:47.983 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:33:47.981+0000 7fe030ff9640 1 -- 172.21.15.1:0/1270629975 --> v1:172.21.15.1:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fe034144730 con 0x7fe03413c480 2024-03-25T18:33:47.991 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:33:47.990+0000 7fe030ff9640 1 -- 172.21.15.1:0/1270629975 <== mon.0 v1:172.21.15.1:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1618198614 0 0) 0x7fe034144730 con 0x7fe03413c480 2024-03-25T18:33:47.991 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:33:47.990+0000 7fe030ff9640 1 -- 172.21.15.1:0/1270629975 --> v1:172.21.15.1:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fe0341dbf50 con 0x7fe03413c480 2024-03-25T18:33:47.991 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:33:47.990+0000 7fe030ff9640 1 -- 172.21.15.1:0/1270629975 <== mon.0 v1:172.21.15.1:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1586965025 0 0) 0x7fe02c001ec0 con 0x7fe03413c480 2024-03-25T18:33:47.991 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:33:47.990+0000 7fe030ff9640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:33:47.991 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:33:47.990+0000 7fe030ff9640 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2024-03-25T18:33:47.991 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:33:47.990+0000 7fe030ff9640 10 monclient(hunting): dump: 2024-03-25T18:33:47.991 INFO:teuthology.misc.health.smithi001.stderr:epoch 1 2024-03-25T18:33:47.991 INFO:teuthology.misc.health.smithi001.stderr:fsid c8ea72cf-9e99-4383-9b52-6c23d365fdfc 2024-03-25T18:33:47.991 INFO:teuthology.misc.health.smithi001.stderr:last_changed 2024-03-25T18:25:50.510519+0000 2024-03-25T18:33:47.991 INFO:teuthology.misc.health.smithi001.stderr:created 2024-03-25T18:25:50.510519+0000 2024-03-25T18:33:47.992 INFO:teuthology.misc.health.smithi001.stderr:min_mon_release 19 (squid) 2024-03-25T18:33:47.992 INFO:teuthology.misc.health.smithi001.stderr:election_strategy: 3 2024-03-25T18:33:47.992 INFO:teuthology.misc.health.smithi001.stderr:0: v1:172.21.15.1:6789/0 mon.a 2024-03-25T18:33:47.992 INFO:teuthology.misc.health.smithi001.stderr:1: v1:172.21.15.115:6789/0 mon.b 2024-03-25T18:33:47.992 INFO:teuthology.misc.health.smithi001.stderr:2: v1:172.21.15.1:6790/0 mon.c 2024-03-25T18:33:47.992 INFO:teuthology.misc.health.smithi001.stderr: 2024-03-25T18:33:47.992 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:33:47.990+0000 7fe030ff9640 10 monclient(hunting): _finish_auth 0 2024-03-25T18:33:48.000 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:33:47.999+0000 7fe030ff9640 10 client.?.objecter ms_handle_connect 0x7fe0340b4670 2024-03-25T18:33:48.000 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:33:47.999+0000 7fe030ff9640 10 client.?.objecter resend_mon_ops 2024-03-25T18:33:48.003 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:33:48.002+0000 7fe030ff9640 1 -- 172.21.15.1:0/1270629975 <== mon.1 v1:172.21.15.115:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2604392250 0 0) 0x7fe020004eb0 con 0x7fe0340b4ff0 2024-03-25T18:33:48.003 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:33:48.002+0000 7fe030ff9640 1 -- 172.21.15.1:0/1270629975 --> v1:172.21.15.115:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fe034144730 con 0x7fe0340b4ff0 2024-03-25T18:33:48.003 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:33:48.002+0000 7fe030ff9640 1 -- 172.21.15.1:0/1270629975 <== mon.1 v1:172.21.15.115:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1586965025 0 0) 0x7fe0280018b0 con 0x7fe0340b4ff0 2024-03-25T18:33:48.004 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:33:48.002+0000 7fe030ff9640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:33:48.004 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:33:48.002+0000 7fe030ff9640 10 monclient(hunting): got monmap 1 from mon.b (according to old e1) 2024-03-25T18:33:48.004 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:33:48.002+0000 7fe030ff9640 10 monclient(hunting): dump: 2024-03-25T18:33:48.004 INFO:teuthology.misc.health.smithi001.stderr:epoch 1 2024-03-25T18:33:48.004 INFO:teuthology.misc.health.smithi001.stderr:fsid c8ea72cf-9e99-4383-9b52-6c23d365fdfc 2024-03-25T18:33:48.004 INFO:teuthology.misc.health.smithi001.stderr:last_changed 2024-03-25T18:25:50.510519+0000 2024-03-25T18:33:48.004 INFO:teuthology.misc.health.smithi001.stderr:created 2024-03-25T18:25:50.510519+0000 2024-03-25T18:33:48.004 INFO:teuthology.misc.health.smithi001.stderr:min_mon_release 19 (squid) 2024-03-25T18:33:48.004 INFO:teuthology.misc.health.smithi001.stderr:election_strategy: 3 2024-03-25T18:33:48.004 INFO:teuthology.misc.health.smithi001.stderr:0: v1:172.21.15.1:6789/0 mon.a 2024-03-25T18:33:48.004 INFO:teuthology.misc.health.smithi001.stderr:1: v1:172.21.15.115:6789/0 mon.b 2024-03-25T18:33:48.004 INFO:teuthology.misc.health.smithi001.stderr:2: v1:172.21.15.1:6790/0 mon.c 2024-03-25T18:33:48.004 INFO:teuthology.misc.health.smithi001.stderr: 2024-03-25T18:33:48.010 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:33:48.008+0000 7fe030ff9640 1 -- 172.21.15.1:0/1270629975 <== mon.0 v1:172.21.15.1:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 348273689 0 0) 0x7fe0341dbf50 con 0x7fe03413c480 2024-03-25T18:33:48.010 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:33:48.008+0000 7fe030ff9640 1 -- 172.21.15.1:0/1270629975 >> v1:172.21.15.1:6790/0 conn(0x7fe0340b4670 legacy=0x7fe034144e30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:33:48.010 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:33:48.008+0000 7fe030ff9640 1 -- 172.21.15.1:0/1270629975 >> v1:172.21.15.115:6789/0 conn(0x7fe0340b4ff0 legacy=0x7fe0341db130 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:33:48.010 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:33:48.008+0000 7fe030ff9640 10 monclient: _finish_hunting 0 2024-03-25T18:33:48.010 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:33:48.008+0000 7fe030ff9640 1 monclient: found mon.a 2024-03-25T18:33:48.010 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:33:48.008+0000 7fe030ff9640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:33:48.010 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:33:48.008+0000 7fe030ff9640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.1:6789/0 2024-03-25T18:33:48.010 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:33:48.008+0000 7fe030ff9640 1 -- 172.21.15.1:0/1270629975 --> v1:172.21.15.1:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe03413eb00 con 0x7fe03413c480 2024-03-25T18:33:48.011 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:33:48.008+0000 7fe030ff9640 10 monclient: _finish_auth 0 2024-03-25T18:33:48.011 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:33:48.008+0000 7fe030ff9640 10 monclient: _check_auth_tickets 2024-03-25T18:33:48.011 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:33:48.008+0000 7fe030ff9640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:33:48.011 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:33:48.008+0000 7fe03a64e640 5 monclient: authenticate success, global_id 4527 2024-03-25T18:33:48.011 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:33:48.008+0000 7fe03a64e640 10 monclient: _renew_subs 2024-03-25T18:33:48.011 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:33:48.008+0000 7fe03a64e640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.1:6789/0 2024-03-25T18:33:48.011 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:33:48.008+0000 7fe03a64e640 1 -- 172.21.15.1:0/1270629975 --> v1:172.21.15.1:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe034145540 con 0x7fe03413c480 2024-03-25T18:33:48.011 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:33:48.009+0000 7fe03a64e640 10 client.4527.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T18:33:48.011 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:33:48.009+0000 7fe03a64e640 10 monclient: _renew_subs 2024-03-25T18:33:48.011 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:33:48.009+0000 7fe03a64e640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.1:6789/0 2024-03-25T18:33:48.011 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:33:48.009+0000 7fe03a64e640 1 -- 172.21.15.1:0/1270629975 --> v1:172.21.15.1:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7fe03413f730 con 0x7fe03413c480 2024-03-25T18:33:48.011 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:33:48.009+0000 7fe03a64e640 1 librados: init done 2024-03-25T18:33:48.012 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:33:48.010+0000 7fe03a64e640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T18:33:48.012 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:33:48.010+0000 7fe03a64e640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:33:48.012 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:33:48.010+0000 7fe03a64e640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.1:6789/0 2024-03-25T18:33:48.012 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:33:48.010+0000 7fe03a64e640 1 -- 172.21.15.1:0/1270629975 --> v1:172.21.15.1:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe03406dbd0 con 0x7fe03413c480 2024-03-25T18:33:48.337 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:33:48.335+0000 7fe030ff9640 1 -- 172.21.15.1:0/1270629975 <== mon.0 v1:172.21.15.1:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fe02c003c20 con 0x7fe03413c480 2024-03-25T18:33:48.337 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:33:48.335+0000 7fe030ff9640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:33:48.337 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:33:48.335+0000 7fe030ff9640 1 -- 172.21.15.1:0/1270629975 <== mon.0 v1:172.21.15.1:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1586965025 0 0) 0x7fe02c002100 con 0x7fe03413c480 2024-03-25T18:33:48.337 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:33:48.335+0000 7fe030ff9640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:33:48.337 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:33:48.335+0000 7fe030ff9640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-25T18:33:48.337 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:33:48.335+0000 7fe030ff9640 10 monclient: dump: 2024-03-25T18:33:48.337 INFO:teuthology.misc.health.smithi001.stderr:epoch 1 2024-03-25T18:33:48.337 INFO:teuthology.misc.health.smithi001.stderr:fsid c8ea72cf-9e99-4383-9b52-6c23d365fdfc 2024-03-25T18:33:48.337 INFO:teuthology.misc.health.smithi001.stderr:last_changed 2024-03-25T18:25:50.510519+0000 2024-03-25T18:33:48.337 INFO:teuthology.misc.health.smithi001.stderr:created 2024-03-25T18:25:50.510519+0000 2024-03-25T18:33:48.337 INFO:teuthology.misc.health.smithi001.stderr:min_mon_release 19 (squid) 2024-03-25T18:33:48.338 INFO:teuthology.misc.health.smithi001.stderr:election_strategy: 3 2024-03-25T18:33:48.338 INFO:teuthology.misc.health.smithi001.stderr:0: v1:172.21.15.1:6789/0 mon.a 2024-03-25T18:33:48.338 INFO:teuthology.misc.health.smithi001.stderr:1: v1:172.21.15.115:6789/0 mon.b 2024-03-25T18:33:48.338 INFO:teuthology.misc.health.smithi001.stderr:2: v1:172.21.15.1:6790/0 mon.c 2024-03-25T18:33:48.338 INFO:teuthology.misc.health.smithi001.stderr: 2024-03-25T18:33:48.338 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:33:48.335+0000 7fe0327fc640 4 set_mon_vals no callback set 2024-03-25T18:33:48.339 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:33:48.336+0000 7fe030ff9640 1 -- 172.21.15.1:0/1270629975 <== mon.0 v1:172.21.15.1:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 21787975 0 0) 0x7fe02c019da0 con 0x7fe03413c480 2024-03-25T18:33:48.339 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:33:48.337+0000 7fe033fff640 10 monclient: get_auth_request con 0x7fe02006ddf0 auth_method 0 2024-03-25T18:33:48.339 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:33:48.337+0000 7fe030ff9640 10 client.4527.objecter ms_handle_connect 0x7fe02006ddf0 2024-03-25T18:33:48.348 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:33:48.347+0000 7fe030ff9640 1 -- 172.21.15.1:0/1270629975 <== mon.0 v1:172.21.15.1:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 996254230 0 0) 0x7fe02c0539f0 con 0x7fe03413c480 2024-03-25T18:33:48.348 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:33:48.347+0000 7fe030ff9640 10 client.4527.objecter ms_dispatch 0x7fe034000b90 osd_map(10..10 src has 1..10) v4 2024-03-25T18:33:48.348 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:33:48.347+0000 7fe030ff9640 3 client.4527.objecter handle_osd_map got epochs [10,10] > 0 2024-03-25T18:33:48.349 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:33:48.347+0000 7fe030ff9640 3 client.4527.objecter handle_osd_map decoding full epoch 10 2024-03-25T18:33:48.349 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:33:48.347+0000 7fe030ff9640 20 client.4527.objecter dump_active .. 0 homeless 2024-03-25T18:33:48.349 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:33:48.347+0000 7fe030ff9640 1 -- 172.21.15.1:0/1270629975 <== mon.0 v1:172.21.15.1:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7fe02c04f330 con 0x7fe03413c480 2024-03-25T18:33:48.349 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:33:48.347+0000 7fe030ff9640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:33:48.349 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:33:48.347+0000 7fe030ff9640 10 monclient: _finish_command 1 = system:0 2024-03-25T18:33:48.612 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:33:48.612+0000 7fe03a64e640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:33:48.613 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:33:48.612+0000 7fe03a64e640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:33:48.613 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:33:48.612+0000 7fe03a64e640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.1:6789/0 2024-03-25T18:33:48.613 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:33:48.612+0000 7fe03a64e640 1 -- 172.21.15.1:0/1270629975 --> v1:172.21.15.1:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fe0340688c0 con 0x7fe03413c480 2024-03-25T18:33:48.636 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:33:48.635+0000 7fe030ff9640 1 -- 172.21.15.1:0/1270629975 <== mon.0 v1:172.21.15.1:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 4092213348) 0x7fe02c051430 con 0x7fe03413c480 2024-03-25T18:33:48.637 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:33:48.636+0000 7fe030ff9640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:33:48.637 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:33:48.636+0000 7fe030ff9640 10 monclient: _finish_command 2 = system:0 2024-03-25T18:33:48.637 INFO:teuthology.misc.health.smithi001.stdout: 2024-03-25T18:33:48.637 INFO:teuthology.misc.health.smithi001.stdout:{"epoch":10,"fsid":"c8ea72cf-9e99-4383-9b52-6c23d365fdfc","created":"2024-03-25T18:26:38.856638+0000","modified":"2024-03-25T18:26:44.224208+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T18:26:44.224208+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"b9f34922-3a84-49be-8426-807c1eceaefc","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"42bf306f-55e3-4634-9874-8b4b73ac6644","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"292cb23a-18a9-4207-a7cc-4d5727feaec0","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"45e0e9ab-898d-4673-9590-b4117bc5e84a","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"831a5a09-73ab-4a61-9872-088e76bdbc66","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"88cd903d-3b55-4840-9396-6e3cfe005282","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"89db7355-bc9d-46f4-8ab0-e8d9e744e026","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"cac47bee-0914-4f08-896f-348838882866","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T18:33:48.639 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:33:48.638+0000 7fe03a64e640 10 librados: watch_flush enter 2024-03-25T18:33:48.639 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:33:48.638+0000 7fe03a64e640 10 librados: watch_flush exit 2024-03-25T18:33:48.639 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:33:48.638+0000 7fe03a64e640 20 client.4527.objecter shutdown clearing up homeless session... 2024-03-25T18:33:48.639 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:33:48.638+0000 7fe03a64e640 10 client.4527.objecter successfully canceled tick 2024-03-25T18:33:48.639 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:33:48.638+0000 7fe03a64e640 1 -- 172.21.15.1:0/1270629975 >> v1:172.21.15.115:6800/4237379176 conn(0x7fe02006ddf0 legacy=0x7fe0200702b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:33:48.639 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:33:48.638+0000 7fe03a64e640 10 monclient: shutdown 2024-03-25T18:33:48.640 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:33:48.638+0000 7fe03a64e640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:33:48.640 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:33:48.638+0000 7fe03a64e640 1 -- 172.21.15.1:0/1270629975 >> v1:172.21.15.1:6789/0 conn(0x7fe03413c480 legacy=0x7fe0341db840 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:33:48.640 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:33:48.638+0000 7fe03a64e640 1 -- 172.21.15.1:0/1270629975 shutdown_connections 2024-03-25T18:33:48.640 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:33:48.638+0000 7fe03a64e640 1 -- 172.21.15.1:0/1270629975 >> 172.21.15.1:0/1270629975 conn(0x7fe0340617a0 msgr2=0x7fe034140760 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:33:48.640 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:33:48.638+0000 7fe03a64e640 1 -- 172.21.15.1:0/1270629975 shutdown_connections 2024-03-25T18:33:48.640 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:33:48.638+0000 7fe03a64e640 1 -- 172.21.15.1:0/1270629975 wait complete. 2024-03-25T18:33:48.640 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:33:48.638+0000 7fe03a64e640 1 librados: shutdown 2024-03-25T18:33:48.651 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T18:33:55.459 DEBUG:teuthology.orchestra.run.smithi001:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T18:33:55.573 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:33:55.571+0000 7fd319983640 10 monclient: get_monmap_and_config 2024-03-25T18:33:55.573 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:33:55.571+0000 7fd319983640 10 monclient: build_initial_monmap 2024-03-25T18:33:55.573 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:33:55.571+0000 7fd319983640 10 monclient: monmap: 2024-03-25T18:33:55.573 INFO:teuthology.misc.health.smithi001.stderr:epoch 0 2024-03-25T18:33:55.573 INFO:teuthology.misc.health.smithi001.stderr:fsid c8ea72cf-9e99-4383-9b52-6c23d365fdfc 2024-03-25T18:33:55.574 INFO:teuthology.misc.health.smithi001.stderr:last_changed 2024-03-25T18:33:55.572721+0000 2024-03-25T18:33:55.574 INFO:teuthology.misc.health.smithi001.stderr:created 2024-03-25T18:33:55.572721+0000 2024-03-25T18:33:55.574 INFO:teuthology.misc.health.smithi001.stderr:min_mon_release 0 (unknown) 2024-03-25T18:33:55.574 INFO:teuthology.misc.health.smithi001.stderr:election_strategy: 3 2024-03-25T18:33:55.574 INFO:teuthology.misc.health.smithi001.stderr:0: v1:172.21.15.1:6789/0 mon.noname-a 2024-03-25T18:33:55.574 INFO:teuthology.misc.health.smithi001.stderr:1: v1:172.21.15.115:6789/0 mon.noname-c 2024-03-25T18:33:55.574 INFO:teuthology.misc.health.smithi001.stderr:2: v1:172.21.15.1:6790/0 mon.noname-b 2024-03-25T18:33:55.574 INFO:teuthology.misc.health.smithi001.stderr: 2024-03-25T18:33:55.576 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:33:55.572+0000 7fd319983640 1 Processor -- start 2024-03-25T18:33:55.576 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:33:55.572+0000 7fd319983640 1 -- start start 2024-03-25T18:33:55.576 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:33:55.572+0000 7fd319983640 10 monclient: init 2024-03-25T18:33:55.576 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:33:55.572+0000 7fd319983640 10 monclient: _reopen_session rank -1 2024-03-25T18:33:55.576 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:33:55.572+0000 7fd319983640 10 monclient: _add_conns ranks=[0,1,2] 2024-03-25T18:33:55.576 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:33:55.572+0000 7fd319983640 10 monclient(hunting): picked mon.noname-a con 0x7fd3140b4ff0 addr v1:172.21.15.1:6789/0 2024-03-25T18:33:55.577 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:33:55.572+0000 7fd319983640 10 monclient(hunting): picked mon.noname-c con 0x7fd3140b4670 addr v1:172.21.15.115:6789/0 2024-03-25T18:33:55.577 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:33:55.573+0000 7fd319983640 10 monclient(hunting): picked mon.noname-b con 0x7fd31413c480 addr v1:172.21.15.1:6790/0 2024-03-25T18:33:55.577 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:33:55.573+0000 7fd319983640 1 -- --> v1:172.21.15.1:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd3140ab0e0 con 0x7fd3140b4ff0 2024-03-25T18:33:55.577 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:33:55.573+0000 7fd319983640 1 -- --> v1:172.21.15.115:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd3140ab3c0 con 0x7fd3140b4670 2024-03-25T18:33:55.577 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:33:55.573+0000 7fd319983640 1 -- --> v1:172.21.15.1:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd3140a9f30 con 0x7fd31413c480 2024-03-25T18:33:55.577 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:33:55.573+0000 7fd319983640 10 monclient(hunting): _renew_subs 2024-03-25T18:33:55.577 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:33:55.573+0000 7fd319983640 10 monclient(hunting): authenticate will time out at 7916.043720s 2024-03-25T18:33:55.591 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:33:55.589+0000 7fd3137fe640 1 --1- >> v1:172.21.15.1:6790/0 conn(0x7fd31413c480 0x7fd31414dc00 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.1:6790/0 says I am v1:172.21.15.1:39634/0 (socket says 172.21.15.1:39634) 2024-03-25T18:33:55.591 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:33:55.589+0000 7fd3137fe640 1 -- 172.21.15.1:0/2966106829 learned_addr learned my addr 172.21.15.1:0/2966106829 (peer_addr_for_me v1:172.21.15.1:0/0) 2024-03-25T18:33:55.605 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:33:55.603+0000 7fd3127fc640 1 -- 172.21.15.1:0/2966106829 <== mon.0 v1:172.21.15.1:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2330752209 0 0) 0x7fd3140ab0e0 con 0x7fd3140b4ff0 2024-03-25T18:33:55.605 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:33:55.603+0000 7fd3127fc640 10 monclient(hunting): _negotiate 2024-03-25T18:33:55.605 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:33:55.603+0000 7fd3127fc640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:33:55.605 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:33:55.603+0000 7fd3127fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:33:55.606 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:33:55.603+0000 7fd3127fc640 10 monclient(hunting): my global_id is 4530 2024-03-25T18:33:55.606 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:33:55.603+0000 7fd3127fc640 1 -- 172.21.15.1:0/2966106829 --> v1:172.21.15.1:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fd300003520 con 0x7fd3140b4ff0 2024-03-25T18:33:55.607 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:33:55.604+0000 7fd3127fc640 1 -- 172.21.15.1:0/2966106829 <== mon.1 v1:172.21.15.115:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3784831834 0 0) 0x7fd3140ab3c0 con 0x7fd3140b4670 2024-03-25T18:33:55.607 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:33:55.604+0000 7fd3127fc640 10 monclient(hunting): _negotiate 2024-03-25T18:33:55.607 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:33:55.604+0000 7fd3127fc640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:33:55.607 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:33:55.604+0000 7fd3127fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:33:55.607 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:33:55.604+0000 7fd3127fc640 10 monclient(hunting): my global_id is 4540 2024-03-25T18:33:55.607 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:33:55.605+0000 7fd3127fc640 1 -- 172.21.15.1:0/2966106829 --> v1:172.21.15.115:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fd3140ab0e0 con 0x7fd3140b4670 2024-03-25T18:33:55.608 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:33:55.605+0000 7fd3127fc640 1 -- 172.21.15.1:0/2966106829 <== mon.2 v1:172.21.15.1:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3670634061 0 0) 0x7fd3140a9f30 con 0x7fd31413c480 2024-03-25T18:33:55.608 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:33:55.605+0000 7fd3127fc640 10 monclient(hunting): _negotiate 2024-03-25T18:33:55.608 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:33:55.605+0000 7fd3127fc640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:33:55.608 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:33:55.605+0000 7fd3127fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:33:55.608 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:33:55.605+0000 7fd3127fc640 10 monclient(hunting): my global_id is 4544 2024-03-25T18:33:55.608 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:33:55.605+0000 7fd3127fc640 1 -- 172.21.15.1:0/2966106829 --> v1:172.21.15.1:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fd3140ab3c0 con 0x7fd31413c480 2024-03-25T18:33:55.616 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:33:55.613+0000 7fd3127fc640 1 -- 172.21.15.1:0/2966106829 <== mon.1 v1:172.21.15.115:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2687656531 0 0) 0x7fd3140ab0e0 con 0x7fd3140b4670 2024-03-25T18:33:55.616 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:33:55.613+0000 7fd3127fc640 1 -- 172.21.15.1:0/2966106829 >> v1:172.21.15.1:6790/0 conn(0x7fd31413c480 legacy=0x7fd31414dc00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:33:55.616 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:33:55.613+0000 7fd3127fc640 1 -- 172.21.15.1:0/2966106829 >> v1:172.21.15.1:6789/0 conn(0x7fd3140b4ff0 legacy=0x7fd31413bd70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:33:55.616 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:33:55.613+0000 7fd3127fc640 10 monclient: _finish_hunting 0 2024-03-25T18:33:55.616 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:33:55.613+0000 7fd3127fc640 1 monclient: found mon.noname-c 2024-03-25T18:33:55.616 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:33:55.613+0000 7fd3127fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:33:55.616 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:33:55.613+0000 7fd3127fc640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.115:6789/0 2024-03-25T18:33:55.616 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:33:55.613+0000 7fd3127fc640 1 -- 172.21.15.1:0/2966106829 --> v1:172.21.15.115:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd3140a9b70 con 0x7fd3140b4670 2024-03-25T18:33:55.616 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:33:55.613+0000 7fd3127fc640 10 monclient: _finish_auth 0 2024-03-25T18:33:55.616 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:33:55.613+0000 7fd3127fc640 10 monclient: _check_auth_tickets 2024-03-25T18:33:55.617 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:33:55.613+0000 7fd3127fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:33:55.617 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:33:55.613+0000 7fd3127fc640 1 -- 172.21.15.1:0/2966106829 <== mon.1 v1:172.21.15.115:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1586965025 0 0) 0x7fd2fc001670 con 0x7fd3140b4670 2024-03-25T18:33:55.617 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:33:55.613+0000 7fd319983640 5 monclient: authenticate success, global_id 4540 2024-03-25T18:33:55.617 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:33:55.613+0000 7fd319983640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T18:33:55.617 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:33:55.613+0000 7fd3127fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:33:55.617 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:33:55.613+0000 7fd3127fc640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-03-25T18:33:55.617 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:33:55.613+0000 7fd3127fc640 10 monclient: dump: 2024-03-25T18:33:55.617 INFO:teuthology.misc.health.smithi001.stderr:epoch 1 2024-03-25T18:33:55.617 INFO:teuthology.misc.health.smithi001.stderr:fsid c8ea72cf-9e99-4383-9b52-6c23d365fdfc 2024-03-25T18:33:55.617 INFO:teuthology.misc.health.smithi001.stderr:last_changed 2024-03-25T18:25:50.510519+0000 2024-03-25T18:33:55.617 INFO:teuthology.misc.health.smithi001.stderr:created 2024-03-25T18:25:50.510519+0000 2024-03-25T18:33:55.618 INFO:teuthology.misc.health.smithi001.stderr:min_mon_release 19 (squid) 2024-03-25T18:33:55.618 INFO:teuthology.misc.health.smithi001.stderr:election_strategy: 3 2024-03-25T18:33:55.618 INFO:teuthology.misc.health.smithi001.stderr:0: v1:172.21.15.1:6789/0 mon.a 2024-03-25T18:33:55.618 INFO:teuthology.misc.health.smithi001.stderr:1: v1:172.21.15.115:6789/0 mon.b 2024-03-25T18:33:55.618 INFO:teuthology.misc.health.smithi001.stderr:2: v1:172.21.15.1:6790/0 mon.c 2024-03-25T18:33:55.618 INFO:teuthology.misc.health.smithi001.stderr: 2024-03-25T18:33:55.618 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:33:55.613+0000 7fd319983640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T18:33:55.623 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:33:55.621+0000 7fd3127fc640 1 -- 172.21.15.1:0/2966106829 <== mon.1 v1:172.21.15.115:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fd2fc0017d0 con 0x7fd3140b4670 2024-03-25T18:33:55.623 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:33:55.621+0000 7fd3127fc640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:33:55.623 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:33:55.621+0000 7fd3127fc640 1 -- 172.21.15.1:0/2966106829 <== mon.1 v1:172.21.15.115:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1586965025 0 0) 0x7fd2fc001be0 con 0x7fd3140b4670 2024-03-25T18:33:55.623 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:33:55.621+0000 7fd319983640 10 monclient: get_monmap_and_config success 2024-03-25T18:33:55.624 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:33:55.621+0000 7fd319983640 4 set_mon_vals no callback set 2024-03-25T18:33:55.624 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:33:55.621+0000 7fd3127fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:33:55.624 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:33:55.621+0000 7fd3127fc640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-25T18:33:55.624 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:33:55.621+0000 7fd3127fc640 10 monclient: dump: 2024-03-25T18:33:55.624 INFO:teuthology.misc.health.smithi001.stderr:epoch 1 2024-03-25T18:33:55.624 INFO:teuthology.misc.health.smithi001.stderr:fsid c8ea72cf-9e99-4383-9b52-6c23d365fdfc 2024-03-25T18:33:55.624 INFO:teuthology.misc.health.smithi001.stderr:last_changed 2024-03-25T18:25:50.510519+0000 2024-03-25T18:33:55.624 INFO:teuthology.misc.health.smithi001.stderr:created 2024-03-25T18:25:50.510519+0000 2024-03-25T18:33:55.624 INFO:teuthology.misc.health.smithi001.stderr:min_mon_release 19 (squid) 2024-03-25T18:33:55.624 INFO:teuthology.misc.health.smithi001.stderr:election_strategy: 3 2024-03-25T18:33:55.624 INFO:teuthology.misc.health.smithi001.stderr:0: v1:172.21.15.1:6789/0 mon.a 2024-03-25T18:33:55.625 INFO:teuthology.misc.health.smithi001.stderr:1: v1:172.21.15.115:6789/0 mon.b 2024-03-25T18:33:55.625 INFO:teuthology.misc.health.smithi001.stderr:2: v1:172.21.15.1:6790/0 mon.c 2024-03-25T18:33:55.625 INFO:teuthology.misc.health.smithi001.stderr: 2024-03-25T18:33:55.628 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:33:55.622+0000 7fd319983640 10 monclient: shutdown 2024-03-25T18:33:55.628 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:33:55.622+0000 7fd319983640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:33:55.628 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:33:55.622+0000 7fd319983640 1 -- 172.21.15.1:0/2966106829 >> v1:172.21.15.115:6789/0 conn(0x7fd3140b4670 legacy=0x7fd3141425c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:33:55.628 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:33:55.622+0000 7fd319983640 1 -- 172.21.15.1:0/2966106829 shutdown_connections 2024-03-25T18:33:55.628 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:33:55.622+0000 7fd319983640 1 -- 172.21.15.1:0/2966106829 >> 172.21.15.1:0/2966106829 conn(0x7fd3140617a0 msgr2=0x7fd314061bb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:33:55.628 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:33:55.622+0000 7fd319983640 1 -- 172.21.15.1:0/2966106829 shutdown_connections 2024-03-25T18:33:55.628 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:33:55.622+0000 7fd319983640 1 -- 172.21.15.1:0/2966106829 wait complete. 2024-03-25T18:33:55.628 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:33:55.623+0000 7fd319983640 10 monclient: build_initial_monmap 2024-03-25T18:33:55.628 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:33:55.623+0000 7fd319983640 10 monclient: monmap: 2024-03-25T18:33:55.628 INFO:teuthology.misc.health.smithi001.stderr:epoch 0 2024-03-25T18:33:55.629 INFO:teuthology.misc.health.smithi001.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T18:33:55.629 INFO:teuthology.misc.health.smithi001.stderr:last_changed 0.000000 2024-03-25T18:33:55.629 INFO:teuthology.misc.health.smithi001.stderr:created 0.000000 2024-03-25T18:33:55.629 INFO:teuthology.misc.health.smithi001.stderr:min_mon_release 0 (unknown) 2024-03-25T18:33:55.629 INFO:teuthology.misc.health.smithi001.stderr:election_strategy: 1 2024-03-25T18:33:55.629 INFO:teuthology.misc.health.smithi001.stderr:0: v1:172.21.15.1:6789/0 mon.noname-a 2024-03-25T18:33:55.629 INFO:teuthology.misc.health.smithi001.stderr:1: v1:172.21.15.115:6789/0 mon.noname-b 2024-03-25T18:33:55.629 INFO:teuthology.misc.health.smithi001.stderr:2: v1:172.21.15.1:6790/0 mon.noname-c 2024-03-25T18:33:55.629 INFO:teuthology.misc.health.smithi001.stderr: 2024-03-25T18:33:55.629 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:33:55.624+0000 7fd319983640 1 librados: starting msgr at 2024-03-25T18:33:55.629 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:33:55.624+0000 7fd319983640 1 librados: starting objecter 2024-03-25T18:33:55.630 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:33:55.624+0000 7fd319983640 1 Processor -- start 2024-03-25T18:33:55.630 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:33:55.624+0000 7fd319983640 1 -- start start 2024-03-25T18:33:55.630 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:33:55.624+0000 7fd319983640 1 librados: setting wanted keys 2024-03-25T18:33:55.630 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:33:55.624+0000 7fd319983640 1 librados: calling monclient init 2024-03-25T18:33:55.630 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:33:55.624+0000 7fd319983640 10 monclient: init 2024-03-25T18:33:55.630 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:33:55.625+0000 7fd319983640 10 monclient: _reopen_session rank -1 2024-03-25T18:33:55.630 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:33:55.625+0000 7fd319983640 10 monclient: _add_conns ranks=[2,1,0] 2024-03-25T18:33:55.630 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:33:55.625+0000 7fd319983640 10 monclient(hunting): picked mon.noname-c con 0x7fd3140b4670 addr v1:172.21.15.1:6790/0 2024-03-25T18:33:55.630 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:33:55.625+0000 7fd319983640 10 monclient(hunting): picked mon.noname-b con 0x7fd3140b4ff0 addr v1:172.21.15.115:6789/0 2024-03-25T18:33:55.630 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:33:55.625+0000 7fd319983640 10 monclient(hunting): picked mon.noname-a con 0x7fd31413c480 addr v1:172.21.15.1:6789/0 2024-03-25T18:33:55.631 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:33:55.625+0000 7fd319983640 1 -- --> v1:172.21.15.1:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd3141445b0 con 0x7fd31413c480 2024-03-25T18:33:55.631 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:33:55.625+0000 7fd319983640 1 -- --> v1:172.21.15.115:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd3141461f0 con 0x7fd3140b4ff0 2024-03-25T18:33:55.631 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:33:55.625+0000 7fd319983640 1 -- --> v1:172.21.15.1:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd314144800 con 0x7fd3140b4670 2024-03-25T18:33:55.631 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:33:55.625+0000 7fd319983640 10 monclient(hunting): _renew_subs 2024-03-25T18:33:55.631 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:33:55.625+0000 7fd319983640 10 monclient(hunting): authenticate will time out at 7916.095916s 2024-03-25T18:33:55.638 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:33:55.636+0000 7fd3137fe640 1 --1- >> v1:172.21.15.1:6789/0 conn(0x7fd31413c480 0x7fd3141e2750 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.1:6789/0 says I am v1:172.21.15.1:52454/0 (socket says 172.21.15.1:52454) 2024-03-25T18:33:55.638 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:33:55.636+0000 7fd3137fe640 1 -- 172.21.15.1:0/1937451320 learned_addr learned my addr 172.21.15.1:0/1937451320 (peer_addr_for_me v1:172.21.15.1:0/0) 2024-03-25T18:33:55.638 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:33:55.636+0000 7fd310ff9640 10 client.?.objecter ms_handle_connect 0x7fd31413c480 2024-03-25T18:33:55.638 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:33:55.636+0000 7fd310ff9640 10 client.?.objecter resend_mon_ops 2024-03-25T18:33:55.644 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:33:55.642+0000 7fd310ff9640 10 client.?.objecter ms_handle_connect 0x7fd3140b4ff0 2024-03-25T18:33:55.644 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:33:55.642+0000 7fd310ff9640 10 client.?.objecter resend_mon_ops 2024-03-25T18:33:55.645 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:33:55.643+0000 7fd310ff9640 10 client.?.objecter ms_handle_connect 0x7fd3140b4670 2024-03-25T18:33:55.645 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:33:55.643+0000 7fd310ff9640 10 client.?.objecter resend_mon_ops 2024-03-25T18:33:55.646 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:33:55.644+0000 7fd310ff9640 1 -- 172.21.15.1:0/1937451320 <== mon.0 v1:172.21.15.1:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3055543791 0 0) 0x7fd3141445b0 con 0x7fd31413c480 2024-03-25T18:33:55.646 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:33:55.644+0000 7fd310ff9640 10 monclient(hunting): _negotiate 2024-03-25T18:33:55.646 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:33:55.644+0000 7fd310ff9640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:33:55.646 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:33:55.644+0000 7fd310ff9640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:33:55.646 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:33:55.644+0000 7fd310ff9640 10 monclient(hunting): my global_id is 4533 2024-03-25T18:33:55.646 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:33:55.644+0000 7fd310ff9640 1 -- 172.21.15.1:0/1937451320 --> v1:172.21.15.1:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fd2f4004eb0 con 0x7fd31413c480 2024-03-25T18:33:55.650 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:33:55.648+0000 7fd310ff9640 1 -- 172.21.15.1:0/1937451320 <== mon.1 v1:172.21.15.115:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 102926619 0 0) 0x7fd3141461f0 con 0x7fd3140b4ff0 2024-03-25T18:33:55.650 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:33:55.648+0000 7fd310ff9640 10 monclient(hunting): _negotiate 2024-03-25T18:33:55.650 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:33:55.648+0000 7fd310ff9640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:33:55.651 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:33:55.648+0000 7fd310ff9640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:33:55.651 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:33:55.648+0000 7fd310ff9640 10 monclient(hunting): my global_id is 4543 2024-03-25T18:33:55.651 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:33:55.648+0000 7fd310ff9640 1 -- 172.21.15.1:0/1937451320 --> v1:172.21.15.115:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fd3141445b0 con 0x7fd3140b4ff0 2024-03-25T18:33:55.661 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:33:55.659+0000 7fd310ff9640 1 -- 172.21.15.1:0/1937451320 <== mon.2 v1:172.21.15.1:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 154118862 0 0) 0x7fd314144800 con 0x7fd3140b4670 2024-03-25T18:33:55.661 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:33:55.659+0000 7fd310ff9640 10 monclient(hunting): _negotiate 2024-03-25T18:33:55.661 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:33:55.659+0000 7fd310ff9640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:33:55.661 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:33:55.659+0000 7fd310ff9640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:33:55.661 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:33:55.659+0000 7fd310ff9640 10 monclient(hunting): my global_id is 4547 2024-03-25T18:33:55.661 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:33:55.659+0000 7fd310ff9640 1 -- 172.21.15.1:0/1937451320 --> v1:172.21.15.1:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fd3141461f0 con 0x7fd3140b4670 2024-03-25T18:33:55.668 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:33:55.666+0000 7fd310ff9640 1 -- 172.21.15.1:0/1937451320 <== mon.1 v1:172.21.15.115:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1790262856 0 0) 0x7fd3141445b0 con 0x7fd3140b4ff0 2024-03-25T18:33:55.669 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:33:55.666+0000 7fd310ff9640 1 -- 172.21.15.1:0/1937451320 --> v1:172.21.15.115:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fd314144800 con 0x7fd3140b4ff0 2024-03-25T18:33:55.669 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:33:55.666+0000 7fd310ff9640 1 -- 172.21.15.1:0/1937451320 <== mon.1 v1:172.21.15.115:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1586965025 0 0) 0x7fd2fc001ae0 con 0x7fd3140b4ff0 2024-03-25T18:33:55.669 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:33:55.666+0000 7fd310ff9640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:33:55.669 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:33:55.666+0000 7fd310ff9640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-03-25T18:33:55.669 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:33:55.666+0000 7fd310ff9640 10 monclient(hunting): dump: 2024-03-25T18:33:55.669 INFO:teuthology.misc.health.smithi001.stderr:epoch 1 2024-03-25T18:33:55.669 INFO:teuthology.misc.health.smithi001.stderr:fsid c8ea72cf-9e99-4383-9b52-6c23d365fdfc 2024-03-25T18:33:55.669 INFO:teuthology.misc.health.smithi001.stderr:last_changed 2024-03-25T18:25:50.510519+0000 2024-03-25T18:33:55.669 INFO:teuthology.misc.health.smithi001.stderr:created 2024-03-25T18:25:50.510519+0000 2024-03-25T18:33:55.670 INFO:teuthology.misc.health.smithi001.stderr:min_mon_release 19 (squid) 2024-03-25T18:33:55.670 INFO:teuthology.misc.health.smithi001.stderr:election_strategy: 3 2024-03-25T18:33:55.670 INFO:teuthology.misc.health.smithi001.stderr:0: v1:172.21.15.1:6789/0 mon.a 2024-03-25T18:33:55.670 INFO:teuthology.misc.health.smithi001.stderr:1: v1:172.21.15.115:6789/0 mon.b 2024-03-25T18:33:55.670 INFO:teuthology.misc.health.smithi001.stderr:2: v1:172.21.15.1:6790/0 mon.c 2024-03-25T18:33:55.670 INFO:teuthology.misc.health.smithi001.stderr: 2024-03-25T18:33:55.670 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:33:55.666+0000 7fd310ff9640 10 monclient(hunting): _finish_auth 0 2024-03-25T18:33:55.673 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:33:55.671+0000 7fd310ff9640 1 -- 172.21.15.1:0/1937451320 <== mon.0 v1:172.21.15.1:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 259458573 0 0) 0x7fd2f4004eb0 con 0x7fd31413c480 2024-03-25T18:33:55.673 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:33:55.671+0000 7fd310ff9640 1 -- 172.21.15.1:0/1937451320 --> v1:172.21.15.1:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fd3141445b0 con 0x7fd31413c480 2024-03-25T18:33:55.673 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:33:55.671+0000 7fd310ff9640 1 -- 172.21.15.1:0/1937451320 <== mon.0 v1:172.21.15.1:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1586965025 0 0) 0x7fd30c001bd0 con 0x7fd31413c480 2024-03-25T18:33:55.673 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:33:55.671+0000 7fd310ff9640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:33:55.674 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:33:55.671+0000 7fd310ff9640 10 monclient(hunting): got monmap 1 from mon.a (according to old e1) 2024-03-25T18:33:55.674 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:33:55.671+0000 7fd310ff9640 10 monclient(hunting): dump: 2024-03-25T18:33:55.674 INFO:teuthology.misc.health.smithi001.stderr:epoch 1 2024-03-25T18:33:55.674 INFO:teuthology.misc.health.smithi001.stderr:fsid c8ea72cf-9e99-4383-9b52-6c23d365fdfc 2024-03-25T18:33:55.674 INFO:teuthology.misc.health.smithi001.stderr:last_changed 2024-03-25T18:25:50.510519+0000 2024-03-25T18:33:55.674 INFO:teuthology.misc.health.smithi001.stderr:created 2024-03-25T18:25:50.510519+0000 2024-03-25T18:33:55.674 INFO:teuthology.misc.health.smithi001.stderr:min_mon_release 19 (squid) 2024-03-25T18:33:55.674 INFO:teuthology.misc.health.smithi001.stderr:election_strategy: 3 2024-03-25T18:33:55.674 INFO:teuthology.misc.health.smithi001.stderr:0: v1:172.21.15.1:6789/0 mon.a 2024-03-25T18:33:55.675 INFO:teuthology.misc.health.smithi001.stderr:1: v1:172.21.15.115:6789/0 mon.b 2024-03-25T18:33:55.675 INFO:teuthology.misc.health.smithi001.stderr:2: v1:172.21.15.1:6790/0 mon.c 2024-03-25T18:33:55.675 INFO:teuthology.misc.health.smithi001.stderr: 2024-03-25T18:33:55.679 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:33:55.677+0000 7fd310ff9640 1 -- 172.21.15.1:0/1937451320 <== mon.1 v1:172.21.15.115:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 3808597119 0 0) 0x7fd314144800 con 0x7fd3140b4ff0 2024-03-25T18:33:55.679 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:33:55.677+0000 7fd310ff9640 1 -- 172.21.15.1:0/1937451320 >> v1:172.21.15.1:6790/0 conn(0x7fd3140b4670 legacy=0x7fd3141db480 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:33:55.679 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:33:55.677+0000 7fd310ff9640 1 -- 172.21.15.1:0/1937451320 >> v1:172.21.15.1:6789/0 conn(0x7fd31413c480 legacy=0x7fd3141e2750 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:33:55.679 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:33:55.677+0000 7fd310ff9640 10 monclient: _finish_hunting 0 2024-03-25T18:33:55.679 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:33:55.677+0000 7fd310ff9640 1 monclient: found mon.b 2024-03-25T18:33:55.679 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:33:55.677+0000 7fd310ff9640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:33:55.680 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:33:55.677+0000 7fd310ff9640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.115:6789/0 2024-03-25T18:33:55.680 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:33:55.677+0000 7fd310ff9640 1 -- 172.21.15.1:0/1937451320 --> v1:172.21.15.115:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd314144ae0 con 0x7fd3140b4ff0 2024-03-25T18:33:55.680 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:33:55.677+0000 7fd310ff9640 10 monclient: _finish_auth 0 2024-03-25T18:33:55.680 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:33:55.677+0000 7fd310ff9640 10 monclient: _check_auth_tickets 2024-03-25T18:33:55.680 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:33:55.677+0000 7fd310ff9640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:33:55.680 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:33:55.677+0000 7fd319983640 5 monclient: authenticate success, global_id 4543 2024-03-25T18:33:55.680 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:33:55.677+0000 7fd319983640 10 monclient: _renew_subs 2024-03-25T18:33:55.680 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:33:55.677+0000 7fd319983640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.115:6789/0 2024-03-25T18:33:55.680 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:33:55.677+0000 7fd319983640 1 -- 172.21.15.1:0/1937451320 --> v1:172.21.15.115:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd314144dc0 con 0x7fd3140b4ff0 2024-03-25T18:33:55.680 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:33:55.677+0000 7fd319983640 10 client.4543.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T18:33:55.680 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:33:55.677+0000 7fd319983640 10 monclient: _renew_subs 2024-03-25T18:33:55.681 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:33:55.677+0000 7fd319983640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.115:6789/0 2024-03-25T18:33:55.681 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:33:55.677+0000 7fd319983640 1 -- 172.21.15.1:0/1937451320 --> v1:172.21.15.115:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7fd3141466b0 con 0x7fd3140b4ff0 2024-03-25T18:33:55.681 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:33:55.677+0000 7fd319983640 1 librados: init done 2024-03-25T18:33:55.681 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:33:55.678+0000 7fd319983640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T18:33:55.681 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:33:55.678+0000 7fd319983640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:33:55.681 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:33:55.678+0000 7fd319983640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.115:6789/0 2024-03-25T18:33:55.682 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:33:55.678+0000 7fd319983640 1 -- 172.21.15.1:0/1937451320 --> v1:172.21.15.115:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd31406dbd0 con 0x7fd3140b4ff0 2024-03-25T18:33:55.698 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:33:55.695+0000 7fd310ff9640 1 -- 172.21.15.1:0/1937451320 <== mon.1 v1:172.21.15.115:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fd2fc001210 con 0x7fd3140b4ff0 2024-03-25T18:33:55.698 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:33:55.695+0000 7fd310ff9640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:33:55.698 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:33:55.695+0000 7fd310ff9640 1 -- 172.21.15.1:0/1937451320 <== mon.1 v1:172.21.15.115:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1586965025 0 0) 0x7fd2fc003a40 con 0x7fd3140b4ff0 2024-03-25T18:33:55.698 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:33:55.695+0000 7fd310ff9640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:33:55.698 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:33:55.695+0000 7fd310ff9640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-25T18:33:55.698 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:33:55.695+0000 7fd310ff9640 10 monclient: dump: 2024-03-25T18:33:55.698 INFO:teuthology.misc.health.smithi001.stderr:epoch 1 2024-03-25T18:33:55.698 INFO:teuthology.misc.health.smithi001.stderr:fsid c8ea72cf-9e99-4383-9b52-6c23d365fdfc 2024-03-25T18:33:55.698 INFO:teuthology.misc.health.smithi001.stderr:last_changed 2024-03-25T18:25:50.510519+0000 2024-03-25T18:33:55.698 INFO:teuthology.misc.health.smithi001.stderr:created 2024-03-25T18:25:50.510519+0000 2024-03-25T18:33:55.699 INFO:teuthology.misc.health.smithi001.stderr:min_mon_release 19 (squid) 2024-03-25T18:33:55.699 INFO:teuthology.misc.health.smithi001.stderr:election_strategy: 3 2024-03-25T18:33:55.699 INFO:teuthology.misc.health.smithi001.stderr:0: v1:172.21.15.1:6789/0 mon.a 2024-03-25T18:33:55.699 INFO:teuthology.misc.health.smithi001.stderr:1: v1:172.21.15.115:6789/0 mon.b 2024-03-25T18:33:55.699 INFO:teuthology.misc.health.smithi001.stderr:2: v1:172.21.15.1:6790/0 mon.c 2024-03-25T18:33:55.699 INFO:teuthology.misc.health.smithi001.stderr: 2024-03-25T18:33:55.699 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:33:55.695+0000 7fd311ffb640 4 set_mon_vals no callback set 2024-03-25T18:33:55.700 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:33:55.696+0000 7fd310ff9640 1 -- 172.21.15.1:0/1937451320 <== mon.1 v1:172.21.15.115:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 21787975 0 0) 0x7fd2fc019a10 con 0x7fd3140b4ff0 2024-03-25T18:33:55.700 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:33:55.697+0000 7fd312ffd640 10 monclient: get_auth_request con 0x7fd2f406ded0 auth_method 0 2024-03-25T18:33:55.700 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:33:55.697+0000 7fd310ff9640 10 client.4543.objecter ms_handle_connect 0x7fd2f406ded0 2024-03-25T18:33:55.701 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:33:55.699+0000 7fd310ff9640 1 -- 172.21.15.1:0/1937451320 <== mon.1 v1:172.21.15.115:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 996254230 0 0) 0x7fd2fc0536a0 con 0x7fd3140b4ff0 2024-03-25T18:33:55.701 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:33:55.699+0000 7fd310ff9640 10 client.4543.objecter ms_dispatch 0x7fd314000b90 osd_map(10..10 src has 1..10) v4 2024-03-25T18:33:55.702 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:33:55.699+0000 7fd310ff9640 3 client.4543.objecter handle_osd_map got epochs [10,10] > 0 2024-03-25T18:33:55.702 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:33:55.699+0000 7fd310ff9640 3 client.4543.objecter handle_osd_map decoding full epoch 10 2024-03-25T18:33:55.702 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:33:55.699+0000 7fd310ff9640 20 client.4543.objecter dump_active .. 0 homeless 2024-03-25T18:33:55.970 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:33:55.968+0000 7fd310ff9640 1 -- 172.21.15.1:0/1937451320 <== mon.1 v1:172.21.15.115:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7fd2fc04efe0 con 0x7fd3140b4ff0 2024-03-25T18:33:55.970 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:33:55.968+0000 7fd310ff9640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:33:55.970 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:33:55.968+0000 7fd310ff9640 10 monclient: _finish_command 1 = system:0 2024-03-25T18:33:56.226 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:33:56.224+0000 7fd319983640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:33:56.226 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:33:56.224+0000 7fd319983640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:33:56.226 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:33:56.224+0000 7fd319983640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.115:6789/0 2024-03-25T18:33:56.226 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:33:56.224+0000 7fd319983640 1 -- 172.21.15.1:0/1937451320 --> v1:172.21.15.115:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fd3140688c0 con 0x7fd3140b4ff0 2024-03-25T18:33:56.255 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:33:56.253+0000 7fd310ff9640 1 -- 172.21.15.1:0/1937451320 <== mon.1 v1:172.21.15.115:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 4092213348) 0x7fd2fc0510e0 con 0x7fd3140b4ff0 2024-03-25T18:33:56.255 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:33:56.253+0000 7fd310ff9640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:33:56.255 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:33:56.253+0000 7fd310ff9640 10 monclient: _finish_command 2 = system:0 2024-03-25T18:33:56.255 INFO:teuthology.misc.health.smithi001.stdout: 2024-03-25T18:33:56.255 INFO:teuthology.misc.health.smithi001.stdout:{"epoch":10,"fsid":"c8ea72cf-9e99-4383-9b52-6c23d365fdfc","created":"2024-03-25T18:26:38.856638+0000","modified":"2024-03-25T18:26:44.224208+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T18:26:44.224208+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"b9f34922-3a84-49be-8426-807c1eceaefc","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"42bf306f-55e3-4634-9874-8b4b73ac6644","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"292cb23a-18a9-4207-a7cc-4d5727feaec0","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"45e0e9ab-898d-4673-9590-b4117bc5e84a","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"831a5a09-73ab-4a61-9872-088e76bdbc66","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"88cd903d-3b55-4840-9396-6e3cfe005282","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"89db7355-bc9d-46f4-8ab0-e8d9e744e026","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"cac47bee-0914-4f08-896f-348838882866","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T18:33:56.258 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:33:56.256+0000 7fd319983640 10 librados: watch_flush enter 2024-03-25T18:33:56.259 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:33:56.256+0000 7fd319983640 10 librados: watch_flush exit 2024-03-25T18:33:56.259 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:33:56.256+0000 7fd319983640 20 client.4543.objecter shutdown clearing up homeless session... 2024-03-25T18:33:56.259 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:33:56.256+0000 7fd319983640 10 client.4543.objecter successfully canceled tick 2024-03-25T18:33:56.259 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:33:56.256+0000 7fd319983640 1 -- 172.21.15.1:0/1937451320 >> v1:172.21.15.115:6800/4237379176 conn(0x7fd2f406ded0 legacy=0x7fd2f4070390 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:33:56.259 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:33:56.256+0000 7fd319983640 10 monclient: shutdown 2024-03-25T18:33:56.259 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:33:56.256+0000 7fd319983640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:33:56.259 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:33:56.256+0000 7fd319983640 1 -- 172.21.15.1:0/1937451320 >> v1:172.21.15.115:6789/0 conn(0x7fd3140b4ff0 legacy=0x7fd3141dbb90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:33:56.259 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:33:56.256+0000 7fd319983640 1 -- 172.21.15.1:0/1937451320 shutdown_connections 2024-03-25T18:33:56.259 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:33:56.256+0000 7fd319983640 1 -- 172.21.15.1:0/1937451320 >> 172.21.15.1:0/1937451320 conn(0x7fd3140617a0 msgr2=0x7fd3141405c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:33:56.259 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:33:56.256+0000 7fd319983640 1 -- 172.21.15.1:0/1937451320 shutdown_connections 2024-03-25T18:33:56.259 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:33:56.256+0000 7fd319983640 1 -- 172.21.15.1:0/1937451320 wait complete. 2024-03-25T18:33:56.259 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:33:56.256+0000 7fd319983640 1 librados: shutdown 2024-03-25T18:33:56.270 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T18:34:02.315 INFO:tasks.ceph.mon.a.smithi001.stderr:==00:00:07:40.873 34876== Warning: unimplemented fcntl command: 1036 2024-03-25T18:34:02.322 INFO:tasks.ceph.mon.a.smithi001.stderr:==00:00:07:40.880 34876== Warning: unimplemented fcntl command: 1036 2024-03-25T18:34:02.334 INFO:tasks.ceph.mon.b.smithi115.stderr:==00:00:07:40.817 34836== Warning: unimplemented fcntl command: 1036 2024-03-25T18:34:02.346 INFO:tasks.ceph.mon.a.smithi001.stderr:==00:00:07:40.905 34876== Warning: unimplemented fcntl command: 1036 2024-03-25T18:34:02.386 INFO:tasks.ceph.mon.b.smithi115.stderr:==00:00:07:40.870 34836== Warning: unimplemented fcntl command: 1036 2024-03-25T18:34:02.387 INFO:tasks.ceph.mon.c.smithi001.stderr:==00:00:07:40.945 34877== Warning: unimplemented fcntl command: 1036 2024-03-25T18:34:02.395 INFO:tasks.ceph.mon.c.smithi001.stderr:==00:00:07:40.953 34877== Warning: unimplemented fcntl command: 1036 2024-03-25T18:34:02.423 INFO:tasks.ceph.mon.c.smithi001.stderr:==00:00:07:40.981 34877== Warning: unimplemented fcntl command: 1036 2024-03-25T18:34:02.427 INFO:tasks.ceph.mon.b.smithi115.stderr:==00:00:07:40.910 34836== Warning: unimplemented fcntl command: 1036 2024-03-25T18:34:03.076 DEBUG:teuthology.orchestra.run.smithi001:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T18:34:03.190 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:34:03.188+0000 7fce3e90d640 10 monclient: get_monmap_and_config 2024-03-25T18:34:03.190 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:34:03.188+0000 7fce3e90d640 10 monclient: build_initial_monmap 2024-03-25T18:34:03.190 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:34:03.188+0000 7fce3e90d640 10 monclient: monmap: 2024-03-25T18:34:03.190 INFO:teuthology.misc.health.smithi001.stderr:epoch 0 2024-03-25T18:34:03.190 INFO:teuthology.misc.health.smithi001.stderr:fsid c8ea72cf-9e99-4383-9b52-6c23d365fdfc 2024-03-25T18:34:03.191 INFO:teuthology.misc.health.smithi001.stderr:last_changed 2024-03-25T18:34:03.189935+0000 2024-03-25T18:34:03.191 INFO:teuthology.misc.health.smithi001.stderr:created 2024-03-25T18:34:03.189935+0000 2024-03-25T18:34:03.191 INFO:teuthology.misc.health.smithi001.stderr:min_mon_release 0 (unknown) 2024-03-25T18:34:03.191 INFO:teuthology.misc.health.smithi001.stderr:election_strategy: 3 2024-03-25T18:34:03.191 INFO:teuthology.misc.health.smithi001.stderr:0: v1:172.21.15.1:6789/0 mon.noname-a 2024-03-25T18:34:03.191 INFO:teuthology.misc.health.smithi001.stderr:1: v1:172.21.15.115:6789/0 mon.noname-c 2024-03-25T18:34:03.191 INFO:teuthology.misc.health.smithi001.stderr:2: v1:172.21.15.1:6790/0 mon.noname-b 2024-03-25T18:34:03.191 INFO:teuthology.misc.health.smithi001.stderr: 2024-03-25T18:34:03.192 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:34:03.189+0000 7fce3e90d640 1 Processor -- start 2024-03-25T18:34:03.192 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:34:03.189+0000 7fce3e90d640 1 -- start start 2024-03-25T18:34:03.192 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:34:03.189+0000 7fce3e90d640 10 monclient: init 2024-03-25T18:34:03.193 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:34:03.190+0000 7fce3e90d640 10 monclient: _reopen_session rank -1 2024-03-25T18:34:03.193 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:34:03.190+0000 7fce3e90d640 10 monclient: _add_conns ranks=[2,0,1] 2024-03-25T18:34:03.193 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:34:03.190+0000 7fce3e90d640 10 monclient(hunting): picked mon.noname-b con 0x7fce380b4ff0 addr v1:172.21.15.1:6790/0 2024-03-25T18:34:03.193 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:34:03.190+0000 7fce3e90d640 10 monclient(hunting): picked mon.noname-a con 0x7fce380b4670 addr v1:172.21.15.1:6789/0 2024-03-25T18:34:03.193 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:34:03.190+0000 7fce3e90d640 10 monclient(hunting): picked mon.noname-c con 0x7fce3813c480 addr v1:172.21.15.115:6789/0 2024-03-25T18:34:03.193 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:34:03.190+0000 7fce3e90d640 1 -- --> v1:172.21.15.1:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fce380ab0e0 con 0x7fce380b4670 2024-03-25T18:34:03.193 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:34:03.190+0000 7fce3e90d640 1 -- --> v1:172.21.15.115:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fce380ab3c0 con 0x7fce3813c480 2024-03-25T18:34:03.193 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:34:03.190+0000 7fce3e90d640 1 -- --> v1:172.21.15.1:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fce380a9f30 con 0x7fce380b4ff0 2024-03-25T18:34:03.193 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:34:03.190+0000 7fce3e90d640 10 monclient(hunting): _renew_subs 2024-03-25T18:34:03.193 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:34:03.190+0000 7fce3e90d640 10 monclient(hunting): authenticate will time out at 7923.661110s 2024-03-25T18:34:03.202 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:34:03.201+0000 7fce3ceaa640 1 --1- >> v1:172.21.15.115:6789/0 conn(0x7fce3813c480 0x7fce3814dc00 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.115:6789/0 says I am v1:172.21.15.1:47342/0 (socket says 172.21.15.1:47342) 2024-03-25T18:34:03.202 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:34:03.201+0000 7fce3ceaa640 1 -- 172.21.15.1:0/4224724764 learned_addr learned my addr 172.21.15.1:0/4224724764 (peer_addr_for_me v1:172.21.15.1:0/0) 2024-03-25T18:34:03.220 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:34:03.218+0000 7fce36ffd640 1 -- 172.21.15.1:0/4224724764 <== mon.2 v1:172.21.15.1:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 358818304 0 0) 0x7fce380a9f30 con 0x7fce380b4ff0 2024-03-25T18:34:03.220 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:34:03.218+0000 7fce36ffd640 10 monclient(hunting): _negotiate 2024-03-25T18:34:03.220 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:34:03.218+0000 7fce36ffd640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:34:03.220 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:34:03.218+0000 7fce36ffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:34:03.220 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:34:03.218+0000 7fce36ffd640 10 monclient(hunting): my global_id is 4550 2024-03-25T18:34:03.220 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:34:03.218+0000 7fce36ffd640 1 -- 172.21.15.1:0/4224724764 --> v1:172.21.15.1:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fce20003520 con 0x7fce380b4ff0 2024-03-25T18:34:03.221 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:34:03.219+0000 7fce36ffd640 1 -- 172.21.15.1:0/4224724764 <== mon.1 v1:172.21.15.115:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3706738462 0 0) 0x7fce380ab3c0 con 0x7fce3813c480 2024-03-25T18:34:03.221 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:34:03.219+0000 7fce36ffd640 10 monclient(hunting): _negotiate 2024-03-25T18:34:03.221 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:34:03.219+0000 7fce36ffd640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:34:03.221 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:34:03.219+0000 7fce36ffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:34:03.221 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:34:03.219+0000 7fce36ffd640 10 monclient(hunting): my global_id is 4546 2024-03-25T18:34:03.221 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:34:03.219+0000 7fce36ffd640 1 -- 172.21.15.1:0/4224724764 --> v1:172.21.15.115:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fce380a9f30 con 0x7fce3813c480 2024-03-25T18:34:03.226 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:34:03.224+0000 7fce36ffd640 1 -- 172.21.15.1:0/4224724764 <== mon.0 v1:172.21.15.1:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 61241013 0 0) 0x7fce380ab0e0 con 0x7fce380b4670 2024-03-25T18:34:03.226 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:34:03.224+0000 7fce36ffd640 10 monclient(hunting): _negotiate 2024-03-25T18:34:03.226 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:34:03.224+0000 7fce36ffd640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:34:03.226 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:34:03.224+0000 7fce36ffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:34:03.226 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:34:03.224+0000 7fce36ffd640 10 monclient(hunting): my global_id is 4536 2024-03-25T18:34:03.226 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:34:03.224+0000 7fce36ffd640 1 -- 172.21.15.1:0/4224724764 --> v1:172.21.15.1:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fce380ab3c0 con 0x7fce380b4670 2024-03-25T18:34:03.234 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:34:03.231+0000 7fce36ffd640 1 -- 172.21.15.1:0/4224724764 <== mon.2 v1:172.21.15.1:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 916421959 0 0) 0x7fce20003520 con 0x7fce380b4ff0 2024-03-25T18:34:03.234 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:34:03.231+0000 7fce36ffd640 1 -- 172.21.15.1:0/4224724764 >> v1:172.21.15.115:6789/0 conn(0x7fce3813c480 legacy=0x7fce3814dc00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:34:03.234 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:34:03.231+0000 7fce36ffd640 1 -- 172.21.15.1:0/4224724764 >> v1:172.21.15.1:6789/0 conn(0x7fce380b4670 legacy=0x7fce381425c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:34:03.234 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:34:03.231+0000 7fce36ffd640 10 monclient: _finish_hunting 0 2024-03-25T18:34:03.234 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:34:03.231+0000 7fce36ffd640 1 monclient: found mon.noname-b 2024-03-25T18:34:03.234 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:34:03.231+0000 7fce36ffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:34:03.234 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:34:03.231+0000 7fce36ffd640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.1:6790/0 2024-03-25T18:34:03.234 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:34:03.231+0000 7fce36ffd640 1 -- 172.21.15.1:0/4224724764 --> v1:172.21.15.1:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fce380a9b70 con 0x7fce380b4ff0 2024-03-25T18:34:03.234 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:34:03.231+0000 7fce36ffd640 10 monclient: _finish_auth 0 2024-03-25T18:34:03.234 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:34:03.231+0000 7fce36ffd640 10 monclient: _check_auth_tickets 2024-03-25T18:34:03.234 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:34:03.231+0000 7fce36ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:34:03.234 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:34:03.231+0000 7fce36ffd640 1 -- 172.21.15.1:0/4224724764 <== mon.2 v1:172.21.15.1:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1586965025 0 0) 0x7fce28001690 con 0x7fce380b4ff0 2024-03-25T18:34:03.234 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:34:03.231+0000 7fce36ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:34:03.235 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:34:03.231+0000 7fce36ffd640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-03-25T18:34:03.235 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:34:03.231+0000 7fce36ffd640 10 monclient: dump: 2024-03-25T18:34:03.235 INFO:teuthology.misc.health.smithi001.stderr:epoch 1 2024-03-25T18:34:03.235 INFO:teuthology.misc.health.smithi001.stderr:fsid c8ea72cf-9e99-4383-9b52-6c23d365fdfc 2024-03-25T18:34:03.235 INFO:teuthology.misc.health.smithi001.stderr:last_changed 2024-03-25T18:25:50.510519+0000 2024-03-25T18:34:03.235 INFO:teuthology.misc.health.smithi001.stderr:created 2024-03-25T18:25:50.510519+0000 2024-03-25T18:34:03.235 INFO:teuthology.misc.health.smithi001.stderr:min_mon_release 19 (squid) 2024-03-25T18:34:03.235 INFO:teuthology.misc.health.smithi001.stderr:election_strategy: 3 2024-03-25T18:34:03.235 INFO:teuthology.misc.health.smithi001.stderr:0: v1:172.21.15.1:6789/0 mon.a 2024-03-25T18:34:03.235 INFO:teuthology.misc.health.smithi001.stderr:1: v1:172.21.15.115:6789/0 mon.b 2024-03-25T18:34:03.235 INFO:teuthology.misc.health.smithi001.stderr:2: v1:172.21.15.1:6790/0 mon.c 2024-03-25T18:34:03.235 INFO:teuthology.misc.health.smithi001.stderr: 2024-03-25T18:34:03.235 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:34:03.231+0000 7fce3e90d640 5 monclient: authenticate success, global_id 4550 2024-03-25T18:34:03.235 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:34:03.231+0000 7fce3e90d640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T18:34:03.240 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:34:03.238+0000 7fce36ffd640 1 -- 172.21.15.1:0/4224724764 <== mon.2 v1:172.21.15.1:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fce280017f0 con 0x7fce380b4ff0 2024-03-25T18:34:03.240 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:34:03.238+0000 7fce36ffd640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:34:03.240 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:34:03.238+0000 7fce36ffd640 1 -- 172.21.15.1:0/4224724764 <== mon.2 v1:172.21.15.1:6790/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1586965025 0 0) 0x7fce28001c00 con 0x7fce380b4ff0 2024-03-25T18:34:03.240 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:34:03.238+0000 7fce36ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:34:03.240 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:34:03.238+0000 7fce36ffd640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-25T18:34:03.240 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:34:03.238+0000 7fce36ffd640 10 monclient: dump: 2024-03-25T18:34:03.240 INFO:teuthology.misc.health.smithi001.stderr:epoch 1 2024-03-25T18:34:03.240 INFO:teuthology.misc.health.smithi001.stderr:fsid c8ea72cf-9e99-4383-9b52-6c23d365fdfc 2024-03-25T18:34:03.240 INFO:teuthology.misc.health.smithi001.stderr:last_changed 2024-03-25T18:25:50.510519+0000 2024-03-25T18:34:03.240 INFO:teuthology.misc.health.smithi001.stderr:created 2024-03-25T18:25:50.510519+0000 2024-03-25T18:34:03.240 INFO:teuthology.misc.health.smithi001.stderr:min_mon_release 19 (squid) 2024-03-25T18:34:03.240 INFO:teuthology.misc.health.smithi001.stderr:election_strategy: 3 2024-03-25T18:34:03.240 INFO:teuthology.misc.health.smithi001.stderr:0: v1:172.21.15.1:6789/0 mon.a 2024-03-25T18:34:03.241 INFO:teuthology.misc.health.smithi001.stderr:1: v1:172.21.15.115:6789/0 mon.b 2024-03-25T18:34:03.241 INFO:teuthology.misc.health.smithi001.stderr:2: v1:172.21.15.1:6790/0 mon.c 2024-03-25T18:34:03.241 INFO:teuthology.misc.health.smithi001.stderr: 2024-03-25T18:34:03.241 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:34:03.238+0000 7fce3e90d640 10 monclient: get_monmap_and_config success 2024-03-25T18:34:03.241 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:34:03.238+0000 7fce3e90d640 4 set_mon_vals no callback set 2024-03-25T18:34:03.242 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:34:03.239+0000 7fce3e90d640 10 monclient: shutdown 2024-03-25T18:34:03.242 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:34:03.239+0000 7fce3e90d640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:34:03.242 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:34:03.239+0000 7fce3e90d640 1 -- 172.21.15.1:0/4224724764 >> v1:172.21.15.1:6790/0 conn(0x7fce380b4ff0 legacy=0x7fce3813bd70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:34:03.242 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:34:03.239+0000 7fce3e90d640 1 -- 172.21.15.1:0/4224724764 shutdown_connections 2024-03-25T18:34:03.242 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:34:03.239+0000 7fce3e90d640 1 -- 172.21.15.1:0/4224724764 >> 172.21.15.1:0/4224724764 conn(0x7fce380617a0 msgr2=0x7fce38061bb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:34:03.242 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:34:03.239+0000 7fce3e90d640 1 -- 172.21.15.1:0/4224724764 shutdown_connections 2024-03-25T18:34:03.242 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:34:03.239+0000 7fce3e90d640 1 -- 172.21.15.1:0/4224724764 wait complete. 2024-03-25T18:34:03.242 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:34:03.240+0000 7fce3e90d640 10 monclient: build_initial_monmap 2024-03-25T18:34:03.242 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:34:03.240+0000 7fce3e90d640 10 monclient: monmap: 2024-03-25T18:34:03.242 INFO:teuthology.misc.health.smithi001.stderr:epoch 0 2024-03-25T18:34:03.243 INFO:teuthology.misc.health.smithi001.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T18:34:03.243 INFO:teuthology.misc.health.smithi001.stderr:last_changed 0.000000 2024-03-25T18:34:03.243 INFO:teuthology.misc.health.smithi001.stderr:created 0.000000 2024-03-25T18:34:03.243 INFO:teuthology.misc.health.smithi001.stderr:min_mon_release 0 (unknown) 2024-03-25T18:34:03.243 INFO:teuthology.misc.health.smithi001.stderr:election_strategy: 1 2024-03-25T18:34:03.243 INFO:teuthology.misc.health.smithi001.stderr:0: v1:172.21.15.1:6789/0 mon.noname-a 2024-03-25T18:34:03.243 INFO:teuthology.misc.health.smithi001.stderr:1: v1:172.21.15.115:6789/0 mon.noname-b 2024-03-25T18:34:03.243 INFO:teuthology.misc.health.smithi001.stderr:2: v1:172.21.15.1:6790/0 mon.noname-c 2024-03-25T18:34:03.243 INFO:teuthology.misc.health.smithi001.stderr: 2024-03-25T18:34:03.243 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:34:03.240+0000 7fce3e90d640 1 librados: starting msgr at 2024-03-25T18:34:03.243 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:34:03.240+0000 7fce3e90d640 1 librados: starting objecter 2024-03-25T18:34:03.243 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:34:03.240+0000 7fce3e90d640 1 Processor -- start 2024-03-25T18:34:03.244 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:34:03.240+0000 7fce3e90d640 1 -- start start 2024-03-25T18:34:03.244 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:34:03.240+0000 7fce3e90d640 1 librados: setting wanted keys 2024-03-25T18:34:03.244 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:34:03.241+0000 7fce3e90d640 1 librados: calling monclient init 2024-03-25T18:34:03.244 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:34:03.241+0000 7fce3e90d640 10 monclient: init 2024-03-25T18:34:03.245 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:34:03.241+0000 7fce3e90d640 10 monclient: _reopen_session rank -1 2024-03-25T18:34:03.245 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:34:03.241+0000 7fce3e90d640 10 monclient: _add_conns ranks=[1,2,0] 2024-03-25T18:34:03.245 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:34:03.241+0000 7fce3e90d640 10 monclient(hunting): picked mon.noname-b con 0x7fce380b4670 addr v1:172.21.15.115:6789/0 2024-03-25T18:34:03.245 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:34:03.241+0000 7fce3e90d640 10 monclient(hunting): picked mon.noname-c con 0x7fce380b4ff0 addr v1:172.21.15.1:6790/0 2024-03-25T18:34:03.245 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:34:03.241+0000 7fce3e90d640 10 monclient(hunting): picked mon.noname-a con 0x7fce3813c480 addr v1:172.21.15.1:6789/0 2024-03-25T18:34:03.245 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:34:03.241+0000 7fce3e90d640 1 -- --> v1:172.21.15.1:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fce380abe60 con 0x7fce3813c480 2024-03-25T18:34:03.245 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:34:03.241+0000 7fce3e90d640 1 -- --> v1:172.21.15.115:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fce380ac140 con 0x7fce380b4670 2024-03-25T18:34:03.245 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:34:03.241+0000 7fce3e90d640 1 -- --> v1:172.21.15.1:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fce380ac420 con 0x7fce380b4ff0 2024-03-25T18:34:03.245 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:34:03.241+0000 7fce3e90d640 10 monclient(hunting): _renew_subs 2024-03-25T18:34:03.245 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:34:03.241+0000 7fce3e90d640 10 monclient(hunting): authenticate will time out at 7923.712268s 2024-03-25T18:34:03.250 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:34:03.248+0000 7fce3ceaa640 1 --1- >> v1:172.21.15.1:6789/0 conn(0x7fce3813c480 0x7fce381e2380 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.1:6789/0 says I am v1:172.21.15.1:36636/0 (socket says 172.21.15.1:36636) 2024-03-25T18:34:03.250 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:34:03.248+0000 7fce3ceaa640 1 -- 172.21.15.1:0/902569874 learned_addr learned my addr 172.21.15.1:0/902569874 (peer_addr_for_me v1:172.21.15.1:0/0) 2024-03-25T18:34:03.251 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:34:03.250+0000 7fce34ff9640 10 client.?.objecter ms_handle_connect 0x7fce3813c480 2024-03-25T18:34:03.251 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:34:03.250+0000 7fce34ff9640 10 client.?.objecter resend_mon_ops 2024-03-25T18:34:03.261 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:34:03.258+0000 7fce34ff9640 1 -- 172.21.15.1:0/902569874 <== mon.0 v1:172.21.15.1:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3494624246 0 0) 0x7fce380abe60 con 0x7fce3813c480 2024-03-25T18:34:03.261 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:34:03.259+0000 7fce34ff9640 10 monclient(hunting): _negotiate 2024-03-25T18:34:03.261 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:34:03.259+0000 7fce34ff9640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:34:03.261 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:34:03.259+0000 7fce34ff9640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:34:03.261 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:34:03.259+0000 7fce34ff9640 10 monclient(hunting): my global_id is 4539 2024-03-25T18:34:03.261 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:34:03.259+0000 7fce34ff9640 1 -- 172.21.15.1:0/902569874 --> v1:172.21.15.1:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fce24004eb0 con 0x7fce3813c480 2024-03-25T18:34:03.269 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:34:03.267+0000 7fce34ff9640 10 client.?.objecter ms_handle_connect 0x7fce380b4670 2024-03-25T18:34:03.269 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:34:03.267+0000 7fce34ff9640 10 client.?.objecter resend_mon_ops 2024-03-25T18:34:03.271 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:34:03.269+0000 7fce34ff9640 10 client.?.objecter ms_handle_connect 0x7fce380b4ff0 2024-03-25T18:34:03.271 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:34:03.269+0000 7fce34ff9640 10 client.?.objecter resend_mon_ops 2024-03-25T18:34:03.281 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:34:03.279+0000 7fce34ff9640 1 -- 172.21.15.1:0/902569874 <== mon.1 v1:172.21.15.115:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 390528395 0 0) 0x7fce380ac140 con 0x7fce380b4670 2024-03-25T18:34:03.281 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:34:03.279+0000 7fce34ff9640 10 monclient(hunting): _negotiate 2024-03-25T18:34:03.281 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:34:03.279+0000 7fce34ff9640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:34:03.281 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:34:03.279+0000 7fce34ff9640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:34:03.281 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:34:03.279+0000 7fce34ff9640 10 monclient(hunting): my global_id is 4549 2024-03-25T18:34:03.281 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:34:03.279+0000 7fce34ff9640 1 -- 172.21.15.1:0/902569874 --> v1:172.21.15.115:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fce380abe60 con 0x7fce380b4670 2024-03-25T18:34:03.282 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:34:03.280+0000 7fce34ff9640 1 -- 172.21.15.1:0/902569874 <== mon.0 v1:172.21.15.1:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3056076747 0 0) 0x7fce24004eb0 con 0x7fce3813c480 2024-03-25T18:34:03.282 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:34:03.280+0000 7fce34ff9640 1 -- 172.21.15.1:0/902569874 --> v1:172.21.15.1:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fce380ac140 con 0x7fce3813c480 2024-03-25T18:34:03.282 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:34:03.280+0000 7fce34ff9640 1 -- 172.21.15.1:0/902569874 <== mon.0 v1:172.21.15.1:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1586965025 0 0) 0x7fce30001c70 con 0x7fce3813c480 2024-03-25T18:34:03.282 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:34:03.280+0000 7fce34ff9640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:34:03.282 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:34:03.280+0000 7fce34ff9640 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2024-03-25T18:34:03.282 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:34:03.280+0000 7fce34ff9640 10 monclient(hunting): dump: 2024-03-25T18:34:03.282 INFO:teuthology.misc.health.smithi001.stderr:epoch 1 2024-03-25T18:34:03.283 INFO:teuthology.misc.health.smithi001.stderr:fsid c8ea72cf-9e99-4383-9b52-6c23d365fdfc 2024-03-25T18:34:03.283 INFO:teuthology.misc.health.smithi001.stderr:last_changed 2024-03-25T18:25:50.510519+0000 2024-03-25T18:34:03.283 INFO:teuthology.misc.health.smithi001.stderr:created 2024-03-25T18:25:50.510519+0000 2024-03-25T18:34:03.283 INFO:teuthology.misc.health.smithi001.stderr:min_mon_release 19 (squid) 2024-03-25T18:34:03.283 INFO:teuthology.misc.health.smithi001.stderr:election_strategy: 3 2024-03-25T18:34:03.283 INFO:teuthology.misc.health.smithi001.stderr:0: v1:172.21.15.1:6789/0 mon.a 2024-03-25T18:34:03.283 INFO:teuthology.misc.health.smithi001.stderr:1: v1:172.21.15.115:6789/0 mon.b 2024-03-25T18:34:03.283 INFO:teuthology.misc.health.smithi001.stderr:2: v1:172.21.15.1:6790/0 mon.c 2024-03-25T18:34:03.283 INFO:teuthology.misc.health.smithi001.stderr: 2024-03-25T18:34:03.283 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:34:03.280+0000 7fce34ff9640 10 monclient(hunting): _finish_auth 0 2024-03-25T18:34:03.287 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:34:03.286+0000 7fce34ff9640 1 -- 172.21.15.1:0/902569874 <== mon.2 v1:172.21.15.1:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1950559035 0 0) 0x7fce380ac420 con 0x7fce380b4ff0 2024-03-25T18:34:03.287 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:34:03.286+0000 7fce34ff9640 10 monclient(hunting): _negotiate 2024-03-25T18:34:03.288 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:34:03.286+0000 7fce34ff9640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:34:03.288 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:34:03.286+0000 7fce34ff9640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:34:03.288 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:34:03.286+0000 7fce34ff9640 10 monclient(hunting): my global_id is 4553 2024-03-25T18:34:03.288 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:34:03.286+0000 7fce34ff9640 1 -- 172.21.15.1:0/902569874 --> v1:172.21.15.1:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fce24004eb0 con 0x7fce380b4ff0 2024-03-25T18:34:03.311 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:34:03.308+0000 7fce34ff9640 1 -- 172.21.15.1:0/902569874 <== mon.0 v1:172.21.15.1:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 3649559706 0 0) 0x7fce380ac140 con 0x7fce3813c480 2024-03-25T18:34:03.311 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:34:03.309+0000 7fce34ff9640 1 -- 172.21.15.1:0/902569874 >> v1:172.21.15.1:6790/0 conn(0x7fce380b4ff0 legacy=0x7fce381381a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:34:03.311 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:34:03.309+0000 7fce34ff9640 1 -- 172.21.15.1:0/902569874 >> v1:172.21.15.115:6789/0 conn(0x7fce380b4670 legacy=0x7fce3813b600 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:34:03.311 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:34:03.309+0000 7fce34ff9640 10 monclient: _finish_hunting 0 2024-03-25T18:34:03.311 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:34:03.309+0000 7fce34ff9640 1 monclient: found mon.a 2024-03-25T18:34:03.311 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:34:03.309+0000 7fce34ff9640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:34:03.311 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:34:03.309+0000 7fce34ff9640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.1:6789/0 2024-03-25T18:34:03.311 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:34:03.309+0000 7fce34ff9640 1 -- 172.21.15.1:0/902569874 --> v1:172.21.15.1:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fce3813a8f0 con 0x7fce3813c480 2024-03-25T18:34:03.311 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:34:03.309+0000 7fce34ff9640 10 monclient: _finish_auth 0 2024-03-25T18:34:03.311 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:34:03.309+0000 7fce34ff9640 10 monclient: _check_auth_tickets 2024-03-25T18:34:03.311 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:34:03.309+0000 7fce34ff9640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:34:03.311 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:34:03.309+0000 7fce3e90d640 5 monclient: authenticate success, global_id 4539 2024-03-25T18:34:03.311 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:34:03.309+0000 7fce3e90d640 10 monclient: _renew_subs 2024-03-25T18:34:03.311 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:34:03.309+0000 7fce3e90d640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.1:6789/0 2024-03-25T18:34:03.312 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:34:03.309+0000 7fce3e90d640 1 -- 172.21.15.1:0/902569874 --> v1:172.21.15.1:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fce380ac840 con 0x7fce3813c480 2024-03-25T18:34:03.312 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:34:03.309+0000 7fce3e90d640 10 client.4539.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T18:34:03.312 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:34:03.309+0000 7fce3e90d640 10 monclient: _renew_subs 2024-03-25T18:34:03.312 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:34:03.309+0000 7fce3e90d640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.1:6789/0 2024-03-25T18:34:03.312 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:34:03.309+0000 7fce3e90d640 1 -- 172.21.15.1:0/902569874 --> v1:172.21.15.1:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7fce381e3aa0 con 0x7fce3813c480 2024-03-25T18:34:03.312 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:34:03.309+0000 7fce3e90d640 1 librados: init done 2024-03-25T18:34:03.312 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:34:03.310+0000 7fce3e90d640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T18:34:03.312 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:34:03.310+0000 7fce3e90d640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:34:03.313 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:34:03.310+0000 7fce3e90d640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.1:6789/0 2024-03-25T18:34:03.313 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:34:03.310+0000 7fce3e90d640 1 -- 172.21.15.1:0/902569874 --> v1:172.21.15.1:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fce3806dbd0 con 0x7fce3813c480 2024-03-25T18:34:03.595 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:34:03.593+0000 7fce34ff9640 1 -- 172.21.15.1:0/902569874 <== mon.0 v1:172.21.15.1:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fce30004b60 con 0x7fce3813c480 2024-03-25T18:34:03.595 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:34:03.593+0000 7fce34ff9640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:34:03.595 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:34:03.593+0000 7fce34ff9640 1 -- 172.21.15.1:0/902569874 <== mon.0 v1:172.21.15.1:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1586965025 0 0) 0x7fce30004e70 con 0x7fce3813c480 2024-03-25T18:34:03.595 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:34:03.593+0000 7fce34ff9640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:34:03.595 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:34:03.593+0000 7fce34ff9640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-25T18:34:03.595 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:34:03.593+0000 7fce34ff9640 10 monclient: dump: 2024-03-25T18:34:03.596 INFO:teuthology.misc.health.smithi001.stderr:epoch 1 2024-03-25T18:34:03.596 INFO:teuthology.misc.health.smithi001.stderr:fsid c8ea72cf-9e99-4383-9b52-6c23d365fdfc 2024-03-25T18:34:03.596 INFO:teuthology.misc.health.smithi001.stderr:last_changed 2024-03-25T18:25:50.510519+0000 2024-03-25T18:34:03.596 INFO:teuthology.misc.health.smithi001.stderr:created 2024-03-25T18:25:50.510519+0000 2024-03-25T18:34:03.596 INFO:teuthology.misc.health.smithi001.stderr:min_mon_release 19 (squid) 2024-03-25T18:34:03.596 INFO:teuthology.misc.health.smithi001.stderr:election_strategy: 3 2024-03-25T18:34:03.596 INFO:teuthology.misc.health.smithi001.stderr:0: v1:172.21.15.1:6789/0 mon.a 2024-03-25T18:34:03.596 INFO:teuthology.misc.health.smithi001.stderr:1: v1:172.21.15.115:6789/0 mon.b 2024-03-25T18:34:03.596 INFO:teuthology.misc.health.smithi001.stderr:2: v1:172.21.15.1:6790/0 mon.c 2024-03-25T18:34:03.596 INFO:teuthology.misc.health.smithi001.stderr: 2024-03-25T18:34:03.596 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:34:03.593+0000 7fce367fc640 4 set_mon_vals no callback set 2024-03-25T18:34:03.597 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:34:03.594+0000 7fce34ff9640 1 -- 172.21.15.1:0/902569874 <== mon.0 v1:172.21.15.1:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 21787975 0 0) 0x7fce3001ae40 con 0x7fce3813c480 2024-03-25T18:34:03.597 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:34:03.595+0000 7fce37fff640 10 monclient: get_auth_request con 0x7fce2406de00 auth_method 0 2024-03-25T18:34:03.597 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:34:03.595+0000 7fce34ff9640 10 client.4539.objecter ms_handle_connect 0x7fce2406de00 2024-03-25T18:34:03.601 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:34:03.599+0000 7fce34ff9640 1 -- 172.21.15.1:0/902569874 <== mon.0 v1:172.21.15.1:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 996254230 0 0) 0x7fce30054b00 con 0x7fce3813c480 2024-03-25T18:34:03.601 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:34:03.599+0000 7fce34ff9640 10 client.4539.objecter ms_dispatch 0x7fce38000b90 osd_map(10..10 src has 1..10) v4 2024-03-25T18:34:03.602 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:34:03.599+0000 7fce34ff9640 3 client.4539.objecter handle_osd_map got epochs [10,10] > 0 2024-03-25T18:34:03.602 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:34:03.599+0000 7fce34ff9640 3 client.4539.objecter handle_osd_map decoding full epoch 10 2024-03-25T18:34:03.602 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:34:03.599+0000 7fce34ff9640 20 client.4539.objecter dump_active .. 0 homeless 2024-03-25T18:34:03.602 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:34:03.599+0000 7fce34ff9640 1 -- 172.21.15.1:0/902569874 <== mon.0 v1:172.21.15.1:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7fce30050440 con 0x7fce3813c480 2024-03-25T18:34:03.602 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:34:03.599+0000 7fce34ff9640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:34:03.602 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:34:03.599+0000 7fce34ff9640 10 monclient: _finish_command 1 = system:0 2024-03-25T18:34:03.823 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:34:03.821+0000 7fce3e90d640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:34:03.823 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:34:03.821+0000 7fce3e90d640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:34:03.823 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:34:03.821+0000 7fce3e90d640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.1:6789/0 2024-03-25T18:34:03.823 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:34:03.821+0000 7fce3e90d640 1 -- 172.21.15.1:0/902569874 --> v1:172.21.15.1:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fce3814f100 con 0x7fce3813c480 2024-03-25T18:34:03.843 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:34:03.841+0000 7fce34ff9640 1 -- 172.21.15.1:0/902569874 <== mon.0 v1:172.21.15.1:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 4092213348) 0x7fce30052540 con 0x7fce3813c480 2024-03-25T18:34:03.843 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:34:03.841+0000 7fce34ff9640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:34:03.843 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:34:03.841+0000 7fce34ff9640 10 monclient: _finish_command 2 = system:0 2024-03-25T18:34:03.843 INFO:teuthology.misc.health.smithi001.stdout: 2024-03-25T18:34:03.843 INFO:teuthology.misc.health.smithi001.stdout:{"epoch":10,"fsid":"c8ea72cf-9e99-4383-9b52-6c23d365fdfc","created":"2024-03-25T18:26:38.856638+0000","modified":"2024-03-25T18:26:44.224208+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T18:26:44.224208+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"b9f34922-3a84-49be-8426-807c1eceaefc","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"42bf306f-55e3-4634-9874-8b4b73ac6644","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"292cb23a-18a9-4207-a7cc-4d5727feaec0","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"45e0e9ab-898d-4673-9590-b4117bc5e84a","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"831a5a09-73ab-4a61-9872-088e76bdbc66","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"88cd903d-3b55-4840-9396-6e3cfe005282","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"89db7355-bc9d-46f4-8ab0-e8d9e744e026","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"cac47bee-0914-4f08-896f-348838882866","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T18:34:03.845 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:34:03.843+0000 7fce3e90d640 10 librados: watch_flush enter 2024-03-25T18:34:03.845 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:34:03.843+0000 7fce3e90d640 10 librados: watch_flush exit 2024-03-25T18:34:03.846 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:34:03.843+0000 7fce3e90d640 20 client.4539.objecter shutdown clearing up homeless session... 2024-03-25T18:34:03.846 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:34:03.843+0000 7fce3e90d640 10 client.4539.objecter successfully canceled tick 2024-03-25T18:34:03.846 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:34:03.843+0000 7fce3e90d640 1 -- 172.21.15.1:0/902569874 >> v1:172.21.15.115:6800/4237379176 conn(0x7fce2406de00 legacy=0x7fce240702c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:34:03.846 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:34:03.843+0000 7fce3e90d640 10 monclient: shutdown 2024-03-25T18:34:03.846 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:34:03.843+0000 7fce3e90d640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:34:03.846 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:34:03.843+0000 7fce3e90d640 1 -- 172.21.15.1:0/902569874 >> v1:172.21.15.1:6789/0 conn(0x7fce3813c480 legacy=0x7fce381e2380 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:34:03.846 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:34:03.843+0000 7fce3e90d640 1 -- 172.21.15.1:0/902569874 shutdown_connections 2024-03-25T18:34:03.846 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:34:03.843+0000 7fce3e90d640 1 -- 172.21.15.1:0/902569874 >> 172.21.15.1:0/902569874 conn(0x7fce380617a0 msgr2=0x7fce38140760 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:34:03.846 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:34:03.843+0000 7fce3e90d640 1 -- 172.21.15.1:0/902569874 shutdown_connections 2024-03-25T18:34:03.846 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:34:03.843+0000 7fce3e90d640 1 -- 172.21.15.1:0/902569874 wait complete. 2024-03-25T18:34:03.846 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:34:03.843+0000 7fce3e90d640 1 librados: shutdown 2024-03-25T18:34:03.856 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T18:34:10.664 DEBUG:teuthology.orchestra.run.smithi001:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T18:34:10.780 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:34:10.778+0000 7f2155baf640 10 monclient: get_monmap_and_config 2024-03-25T18:34:10.780 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:34:10.778+0000 7f2155baf640 10 monclient: build_initial_monmap 2024-03-25T18:34:10.780 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:34:10.778+0000 7f2155baf640 10 monclient: monmap: 2024-03-25T18:34:10.780 INFO:teuthology.misc.health.smithi001.stderr:epoch 0 2024-03-25T18:34:10.780 INFO:teuthology.misc.health.smithi001.stderr:fsid c8ea72cf-9e99-4383-9b52-6c23d365fdfc 2024-03-25T18:34:10.780 INFO:teuthology.misc.health.smithi001.stderr:last_changed 2024-03-25T18:34:10.779961+0000 2024-03-25T18:34:10.780 INFO:teuthology.misc.health.smithi001.stderr:created 2024-03-25T18:34:10.779961+0000 2024-03-25T18:34:10.780 INFO:teuthology.misc.health.smithi001.stderr:min_mon_release 0 (unknown) 2024-03-25T18:34:10.780 INFO:teuthology.misc.health.smithi001.stderr:election_strategy: 3 2024-03-25T18:34:10.780 INFO:teuthology.misc.health.smithi001.stderr:0: v1:172.21.15.1:6789/0 mon.noname-a 2024-03-25T18:34:10.780 INFO:teuthology.misc.health.smithi001.stderr:1: v1:172.21.15.115:6789/0 mon.noname-c 2024-03-25T18:34:10.780 INFO:teuthology.misc.health.smithi001.stderr:2: v1:172.21.15.1:6790/0 mon.noname-b 2024-03-25T18:34:10.781 INFO:teuthology.misc.health.smithi001.stderr: 2024-03-25T18:34:10.782 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:34:10.779+0000 7f2155baf640 1 Processor -- start 2024-03-25T18:34:10.782 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:34:10.779+0000 7f2155baf640 1 -- start start 2024-03-25T18:34:10.782 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:34:10.779+0000 7f2155baf640 10 monclient: init 2024-03-25T18:34:10.782 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:34:10.780+0000 7f2155baf640 10 monclient: _reopen_session rank -1 2024-03-25T18:34:10.782 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:34:10.780+0000 7f2155baf640 10 monclient: _add_conns ranks=[0,1,2] 2024-03-25T18:34:10.782 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:34:10.780+0000 7f2155baf640 10 monclient(hunting): picked mon.noname-a con 0x7f21500b4e70 addr v1:172.21.15.1:6789/0 2024-03-25T18:34:10.782 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:34:10.780+0000 7f2155baf640 10 monclient(hunting): picked mon.noname-c con 0x7f21500b44b0 addr v1:172.21.15.115:6789/0 2024-03-25T18:34:10.783 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:34:10.780+0000 7f2155baf640 10 monclient(hunting): picked mon.noname-b con 0x7f21500b1b00 addr v1:172.21.15.1:6790/0 2024-03-25T18:34:10.783 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:34:10.780+0000 7f2155baf640 1 -- --> v1:172.21.15.1:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f21500a96f0 con 0x7f21500b4e70 2024-03-25T18:34:10.783 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:34:10.780+0000 7f2155baf640 1 -- --> v1:172.21.15.115:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f21500ab0e0 con 0x7f21500b44b0 2024-03-25T18:34:10.783 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:34:10.780+0000 7f2155baf640 1 -- --> v1:172.21.15.1:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2150145f70 con 0x7f21500b1b00 2024-03-25T18:34:10.783 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:34:10.780+0000 7f2155baf640 10 monclient(hunting): _renew_subs 2024-03-25T18:34:10.783 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:34:10.780+0000 7f2155baf640 10 monclient(hunting): authenticate will time out at 7931.250906s 2024-03-25T18:34:10.783 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:34:10.780+0000 7f214effd640 1 --1- >> v1:172.21.15.115:6789/0 conn(0x7f21500b44b0 0x7f2150145860 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.115:6789/0 says I am v1:172.21.15.1:51756/0 (socket says 172.21.15.1:51756) 2024-03-25T18:34:10.783 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:34:10.780+0000 7f214effd640 1 -- 172.21.15.1:0/2136555213 learned_addr learned my addr 172.21.15.1:0/2136555213 (peer_addr_for_me v1:172.21.15.1:0/0) 2024-03-25T18:34:10.807 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:34:10.805+0000 7f214e7fc640 1 -- 172.21.15.1:0/2136555213 <== mon.0 v1:172.21.15.1:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1980321819 0 0) 0x7f21500a96f0 con 0x7f21500b4e70 2024-03-25T18:34:10.807 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:34:10.805+0000 7f214e7fc640 10 monclient(hunting): _negotiate 2024-03-25T18:34:10.807 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:34:10.805+0000 7f214e7fc640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:34:10.807 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:34:10.805+0000 7f214e7fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:34:10.807 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:34:10.805+0000 7f214e7fc640 10 monclient(hunting): my global_id is 4542 2024-03-25T18:34:10.807 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:34:10.806+0000 7f214e7fc640 1 -- 172.21.15.1:0/2136555213 --> v1:172.21.15.1:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f2138003520 con 0x7f21500b4e70 2024-03-25T18:34:10.832 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:34:10.830+0000 7f214e7fc640 1 -- 172.21.15.1:0/2136555213 <== mon.0 v1:172.21.15.1:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 933768912 0 0) 0x7f2138003520 con 0x7f21500b4e70 2024-03-25T18:34:10.833 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:34:10.830+0000 7f214e7fc640 1 -- 172.21.15.1:0/2136555213 >> v1:172.21.15.1:6790/0 conn(0x7f21500b1b00 legacy=0x7f215014dc00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:34:10.833 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:34:10.830+0000 7f214e7fc640 1 -- 172.21.15.1:0/2136555213 >> v1:172.21.15.115:6789/0 conn(0x7f21500b44b0 legacy=0x7f2150145860 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:34:10.833 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:34:10.830+0000 7f214e7fc640 10 monclient: _finish_hunting 0 2024-03-25T18:34:10.833 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:34:10.830+0000 7f214e7fc640 1 monclient: found mon.noname-a 2024-03-25T18:34:10.833 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:34:10.830+0000 7f214e7fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:34:10.833 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:34:10.830+0000 7f214e7fc640 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.1:6789/0 2024-03-25T18:34:10.833 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:34:10.830+0000 7f214e7fc640 1 -- 172.21.15.1:0/2136555213 --> v1:172.21.15.1:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f21500a9b80 con 0x7f21500b4e70 2024-03-25T18:34:10.833 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:34:10.830+0000 7f214e7fc640 10 monclient: _finish_auth 0 2024-03-25T18:34:10.833 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:34:10.830+0000 7f214e7fc640 10 monclient: _check_auth_tickets 2024-03-25T18:34:10.833 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:34:10.830+0000 7f214e7fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:34:10.833 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:34:10.830+0000 7f214e7fc640 1 -- 172.21.15.1:0/2136555213 <== mon.0 v1:172.21.15.1:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1586965025 0 0) 0x7f2140001690 con 0x7f21500b4e70 2024-03-25T18:34:10.833 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:34:10.830+0000 7f214e7fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:34:10.833 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:34:10.830+0000 7f214e7fc640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-03-25T18:34:10.833 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:34:10.830+0000 7f214e7fc640 10 monclient: dump: 2024-03-25T18:34:10.833 INFO:teuthology.misc.health.smithi001.stderr:epoch 1 2024-03-25T18:34:10.834 INFO:teuthology.misc.health.smithi001.stderr:fsid c8ea72cf-9e99-4383-9b52-6c23d365fdfc 2024-03-25T18:34:10.834 INFO:teuthology.misc.health.smithi001.stderr:last_changed 2024-03-25T18:25:50.510519+0000 2024-03-25T18:34:10.834 INFO:teuthology.misc.health.smithi001.stderr:created 2024-03-25T18:25:50.510519+0000 2024-03-25T18:34:10.834 INFO:teuthology.misc.health.smithi001.stderr:min_mon_release 19 (squid) 2024-03-25T18:34:10.834 INFO:teuthology.misc.health.smithi001.stderr:election_strategy: 3 2024-03-25T18:34:10.834 INFO:teuthology.misc.health.smithi001.stderr:0: v1:172.21.15.1:6789/0 mon.a 2024-03-25T18:34:10.834 INFO:teuthology.misc.health.smithi001.stderr:1: v1:172.21.15.115:6789/0 mon.b 2024-03-25T18:34:10.834 INFO:teuthology.misc.health.smithi001.stderr:2: v1:172.21.15.1:6790/0 mon.c 2024-03-25T18:34:10.834 INFO:teuthology.misc.health.smithi001.stderr: 2024-03-25T18:34:10.834 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:34:10.830+0000 7f2155baf640 5 monclient: authenticate success, global_id 4542 2024-03-25T18:34:10.834 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:34:10.830+0000 7f2155baf640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T18:34:10.892 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:34:10.890+0000 7f214e7fc640 1 -- 172.21.15.1:0/2136555213 <== mon.0 v1:172.21.15.1:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f21400017f0 con 0x7f21500b4e70 2024-03-25T18:34:10.892 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:34:10.890+0000 7f214e7fc640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:34:10.892 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:34:10.890+0000 7f214e7fc640 1 -- 172.21.15.1:0/2136555213 <== mon.0 v1:172.21.15.1:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1586965025 0 0) 0x7f2140001c00 con 0x7f21500b4e70 2024-03-25T18:34:10.892 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:34:10.890+0000 7f2155baf640 10 monclient: get_monmap_and_config success 2024-03-25T18:34:10.892 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:34:10.890+0000 7f2155baf640 4 set_mon_vals no callback set 2024-03-25T18:34:10.893 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:34:10.890+0000 7f214e7fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:34:10.893 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:34:10.890+0000 7f214e7fc640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-25T18:34:10.893 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:34:10.890+0000 7f214e7fc640 10 monclient: dump: 2024-03-25T18:34:10.893 INFO:teuthology.misc.health.smithi001.stderr:epoch 1 2024-03-25T18:34:10.893 INFO:teuthology.misc.health.smithi001.stderr:fsid c8ea72cf-9e99-4383-9b52-6c23d365fdfc 2024-03-25T18:34:10.893 INFO:teuthology.misc.health.smithi001.stderr:last_changed 2024-03-25T18:25:50.510519+0000 2024-03-25T18:34:10.893 INFO:teuthology.misc.health.smithi001.stderr:created 2024-03-25T18:25:50.510519+0000 2024-03-25T18:34:10.893 INFO:teuthology.misc.health.smithi001.stderr:min_mon_release 19 (squid) 2024-03-25T18:34:10.893 INFO:teuthology.misc.health.smithi001.stderr:election_strategy: 3 2024-03-25T18:34:10.893 INFO:teuthology.misc.health.smithi001.stderr:0: v1:172.21.15.1:6789/0 mon.a 2024-03-25T18:34:10.893 INFO:teuthology.misc.health.smithi001.stderr:1: v1:172.21.15.115:6789/0 mon.b 2024-03-25T18:34:10.893 INFO:teuthology.misc.health.smithi001.stderr:2: v1:172.21.15.1:6790/0 mon.c 2024-03-25T18:34:10.893 INFO:teuthology.misc.health.smithi001.stderr: 2024-03-25T18:34:10.895 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:34:10.891+0000 7f2155baf640 10 monclient: shutdown 2024-03-25T18:34:10.895 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:34:10.891+0000 7f2155baf640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:34:10.895 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:34:10.891+0000 7f2155baf640 1 -- 172.21.15.1:0/2136555213 >> v1:172.21.15.1:6789/0 conn(0x7f21500b4e70 legacy=0x7f215013f940 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:34:10.895 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:34:10.891+0000 7f2155baf640 1 -- 172.21.15.1:0/2136555213 shutdown_connections 2024-03-25T18:34:10.895 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:34:10.891+0000 7f2155baf640 1 -- 172.21.15.1:0/2136555213 >> 172.21.15.1:0/2136555213 conn(0x7f21500617a0 msgr2=0x7f2150061bb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:34:10.895 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:34:10.891+0000 7f2155baf640 1 -- 172.21.15.1:0/2136555213 shutdown_connections 2024-03-25T18:34:10.895 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:34:10.891+0000 7f2155baf640 1 -- 172.21.15.1:0/2136555213 wait complete. 2024-03-25T18:34:10.895 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:34:10.892+0000 7f2155baf640 10 monclient: build_initial_monmap 2024-03-25T18:34:10.896 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:34:10.892+0000 7f2155baf640 10 monclient: monmap: 2024-03-25T18:34:10.896 INFO:teuthology.misc.health.smithi001.stderr:epoch 0 2024-03-25T18:34:10.896 INFO:teuthology.misc.health.smithi001.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T18:34:10.896 INFO:teuthology.misc.health.smithi001.stderr:last_changed 0.000000 2024-03-25T18:34:10.896 INFO:teuthology.misc.health.smithi001.stderr:created 0.000000 2024-03-25T18:34:10.896 INFO:teuthology.misc.health.smithi001.stderr:min_mon_release 0 (unknown) 2024-03-25T18:34:10.896 INFO:teuthology.misc.health.smithi001.stderr:election_strategy: 1 2024-03-25T18:34:10.896 INFO:teuthology.misc.health.smithi001.stderr:0: v1:172.21.15.1:6789/0 mon.noname-a 2024-03-25T18:34:10.896 INFO:teuthology.misc.health.smithi001.stderr:1: v1:172.21.15.115:6789/0 mon.noname-b 2024-03-25T18:34:10.896 INFO:teuthology.misc.health.smithi001.stderr:2: v1:172.21.15.1:6790/0 mon.noname-c 2024-03-25T18:34:10.896 INFO:teuthology.misc.health.smithi001.stderr: 2024-03-25T18:34:10.896 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:34:10.893+0000 7f2155baf640 1 librados: starting msgr at 2024-03-25T18:34:10.896 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:34:10.893+0000 7f2155baf640 1 librados: starting objecter 2024-03-25T18:34:10.896 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:34:10.893+0000 7f2155baf640 1 Processor -- start 2024-03-25T18:34:10.896 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:34:10.893+0000 7f2155baf640 1 -- start start 2024-03-25T18:34:10.897 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:34:10.893+0000 7f2155baf640 1 librados: setting wanted keys 2024-03-25T18:34:10.897 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:34:10.893+0000 7f2155baf640 1 librados: calling monclient init 2024-03-25T18:34:10.897 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:34:10.893+0000 7f2155baf640 10 monclient: init 2024-03-25T18:34:10.898 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:34:10.894+0000 7f2155baf640 10 monclient: _reopen_session rank -1 2024-03-25T18:34:10.898 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:34:10.894+0000 7f2155baf640 10 monclient: _add_conns ranks=[2,1,0] 2024-03-25T18:34:10.898 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:34:10.894+0000 7f2155baf640 10 monclient(hunting): picked mon.noname-c con 0x7f21500b1b00 addr v1:172.21.15.1:6790/0 2024-03-25T18:34:10.898 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:34:10.894+0000 7f2155baf640 10 monclient(hunting): picked mon.noname-b con 0x7f21500b44b0 addr v1:172.21.15.115:6789/0 2024-03-25T18:34:10.898 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:34:10.894+0000 7f2155baf640 10 monclient(hunting): picked mon.noname-a con 0x7f21500b4e70 addr v1:172.21.15.1:6789/0 2024-03-25T18:34:10.898 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:34:10.894+0000 7f2155baf640 1 -- --> v1:172.21.15.1:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f21501451c0 con 0x7f21500b4e70 2024-03-25T18:34:10.898 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:34:10.894+0000 7f2155baf640 1 -- --> v1:172.21.15.115:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f215013b6e0 con 0x7f21500b44b0 2024-03-25T18:34:10.898 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:34:10.894+0000 7f2155baf640 1 -- --> v1:172.21.15.1:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f215013cb50 con 0x7f21500b1b00 2024-03-25T18:34:10.898 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:34:10.894+0000 7f2155baf640 10 monclient(hunting): _renew_subs 2024-03-25T18:34:10.898 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:34:10.894+0000 7f2155baf640 10 monclient(hunting): authenticate will time out at 7931.365130s 2024-03-25T18:34:10.909 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:34:10.907+0000 7f214ffff640 1 --1- >> v1:172.21.15.1:6789/0 conn(0x7f21500b4e70 0x7f2150142790 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.1:6789/0 says I am v1:172.21.15.1:41102/0 (socket says 172.21.15.1:41102) 2024-03-25T18:34:10.909 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:34:10.907+0000 7f214ffff640 1 -- 172.21.15.1:0/4187244055 learned_addr learned my addr 172.21.15.1:0/4187244055 (peer_addr_for_me v1:172.21.15.1:0/0) 2024-03-25T18:34:10.913 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:34:10.911+0000 7f2137fff640 10 client.?.objecter ms_handle_connect 0x7f21500b4e70 2024-03-25T18:34:10.913 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:34:10.911+0000 7f2137fff640 10 client.?.objecter resend_mon_ops 2024-03-25T18:34:10.923 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:34:10.922+0000 7f2137fff640 1 -- 172.21.15.1:0/4187244055 <== mon.0 v1:172.21.15.1:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 295321916 0 0) 0x7f21501451c0 con 0x7f21500b4e70 2024-03-25T18:34:10.923 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:34:10.922+0000 7f2137fff640 10 monclient(hunting): _negotiate 2024-03-25T18:34:10.924 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:34:10.922+0000 7f2137fff640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:34:10.924 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:34:10.922+0000 7f2137fff640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:34:10.924 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:34:10.922+0000 7f2137fff640 10 monclient(hunting): my global_id is 4545 2024-03-25T18:34:10.924 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:34:10.922+0000 7f2137fff640 1 -- 172.21.15.1:0/4187244055 --> v1:172.21.15.1:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f213c004eb0 con 0x7f21500b4e70 2024-03-25T18:34:10.941 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:34:10.939+0000 7f2137fff640 1 -- 172.21.15.1:0/4187244055 <== mon.0 v1:172.21.15.1:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3933256718 0 0) 0x7f213c004eb0 con 0x7f21500b4e70 2024-03-25T18:34:10.942 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:34:10.939+0000 7f2137fff640 1 -- 172.21.15.1:0/4187244055 --> v1:172.21.15.1:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f21501451c0 con 0x7f21500b4e70 2024-03-25T18:34:10.942 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:34:10.940+0000 7f2137fff640 1 -- 172.21.15.1:0/4187244055 <== mon.0 v1:172.21.15.1:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1586965025 0 0) 0x7f2148001cd0 con 0x7f21500b4e70 2024-03-25T18:34:10.942 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:34:10.940+0000 7f2137fff640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:34:10.942 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:34:10.940+0000 7f2137fff640 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2024-03-25T18:34:10.942 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:34:10.940+0000 7f2137fff640 10 monclient(hunting): dump: 2024-03-25T18:34:10.942 INFO:teuthology.misc.health.smithi001.stderr:epoch 1 2024-03-25T18:34:10.942 INFO:teuthology.misc.health.smithi001.stderr:fsid c8ea72cf-9e99-4383-9b52-6c23d365fdfc 2024-03-25T18:34:10.942 INFO:teuthology.misc.health.smithi001.stderr:last_changed 2024-03-25T18:25:50.510519+0000 2024-03-25T18:34:10.942 INFO:teuthology.misc.health.smithi001.stderr:created 2024-03-25T18:25:50.510519+0000 2024-03-25T18:34:10.942 INFO:teuthology.misc.health.smithi001.stderr:min_mon_release 19 (squid) 2024-03-25T18:34:10.942 INFO:teuthology.misc.health.smithi001.stderr:election_strategy: 3 2024-03-25T18:34:10.942 INFO:teuthology.misc.health.smithi001.stderr:0: v1:172.21.15.1:6789/0 mon.a 2024-03-25T18:34:10.942 INFO:teuthology.misc.health.smithi001.stderr:1: v1:172.21.15.115:6789/0 mon.b 2024-03-25T18:34:10.942 INFO:teuthology.misc.health.smithi001.stderr:2: v1:172.21.15.1:6790/0 mon.c 2024-03-25T18:34:10.943 INFO:teuthology.misc.health.smithi001.stderr: 2024-03-25T18:34:10.943 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:34:10.940+0000 7f2137fff640 10 monclient(hunting): _finish_auth 0 2024-03-25T18:34:10.945 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:34:10.943+0000 7f2137fff640 10 client.?.objecter ms_handle_connect 0x7f21500b44b0 2024-03-25T18:34:10.945 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:34:10.943+0000 7f2137fff640 10 client.?.objecter resend_mon_ops 2024-03-25T18:34:10.950 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:34:10.948+0000 7f2137fff640 1 -- 172.21.15.1:0/4187244055 <== mon.1 v1:172.21.15.115:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 837469764 0 0) 0x7f215013b6e0 con 0x7f21500b44b0 2024-03-25T18:34:10.950 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:34:10.948+0000 7f2137fff640 10 monclient(hunting): _negotiate 2024-03-25T18:34:10.950 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:34:10.948+0000 7f2137fff640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:34:10.950 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:34:10.948+0000 7f2137fff640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:34:10.950 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:34:10.948+0000 7f2137fff640 10 monclient(hunting): my global_id is 4552 2024-03-25T18:34:10.950 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:34:10.948+0000 7f2137fff640 1 -- 172.21.15.1:0/4187244055 --> v1:172.21.15.115:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f213c004eb0 con 0x7f21500b44b0 2024-03-25T18:34:10.955 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:34:10.952+0000 7f2137fff640 1 -- 172.21.15.1:0/4187244055 <== mon.0 v1:172.21.15.1:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 3722415793 0 0) 0x7f21501451c0 con 0x7f21500b4e70 2024-03-25T18:34:10.955 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:34:10.952+0000 7f2137fff640 1 -- 172.21.15.1:0/4187244055 >> v1:172.21.15.1:6790/0 conn(0x7f21500b1b00 legacy=0x7f215013aa40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:34:10.955 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:34:10.952+0000 7f2137fff640 1 -- 172.21.15.1:0/4187244055 >> v1:172.21.15.115:6789/0 conn(0x7f21500b44b0 legacy=0x7f2150143b00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:34:10.955 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:34:10.952+0000 7f2137fff640 10 monclient: _finish_hunting 0 2024-03-25T18:34:10.955 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:34:10.952+0000 7f2137fff640 1 monclient: found mon.a 2024-03-25T18:34:10.955 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:34:10.952+0000 7f2137fff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:34:10.955 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:34:10.952+0000 7f2137fff640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.1:6789/0 2024-03-25T18:34:10.955 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:34:10.952+0000 7f2137fff640 1 -- 172.21.15.1:0/4187244055 --> v1:172.21.15.1:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2150139f50 con 0x7f21500b4e70 2024-03-25T18:34:10.955 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:34:10.952+0000 7f2137fff640 10 monclient: _finish_auth 0 2024-03-25T18:34:10.956 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:34:10.952+0000 7f2137fff640 10 monclient: _check_auth_tickets 2024-03-25T18:34:10.956 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:34:10.952+0000 7f2137fff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:34:10.956 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:34:10.953+0000 7f2155baf640 5 monclient: authenticate success, global_id 4545 2024-03-25T18:34:10.956 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:34:10.953+0000 7f2155baf640 10 monclient: _renew_subs 2024-03-25T18:34:10.956 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:34:10.953+0000 7f2155baf640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.1:6789/0 2024-03-25T18:34:10.956 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:34:10.953+0000 7f2155baf640 1 -- 172.21.15.1:0/4187244055 --> v1:172.21.15.1:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f215013bb70 con 0x7f21500b4e70 2024-03-25T18:34:10.956 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:34:10.953+0000 7f2155baf640 10 client.4545.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T18:34:10.956 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:34:10.953+0000 7f2155baf640 10 monclient: _renew_subs 2024-03-25T18:34:10.956 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:34:10.953+0000 7f2155baf640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.1:6789/0 2024-03-25T18:34:10.956 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:34:10.953+0000 7f2155baf640 1 -- 172.21.15.1:0/4187244055 --> v1:172.21.15.1:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f2150144210 con 0x7f21500b4e70 2024-03-25T18:34:10.956 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:34:10.953+0000 7f2155baf640 1 librados: init done 2024-03-25T18:34:10.957 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:34:10.954+0000 7f2155baf640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T18:34:10.957 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:34:10.954+0000 7f2155baf640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:34:10.957 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:34:10.954+0000 7f2155baf640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.1:6789/0 2024-03-25T18:34:10.957 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:34:10.954+0000 7f2155baf640 1 -- 172.21.15.1:0/4187244055 --> v1:172.21.15.1:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f215013b450 con 0x7f21500b4e70 2024-03-25T18:34:11.261 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:34:11.259+0000 7f2137fff640 1 -- 172.21.15.1:0/4187244055 <== mon.0 v1:172.21.15.1:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f2148001fd0 con 0x7f21500b4e70 2024-03-25T18:34:11.261 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:34:11.259+0000 7f2137fff640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:34:11.261 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:34:11.259+0000 7f2137fff640 1 -- 172.21.15.1:0/4187244055 <== mon.0 v1:172.21.15.1:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1586965025 0 0) 0x7f2148003de0 con 0x7f21500b4e70 2024-03-25T18:34:11.261 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:34:11.259+0000 7f2137fff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:34:11.262 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:34:11.259+0000 7f2137fff640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-25T18:34:11.262 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:34:11.259+0000 7f2137fff640 10 monclient: dump: 2024-03-25T18:34:11.262 INFO:teuthology.misc.health.smithi001.stderr:epoch 1 2024-03-25T18:34:11.262 INFO:teuthology.misc.health.smithi001.stderr:fsid c8ea72cf-9e99-4383-9b52-6c23d365fdfc 2024-03-25T18:34:11.262 INFO:teuthology.misc.health.smithi001.stderr:last_changed 2024-03-25T18:25:50.510519+0000 2024-03-25T18:34:11.262 INFO:teuthology.misc.health.smithi001.stderr:created 2024-03-25T18:25:50.510519+0000 2024-03-25T18:34:11.262 INFO:teuthology.misc.health.smithi001.stderr:min_mon_release 19 (squid) 2024-03-25T18:34:11.262 INFO:teuthology.misc.health.smithi001.stderr:election_strategy: 3 2024-03-25T18:34:11.262 INFO:teuthology.misc.health.smithi001.stderr:0: v1:172.21.15.1:6789/0 mon.a 2024-03-25T18:34:11.262 INFO:teuthology.misc.health.smithi001.stderr:1: v1:172.21.15.115:6789/0 mon.b 2024-03-25T18:34:11.262 INFO:teuthology.misc.health.smithi001.stderr:2: v1:172.21.15.1:6790/0 mon.c 2024-03-25T18:34:11.262 INFO:teuthology.misc.health.smithi001.stderr: 2024-03-25T18:34:11.262 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:34:11.259+0000 7f214dffb640 4 set_mon_vals no callback set 2024-03-25T18:34:11.263 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:34:11.260+0000 7f2137fff640 1 -- 172.21.15.1:0/4187244055 <== mon.0 v1:172.21.15.1:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 21787975 0 0) 0x7f2148004000 con 0x7f21500b4e70 2024-03-25T18:34:11.263 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:34:11.261+0000 7f214f7fe640 10 monclient: get_auth_request con 0x7f213c06dd70 auth_method 0 2024-03-25T18:34:11.263 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:34:11.261+0000 7f2137fff640 10 client.4545.objecter ms_handle_connect 0x7f213c06dd70 2024-03-25T18:34:11.268 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:34:11.266+0000 7f2137fff640 1 -- 172.21.15.1:0/4187244055 <== mon.0 v1:172.21.15.1:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 996254230 0 0) 0x7f2148054b20 con 0x7f21500b4e70 2024-03-25T18:34:11.268 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:34:11.266+0000 7f2137fff640 10 client.4545.objecter ms_dispatch 0x7f2150000b90 osd_map(10..10 src has 1..10) v4 2024-03-25T18:34:11.269 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:34:11.266+0000 7f2137fff640 3 client.4545.objecter handle_osd_map got epochs [10,10] > 0 2024-03-25T18:34:11.269 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:34:11.266+0000 7f2137fff640 3 client.4545.objecter handle_osd_map decoding full epoch 10 2024-03-25T18:34:11.269 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:34:11.266+0000 7f2137fff640 20 client.4545.objecter dump_active .. 0 homeless 2024-03-25T18:34:11.269 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:34:11.266+0000 7f2137fff640 1 -- 172.21.15.1:0/4187244055 <== mon.0 v1:172.21.15.1:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f2148050630 con 0x7f21500b4e70 2024-03-25T18:34:11.269 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:34:11.266+0000 7f2137fff640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:34:11.269 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:34:11.266+0000 7f2137fff640 10 monclient: _finish_command 1 = system:0 2024-03-25T18:34:11.517 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:34:11.515+0000 7f2155baf640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:34:11.517 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:34:11.515+0000 7f2155baf640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:34:11.517 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:34:11.515+0000 7f2155baf640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.1:6789/0 2024-03-25T18:34:11.517 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:34:11.515+0000 7f2155baf640 1 -- 172.21.15.1:0/4187244055 --> v1:172.21.15.1:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f215004f730 con 0x7f21500b4e70 2024-03-25T18:34:11.542 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:34:11.540+0000 7f2137fff640 1 -- 172.21.15.1:0/4187244055 <== mon.0 v1:172.21.15.1:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 4092213348) 0x7f2148052560 con 0x7f21500b4e70 2024-03-25T18:34:11.542 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:34:11.540+0000 7f2137fff640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:34:11.542 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:34:11.540+0000 7f2137fff640 10 monclient: _finish_command 2 = system:0 2024-03-25T18:34:11.542 INFO:teuthology.misc.health.smithi001.stdout: 2024-03-25T18:34:11.542 INFO:teuthology.misc.health.smithi001.stdout:{"epoch":10,"fsid":"c8ea72cf-9e99-4383-9b52-6c23d365fdfc","created":"2024-03-25T18:26:38.856638+0000","modified":"2024-03-25T18:26:44.224208+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T18:26:44.224208+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"b9f34922-3a84-49be-8426-807c1eceaefc","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"42bf306f-55e3-4634-9874-8b4b73ac6644","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"292cb23a-18a9-4207-a7cc-4d5727feaec0","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"45e0e9ab-898d-4673-9590-b4117bc5e84a","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"831a5a09-73ab-4a61-9872-088e76bdbc66","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"88cd903d-3b55-4840-9396-6e3cfe005282","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"89db7355-bc9d-46f4-8ab0-e8d9e744e026","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"cac47bee-0914-4f08-896f-348838882866","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T18:34:11.544 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:34:11.542+0000 7f2155baf640 10 librados: watch_flush enter 2024-03-25T18:34:11.544 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:34:11.542+0000 7f2155baf640 10 librados: watch_flush exit 2024-03-25T18:34:11.544 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:34:11.542+0000 7f2155baf640 20 client.4545.objecter shutdown clearing up homeless session... 2024-03-25T18:34:11.544 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:34:11.542+0000 7f2155baf640 10 client.4545.objecter successfully canceled tick 2024-03-25T18:34:11.545 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:34:11.542+0000 7f2155baf640 1 -- 172.21.15.1:0/4187244055 >> v1:172.21.15.115:6800/4237379176 conn(0x7f213c06dd70 legacy=0x7f213c070230 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:34:11.545 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:34:11.542+0000 7f2155baf640 10 monclient: shutdown 2024-03-25T18:34:11.545 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:34:11.542+0000 7f2155baf640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:34:11.545 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:34:11.542+0000 7f2155baf640 1 -- 172.21.15.1:0/4187244055 >> v1:172.21.15.1:6789/0 conn(0x7f21500b4e70 legacy=0x7f2150142790 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:34:11.545 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:34:11.542+0000 7f2155baf640 1 -- 172.21.15.1:0/4187244055 shutdown_connections 2024-03-25T18:34:11.545 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:34:11.542+0000 7f2155baf640 1 -- 172.21.15.1:0/4187244055 >> 172.21.15.1:0/4187244055 conn(0x7f21500617a0 msgr2=0x7f21500b6690 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:34:11.545 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:34:11.542+0000 7f2155baf640 1 -- 172.21.15.1:0/4187244055 shutdown_connections 2024-03-25T18:34:11.545 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:34:11.542+0000 7f2155baf640 1 -- 172.21.15.1:0/4187244055 wait complete. 2024-03-25T18:34:11.545 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:34:11.542+0000 7f2155baf640 1 librados: shutdown 2024-03-25T18:34:11.556 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T18:34:18.363 DEBUG:teuthology.orchestra.run.smithi001:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T18:34:18.481 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:34:18.479+0000 7fa002296640 10 monclient: get_monmap_and_config 2024-03-25T18:34:18.481 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:34:18.479+0000 7fa002296640 10 monclient: build_initial_monmap 2024-03-25T18:34:18.481 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:34:18.479+0000 7fa002296640 10 monclient: monmap: 2024-03-25T18:34:18.481 INFO:teuthology.misc.health.smithi001.stderr:epoch 0 2024-03-25T18:34:18.481 INFO:teuthology.misc.health.smithi001.stderr:fsid c8ea72cf-9e99-4383-9b52-6c23d365fdfc 2024-03-25T18:34:18.481 INFO:teuthology.misc.health.smithi001.stderr:last_changed 2024-03-25T18:34:18.480830+0000 2024-03-25T18:34:18.481 INFO:teuthology.misc.health.smithi001.stderr:created 2024-03-25T18:34:18.480830+0000 2024-03-25T18:34:18.481 INFO:teuthology.misc.health.smithi001.stderr:min_mon_release 0 (unknown) 2024-03-25T18:34:18.481 INFO:teuthology.misc.health.smithi001.stderr:election_strategy: 3 2024-03-25T18:34:18.481 INFO:teuthology.misc.health.smithi001.stderr:0: v1:172.21.15.1:6789/0 mon.noname-a 2024-03-25T18:34:18.481 INFO:teuthology.misc.health.smithi001.stderr:1: v1:172.21.15.115:6789/0 mon.noname-c 2024-03-25T18:34:18.481 INFO:teuthology.misc.health.smithi001.stderr:2: v1:172.21.15.1:6790/0 mon.noname-b 2024-03-25T18:34:18.481 INFO:teuthology.misc.health.smithi001.stderr: 2024-03-25T18:34:18.483 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:34:18.480+0000 7fa002296640 1 Processor -- start 2024-03-25T18:34:18.483 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:34:18.480+0000 7fa002296640 1 -- start start 2024-03-25T18:34:18.483 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:34:18.480+0000 7fa002296640 10 monclient: init 2024-03-25T18:34:18.483 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:34:18.480+0000 7fa002296640 10 monclient: _reopen_session rank -1 2024-03-25T18:34:18.483 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:34:18.480+0000 7fa002296640 10 monclient: _add_conns ranks=[1,0,2] 2024-03-25T18:34:18.483 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:34:18.480+0000 7fa002296640 10 monclient(hunting): picked mon.noname-c con 0x7f9ffc0b4ff0 addr v1:172.21.15.115:6789/0 2024-03-25T18:34:18.483 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:34:18.480+0000 7fa002296640 10 monclient(hunting): picked mon.noname-a con 0x7f9ffc0b4670 addr v1:172.21.15.1:6789/0 2024-03-25T18:34:18.483 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:34:18.480+0000 7fa002296640 10 monclient(hunting): picked mon.noname-b con 0x7f9ffc13c480 addr v1:172.21.15.1:6790/0 2024-03-25T18:34:18.483 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:34:18.480+0000 7fa002296640 1 -- --> v1:172.21.15.1:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9ffc0ab0e0 con 0x7f9ffc0b4670 2024-03-25T18:34:18.483 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:34:18.481+0000 7fa002296640 1 -- --> v1:172.21.15.115:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9ffc0ab3c0 con 0x7f9ffc0b4ff0 2024-03-25T18:34:18.483 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:34:18.481+0000 7fa002296640 1 -- --> v1:172.21.15.1:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9ffc0a9f30 con 0x7f9ffc13c480 2024-03-25T18:34:18.483 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:34:18.481+0000 7fa002296640 10 monclient(hunting): _renew_subs 2024-03-25T18:34:18.483 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:34:18.481+0000 7fa002296640 10 monclient(hunting): authenticate will time out at 7938.951694s 2024-03-25T18:34:18.488 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:34:18.486+0000 7f9ffbfff640 1 --1- >> v1:172.21.15.115:6789/0 conn(0x7f9ffc0b4ff0 0x7f9ffc13bd70 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.115:6789/0 says I am v1:172.21.15.1:33348/0 (socket says 172.21.15.1:33348) 2024-03-25T18:34:18.488 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:34:18.486+0000 7f9ffbfff640 1 -- 172.21.15.1:0/2178147996 learned_addr learned my addr 172.21.15.1:0/2178147996 (peer_addr_for_me v1:172.21.15.1:0/0) 2024-03-25T18:34:18.495 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:34:18.493+0000 7f9ffaffd640 1 -- 172.21.15.1:0/2178147996 <== mon.1 v1:172.21.15.115:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3261202414 0 0) 0x7f9ffc0ab3c0 con 0x7f9ffc0b4ff0 2024-03-25T18:34:18.495 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:34:18.493+0000 7f9ffaffd640 10 monclient(hunting): _negotiate 2024-03-25T18:34:18.495 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:34:18.493+0000 7f9ffaffd640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:34:18.495 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:34:18.493+0000 7f9ffaffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:34:18.495 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:34:18.493+0000 7f9ffaffd640 10 monclient(hunting): my global_id is 4555 2024-03-25T18:34:18.495 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:34:18.493+0000 7f9ffaffd640 1 -- 172.21.15.1:0/2178147996 --> v1:172.21.15.115:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f9fe4003520 con 0x7f9ffc0b4ff0 2024-03-25T18:34:18.506 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:34:18.505+0000 7f9ffaffd640 1 -- 172.21.15.1:0/2178147996 <== mon.0 v1:172.21.15.1:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 49795398 0 0) 0x7f9ffc0ab0e0 con 0x7f9ffc0b4670 2024-03-25T18:34:18.506 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:34:18.505+0000 7f9ffaffd640 10 monclient(hunting): _negotiate 2024-03-25T18:34:18.507 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:34:18.505+0000 7f9ffaffd640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:34:18.507 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:34:18.505+0000 7f9ffaffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:34:18.507 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:34:18.505+0000 7f9ffaffd640 10 monclient(hunting): my global_id is 4548 2024-03-25T18:34:18.507 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:34:18.505+0000 7f9ffaffd640 1 -- 172.21.15.1:0/2178147996 --> v1:172.21.15.1:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f9ffc0ab3c0 con 0x7f9ffc0b4670 2024-03-25T18:34:18.509 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:34:18.507+0000 7f9ffaffd640 1 -- 172.21.15.1:0/2178147996 <== mon.1 v1:172.21.15.115:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 11183287 0 0) 0x7f9fe4003520 con 0x7f9ffc0b4ff0 2024-03-25T18:34:18.509 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:34:18.507+0000 7f9ffaffd640 1 -- 172.21.15.1:0/2178147996 >> v1:172.21.15.1:6790/0 conn(0x7f9ffc13c480 legacy=0x7f9ffc14dc00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:34:18.509 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:34:18.507+0000 7f9ffaffd640 1 -- 172.21.15.1:0/2178147996 >> v1:172.21.15.1:6789/0 conn(0x7f9ffc0b4670 legacy=0x7f9ffc1425c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:34:18.509 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:34:18.507+0000 7f9ffaffd640 10 monclient: _finish_hunting 0 2024-03-25T18:34:18.509 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:34:18.507+0000 7f9ffaffd640 1 monclient: found mon.noname-c 2024-03-25T18:34:18.509 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:34:18.507+0000 7f9ffaffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:34:18.510 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:34:18.507+0000 7f9ffaffd640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.115:6789/0 2024-03-25T18:34:18.510 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:34:18.507+0000 7f9ffaffd640 1 -- 172.21.15.1:0/2178147996 --> v1:172.21.15.115:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9ffc0a9b70 con 0x7f9ffc0b4ff0 2024-03-25T18:34:18.510 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:34:18.507+0000 7f9ffaffd640 10 monclient: _finish_auth 0 2024-03-25T18:34:18.510 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:34:18.507+0000 7f9ffaffd640 10 monclient: _check_auth_tickets 2024-03-25T18:34:18.510 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:34:18.507+0000 7f9ffaffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:34:18.510 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:34:18.507+0000 7f9ffaffd640 1 -- 172.21.15.1:0/2178147996 <== mon.1 v1:172.21.15.115:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1586965025 0 0) 0x7f9fec001690 con 0x7f9ffc0b4ff0 2024-03-25T18:34:18.510 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:34:18.507+0000 7fa002296640 5 monclient: authenticate success, global_id 4555 2024-03-25T18:34:18.510 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:34:18.507+0000 7fa002296640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T18:34:18.510 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:34:18.507+0000 7f9ffaffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:34:18.510 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:34:18.507+0000 7f9ffaffd640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-03-25T18:34:18.510 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:34:18.507+0000 7f9ffaffd640 10 monclient: dump: 2024-03-25T18:34:18.510 INFO:teuthology.misc.health.smithi001.stderr:epoch 1 2024-03-25T18:34:18.510 INFO:teuthology.misc.health.smithi001.stderr:fsid c8ea72cf-9e99-4383-9b52-6c23d365fdfc 2024-03-25T18:34:18.511 INFO:teuthology.misc.health.smithi001.stderr:last_changed 2024-03-25T18:25:50.510519+0000 2024-03-25T18:34:18.511 INFO:teuthology.misc.health.smithi001.stderr:created 2024-03-25T18:25:50.510519+0000 2024-03-25T18:34:18.511 INFO:teuthology.misc.health.smithi001.stderr:min_mon_release 19 (squid) 2024-03-25T18:34:18.511 INFO:teuthology.misc.health.smithi001.stderr:election_strategy: 3 2024-03-25T18:34:18.511 INFO:teuthology.misc.health.smithi001.stderr:0: v1:172.21.15.1:6789/0 mon.a 2024-03-25T18:34:18.511 INFO:teuthology.misc.health.smithi001.stderr:1: v1:172.21.15.115:6789/0 mon.b 2024-03-25T18:34:18.511 INFO:teuthology.misc.health.smithi001.stderr:2: v1:172.21.15.1:6790/0 mon.c 2024-03-25T18:34:18.511 INFO:teuthology.misc.health.smithi001.stderr: 2024-03-25T18:34:18.511 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:34:18.507+0000 7fa002296640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T18:34:18.517 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:34:18.515+0000 7f9ffaffd640 1 -- 172.21.15.1:0/2178147996 <== mon.1 v1:172.21.15.115:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f9fec0017f0 con 0x7f9ffc0b4ff0 2024-03-25T18:34:18.517 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:34:18.515+0000 7f9ffaffd640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:34:18.517 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:34:18.515+0000 7f9ffaffd640 1 -- 172.21.15.1:0/2178147996 <== mon.1 v1:172.21.15.115:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1586965025 0 0) 0x7f9fec001c00 con 0x7f9ffc0b4ff0 2024-03-25T18:34:18.517 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:34:18.515+0000 7f9ffaffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:34:18.517 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:34:18.515+0000 7f9ffaffd640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-25T18:34:18.517 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:34:18.515+0000 7f9ffaffd640 10 monclient: dump: 2024-03-25T18:34:18.518 INFO:teuthology.misc.health.smithi001.stderr:epoch 1 2024-03-25T18:34:18.518 INFO:teuthology.misc.health.smithi001.stderr:fsid c8ea72cf-9e99-4383-9b52-6c23d365fdfc 2024-03-25T18:34:18.518 INFO:teuthology.misc.health.smithi001.stderr:last_changed 2024-03-25T18:25:50.510519+0000 2024-03-25T18:34:18.518 INFO:teuthology.misc.health.smithi001.stderr:created 2024-03-25T18:25:50.510519+0000 2024-03-25T18:34:18.518 INFO:teuthology.misc.health.smithi001.stderr:min_mon_release 19 (squid) 2024-03-25T18:34:18.518 INFO:teuthology.misc.health.smithi001.stderr:election_strategy: 3 2024-03-25T18:34:18.518 INFO:teuthology.misc.health.smithi001.stderr:0: v1:172.21.15.1:6789/0 mon.a 2024-03-25T18:34:18.518 INFO:teuthology.misc.health.smithi001.stderr:1: v1:172.21.15.115:6789/0 mon.b 2024-03-25T18:34:18.518 INFO:teuthology.misc.health.smithi001.stderr:2: v1:172.21.15.1:6790/0 mon.c 2024-03-25T18:34:18.518 INFO:teuthology.misc.health.smithi001.stderr: 2024-03-25T18:34:18.518 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:34:18.515+0000 7fa002296640 10 monclient: get_monmap_and_config success 2024-03-25T18:34:18.518 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:34:18.515+0000 7fa002296640 4 set_mon_vals no callback set 2024-03-25T18:34:18.519 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:34:18.516+0000 7fa002296640 10 monclient: shutdown 2024-03-25T18:34:18.519 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:34:18.516+0000 7fa002296640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:34:18.520 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:34:18.516+0000 7fa002296640 1 -- 172.21.15.1:0/2178147996 >> v1:172.21.15.115:6789/0 conn(0x7f9ffc0b4ff0 legacy=0x7f9ffc13bd70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:34:18.520 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:34:18.516+0000 7fa002296640 1 -- 172.21.15.1:0/2178147996 shutdown_connections 2024-03-25T18:34:18.520 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:34:18.516+0000 7fa002296640 1 -- 172.21.15.1:0/2178147996 >> 172.21.15.1:0/2178147996 conn(0x7f9ffc0617a0 msgr2=0x7f9ffc061bb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:34:18.520 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:34:18.516+0000 7fa002296640 1 -- 172.21.15.1:0/2178147996 shutdown_connections 2024-03-25T18:34:18.520 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:34:18.517+0000 7fa002296640 1 -- 172.21.15.1:0/2178147996 wait complete. 2024-03-25T18:34:18.520 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:34:18.517+0000 7fa002296640 10 monclient: build_initial_monmap 2024-03-25T18:34:18.520 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:34:18.517+0000 7fa002296640 10 monclient: monmap: 2024-03-25T18:34:18.520 INFO:teuthology.misc.health.smithi001.stderr:epoch 0 2024-03-25T18:34:18.520 INFO:teuthology.misc.health.smithi001.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T18:34:18.520 INFO:teuthology.misc.health.smithi001.stderr:last_changed 0.000000 2024-03-25T18:34:18.520 INFO:teuthology.misc.health.smithi001.stderr:created 0.000000 2024-03-25T18:34:18.520 INFO:teuthology.misc.health.smithi001.stderr:min_mon_release 0 (unknown) 2024-03-25T18:34:18.520 INFO:teuthology.misc.health.smithi001.stderr:election_strategy: 1 2024-03-25T18:34:18.520 INFO:teuthology.misc.health.smithi001.stderr:0: v1:172.21.15.1:6789/0 mon.noname-a 2024-03-25T18:34:18.521 INFO:teuthology.misc.health.smithi001.stderr:1: v1:172.21.15.115:6789/0 mon.noname-b 2024-03-25T18:34:18.521 INFO:teuthology.misc.health.smithi001.stderr:2: v1:172.21.15.1:6790/0 mon.noname-c 2024-03-25T18:34:18.521 INFO:teuthology.misc.health.smithi001.stderr: 2024-03-25T18:34:18.521 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:34:18.518+0000 7fa002296640 1 librados: starting msgr at 2024-03-25T18:34:18.521 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:34:18.518+0000 7fa002296640 1 librados: starting objecter 2024-03-25T18:34:18.521 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:34:18.518+0000 7fa002296640 1 Processor -- start 2024-03-25T18:34:18.523 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:34:18.518+0000 7fa002296640 1 -- start start 2024-03-25T18:34:18.523 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:34:18.518+0000 7fa002296640 1 librados: setting wanted keys 2024-03-25T18:34:18.523 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:34:18.518+0000 7fa002296640 1 librados: calling monclient init 2024-03-25T18:34:18.523 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:34:18.518+0000 7fa002296640 10 monclient: init 2024-03-25T18:34:18.523 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:34:18.519+0000 7fa002296640 10 monclient: _reopen_session rank -1 2024-03-25T18:34:18.523 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:34:18.519+0000 7fa002296640 10 monclient: _add_conns ranks=[0,2,1] 2024-03-25T18:34:18.523 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:34:18.519+0000 7fa002296640 10 monclient(hunting): picked mon.noname-a con 0x7f9ffc0b4670 addr v1:172.21.15.1:6789/0 2024-03-25T18:34:18.523 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:34:18.519+0000 7fa002296640 10 monclient(hunting): picked mon.noname-c con 0x7f9ffc0b4ff0 addr v1:172.21.15.1:6790/0 2024-03-25T18:34:18.523 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:34:18.519+0000 7fa002296640 10 monclient(hunting): picked mon.noname-b con 0x7f9ffc13c480 addr v1:172.21.15.115:6789/0 2024-03-25T18:34:18.523 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:34:18.519+0000 7fa002296640 1 -- --> v1:172.21.15.1:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9ffc13c990 con 0x7f9ffc0b4670 2024-03-25T18:34:18.523 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:34:18.519+0000 7fa002296640 1 -- --> v1:172.21.15.115:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9ffc143cc0 con 0x7f9ffc13c480 2024-03-25T18:34:18.523 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:34:18.519+0000 7fa002296640 1 -- --> v1:172.21.15.1:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9ffc13f800 con 0x7f9ffc0b4ff0 2024-03-25T18:34:18.523 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:34:18.519+0000 7fa002296640 10 monclient(hunting): _renew_subs 2024-03-25T18:34:18.524 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:34:18.519+0000 7fa002296640 10 monclient(hunting): authenticate will time out at 7938.990345s 2024-03-25T18:34:18.525 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:34:18.524+0000 7fa000833640 1 --1- >> v1:172.21.15.115:6789/0 conn(0x7f9ffc13c480 0x7f9ffc13d580 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.115:6789/0 says I am v1:172.21.15.1:33370/0 (socket says 172.21.15.1:33370) 2024-03-25T18:34:18.526 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:34:18.524+0000 7fa000833640 1 -- 172.21.15.1:0/659640812 learned_addr learned my addr 172.21.15.1:0/659640812 (peer_addr_for_me v1:172.21.15.1:0/0) 2024-03-25T18:34:18.529 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:34:18.527+0000 7f9ff8ff9640 10 client.?.objecter ms_handle_connect 0x7f9ffc13c480 2024-03-25T18:34:18.529 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:34:18.527+0000 7f9ff8ff9640 10 client.?.objecter resend_mon_ops 2024-03-25T18:34:18.542 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:34:18.540+0000 7f9ff8ff9640 1 -- 172.21.15.1:0/659640812 <== mon.1 v1:172.21.15.115:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2061550790 0 0) 0x7f9ffc143cc0 con 0x7f9ffc13c480 2024-03-25T18:34:18.542 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:34:18.540+0000 7f9ff8ff9640 10 monclient(hunting): _negotiate 2024-03-25T18:34:18.542 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:34:18.540+0000 7f9ff8ff9640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:34:18.542 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:34:18.540+0000 7f9ff8ff9640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:34:18.542 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:34:18.540+0000 7f9ff8ff9640 10 monclient(hunting): my global_id is 4558 2024-03-25T18:34:18.542 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:34:18.540+0000 7f9ff8ff9640 1 -- 172.21.15.1:0/659640812 --> v1:172.21.15.115:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f9fe8004eb0 con 0x7f9ffc13c480 2024-03-25T18:34:18.547 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:34:18.545+0000 7f9ff8ff9640 10 client.?.objecter ms_handle_connect 0x7f9ffc0b4670 2024-03-25T18:34:18.547 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:34:18.545+0000 7f9ff8ff9640 10 client.?.objecter resend_mon_ops 2024-03-25T18:34:18.549 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:34:18.547+0000 7f9ff8ff9640 10 client.?.objecter ms_handle_connect 0x7f9ffc0b4ff0 2024-03-25T18:34:18.549 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:34:18.547+0000 7f9ff8ff9640 10 client.?.objecter resend_mon_ops 2024-03-25T18:34:18.560 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:34:18.558+0000 7f9ff8ff9640 1 -- 172.21.15.1:0/659640812 <== mon.1 v1:172.21.15.115:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1604431995 0 0) 0x7f9fe8004eb0 con 0x7f9ffc13c480 2024-03-25T18:34:18.560 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:34:18.558+0000 7f9ff8ff9640 1 -- 172.21.15.1:0/659640812 --> v1:172.21.15.115:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f9ffc143cc0 con 0x7f9ffc13c480 2024-03-25T18:34:18.560 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:34:18.558+0000 7f9ff8ff9640 1 -- 172.21.15.1:0/659640812 <== mon.1 v1:172.21.15.115:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1586965025 0 0) 0x7f9ff4001c90 con 0x7f9ffc13c480 2024-03-25T18:34:18.560 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:34:18.558+0000 7f9ff8ff9640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:34:18.560 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:34:18.558+0000 7f9ff8ff9640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-03-25T18:34:18.560 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:34:18.558+0000 7f9ff8ff9640 10 monclient(hunting): dump: 2024-03-25T18:34:18.561 INFO:teuthology.misc.health.smithi001.stderr:epoch 1 2024-03-25T18:34:18.561 INFO:teuthology.misc.health.smithi001.stderr:fsid c8ea72cf-9e99-4383-9b52-6c23d365fdfc 2024-03-25T18:34:18.561 INFO:teuthology.misc.health.smithi001.stderr:last_changed 2024-03-25T18:25:50.510519+0000 2024-03-25T18:34:18.561 INFO:teuthology.misc.health.smithi001.stderr:created 2024-03-25T18:25:50.510519+0000 2024-03-25T18:34:18.561 INFO:teuthology.misc.health.smithi001.stderr:min_mon_release 19 (squid) 2024-03-25T18:34:18.561 INFO:teuthology.misc.health.smithi001.stderr:election_strategy: 3 2024-03-25T18:34:18.561 INFO:teuthology.misc.health.smithi001.stderr:0: v1:172.21.15.1:6789/0 mon.a 2024-03-25T18:34:18.561 INFO:teuthology.misc.health.smithi001.stderr:1: v1:172.21.15.115:6789/0 mon.b 2024-03-25T18:34:18.561 INFO:teuthology.misc.health.smithi001.stderr:2: v1:172.21.15.1:6790/0 mon.c 2024-03-25T18:34:18.561 INFO:teuthology.misc.health.smithi001.stderr: 2024-03-25T18:34:18.561 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:34:18.558+0000 7f9ff8ff9640 10 monclient(hunting): _finish_auth 0 2024-03-25T18:34:18.563 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:34:18.561+0000 7f9ff8ff9640 1 -- 172.21.15.1:0/659640812 <== mon.2 v1:172.21.15.1:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2193650545 0 0) 0x7f9ffc13f800 con 0x7f9ffc0b4ff0 2024-03-25T18:34:18.563 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:34:18.561+0000 7f9ff8ff9640 10 monclient(hunting): _negotiate 2024-03-25T18:34:18.563 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:34:18.561+0000 7f9ff8ff9640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:34:18.563 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:34:18.561+0000 7f9ff8ff9640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:34:18.563 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:34:18.561+0000 7f9ff8ff9640 10 monclient(hunting): my global_id is 4559 2024-03-25T18:34:18.563 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:34:18.561+0000 7f9ff8ff9640 1 -- 172.21.15.1:0/659640812 --> v1:172.21.15.1:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f9fe8004eb0 con 0x7f9ffc0b4ff0 2024-03-25T18:34:18.569 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:34:18.567+0000 7f9ff8ff9640 1 -- 172.21.15.1:0/659640812 <== mon.1 v1:172.21.15.115:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 572559163 0 0) 0x7f9ffc143cc0 con 0x7f9ffc13c480 2024-03-25T18:34:18.569 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:34:18.567+0000 7f9ff8ff9640 1 -- 172.21.15.1:0/659640812 >> v1:172.21.15.1:6790/0 conn(0x7f9ffc0b4ff0 legacy=0x7f9ffc13ce70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:34:18.569 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:34:18.567+0000 7f9ff8ff9640 1 -- 172.21.15.1:0/659640812 >> v1:172.21.15.1:6789/0 conn(0x7f9ffc0b4670 legacy=0x7f9ffc13e320 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:34:18.569 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:34:18.567+0000 7f9ff8ff9640 10 monclient: _finish_hunting 0 2024-03-25T18:34:18.569 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:34:18.567+0000 7f9ff8ff9640 1 monclient: found mon.b 2024-03-25T18:34:18.569 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:34:18.567+0000 7f9ff8ff9640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:34:18.569 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:34:18.567+0000 7f9ff8ff9640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.115:6789/0 2024-03-25T18:34:18.569 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:34:18.567+0000 7f9ff8ff9640 1 -- 172.21.15.1:0/659640812 --> v1:172.21.15.115:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9ffc13f990 con 0x7f9ffc13c480 2024-03-25T18:34:18.569 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:34:18.567+0000 7f9ff8ff9640 10 monclient: _finish_auth 0 2024-03-25T18:34:18.570 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:34:18.567+0000 7f9ff8ff9640 10 monclient: _check_auth_tickets 2024-03-25T18:34:18.570 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:34:18.567+0000 7f9ff8ff9640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:34:18.570 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:34:18.567+0000 7fa002296640 5 monclient: authenticate success, global_id 4558 2024-03-25T18:34:18.570 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:34:18.567+0000 7fa002296640 10 monclient: _renew_subs 2024-03-25T18:34:18.570 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:34:18.567+0000 7fa002296640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.115:6789/0 2024-03-25T18:34:18.570 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:34:18.567+0000 7fa002296640 1 -- 172.21.15.1:0/659640812 --> v1:172.21.15.115:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9ffc1436e0 con 0x7f9ffc13c480 2024-03-25T18:34:18.570 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:34:18.567+0000 7fa002296640 10 client.4558.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T18:34:18.570 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:34:18.567+0000 7fa002296640 10 monclient: _renew_subs 2024-03-25T18:34:18.570 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:34:18.567+0000 7fa002296640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.115:6789/0 2024-03-25T18:34:18.570 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:34:18.567+0000 7fa002296640 1 -- 172.21.15.1:0/659640812 --> v1:172.21.15.115:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f9ffc1e3570 con 0x7f9ffc13c480 2024-03-25T18:34:18.570 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:34:18.567+0000 7fa002296640 1 librados: init done 2024-03-25T18:34:18.571 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:34:18.568+0000 7fa002296640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T18:34:18.571 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:34:18.568+0000 7fa002296640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:34:18.571 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:34:18.568+0000 7fa002296640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.115:6789/0 2024-03-25T18:34:18.571 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:34:18.568+0000 7fa002296640 1 -- 172.21.15.1:0/659640812 --> v1:172.21.15.115:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f9ffc06dbd0 con 0x7f9ffc13c480 2024-03-25T18:34:18.852 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:34:18.850+0000 7f9ff8ff9640 1 -- 172.21.15.1:0/659640812 <== mon.1 v1:172.21.15.115:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f9ff4003990 con 0x7f9ffc13c480 2024-03-25T18:34:18.852 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:34:18.850+0000 7f9ff8ff9640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:34:18.853 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:34:18.850+0000 7f9ff8ff9640 1 -- 172.21.15.1:0/659640812 <== mon.1 v1:172.21.15.115:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1586965025 0 0) 0x7f9ff4003d20 con 0x7f9ffc13c480 2024-03-25T18:34:18.853 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:34:18.850+0000 7f9ff8ff9640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:34:18.853 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:34:18.850+0000 7f9ff8ff9640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-25T18:34:18.853 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:34:18.850+0000 7f9ff8ff9640 10 monclient: dump: 2024-03-25T18:34:18.853 INFO:teuthology.misc.health.smithi001.stderr:epoch 1 2024-03-25T18:34:18.853 INFO:teuthology.misc.health.smithi001.stderr:fsid c8ea72cf-9e99-4383-9b52-6c23d365fdfc 2024-03-25T18:34:18.853 INFO:teuthology.misc.health.smithi001.stderr:last_changed 2024-03-25T18:25:50.510519+0000 2024-03-25T18:34:18.853 INFO:teuthology.misc.health.smithi001.stderr:created 2024-03-25T18:25:50.510519+0000 2024-03-25T18:34:18.853 INFO:teuthology.misc.health.smithi001.stderr:min_mon_release 19 (squid) 2024-03-25T18:34:18.853 INFO:teuthology.misc.health.smithi001.stderr:election_strategy: 3 2024-03-25T18:34:18.853 INFO:teuthology.misc.health.smithi001.stderr:0: v1:172.21.15.1:6789/0 mon.a 2024-03-25T18:34:18.853 INFO:teuthology.misc.health.smithi001.stderr:1: v1:172.21.15.115:6789/0 mon.b 2024-03-25T18:34:18.853 INFO:teuthology.misc.health.smithi001.stderr:2: v1:172.21.15.1:6790/0 mon.c 2024-03-25T18:34:18.853 INFO:teuthology.misc.health.smithi001.stderr: 2024-03-25T18:34:18.853 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:34:18.850+0000 7f9ffa7fc640 4 set_mon_vals no callback set 2024-03-25T18:34:18.854 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:34:18.851+0000 7f9ff8ff9640 1 -- 172.21.15.1:0/659640812 <== mon.1 v1:172.21.15.115:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 21787975 0 0) 0x7f9ff4003f40 con 0x7f9ffc13c480 2024-03-25T18:34:18.854 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:34:18.852+0000 7f9ffbfff640 10 monclient: get_auth_request con 0x7f9fe806dd00 auth_method 0 2024-03-25T18:34:18.854 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:34:18.852+0000 7f9ff8ff9640 10 client.4558.objecter ms_handle_connect 0x7f9fe806dd00 2024-03-25T18:34:18.859 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:34:18.857+0000 7f9ff8ff9640 1 -- 172.21.15.1:0/659640812 <== mon.1 v1:172.21.15.115:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 996254230 0 0) 0x7f9ff4054b20 con 0x7f9ffc13c480 2024-03-25T18:34:18.859 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:34:18.857+0000 7f9ff8ff9640 10 client.4558.objecter ms_dispatch 0x7f9ffc000b90 osd_map(10..10 src has 1..10) v4 2024-03-25T18:34:18.859 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:34:18.857+0000 7f9ff8ff9640 3 client.4558.objecter handle_osd_map got epochs [10,10] > 0 2024-03-25T18:34:18.859 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:34:18.857+0000 7f9ff8ff9640 3 client.4558.objecter handle_osd_map decoding full epoch 10 2024-03-25T18:34:18.859 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:34:18.857+0000 7f9ff8ff9640 20 client.4558.objecter dump_active .. 0 homeless 2024-03-25T18:34:18.859 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:34:18.857+0000 7f9ff8ff9640 1 -- 172.21.15.1:0/659640812 <== mon.1 v1:172.21.15.115:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f9ff4050630 con 0x7f9ffc13c480 2024-03-25T18:34:18.859 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:34:18.857+0000 7f9ff8ff9640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:34:18.860 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:34:18.857+0000 7f9ff8ff9640 10 monclient: _finish_command 1 = system:0 2024-03-25T18:34:19.082 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:34:19.080+0000 7fa002296640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:34:19.082 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:34:19.080+0000 7fa002296640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:34:19.082 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:34:19.080+0000 7fa002296640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.115:6789/0 2024-03-25T18:34:19.082 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:34:19.080+0000 7fa002296640 1 -- 172.21.15.1:0/659640812 --> v1:172.21.15.115:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f9ffc0688c0 con 0x7f9ffc13c480 2024-03-25T18:34:19.170 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:34:19.168+0000 7f9ff8ff9640 1 -- 172.21.15.1:0/659640812 <== mon.1 v1:172.21.15.115:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 4092213348) 0x7f9ff4052560 con 0x7f9ffc13c480 2024-03-25T18:34:19.170 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:34:19.168+0000 7f9ff8ff9640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:34:19.170 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:34:19.168+0000 7f9ff8ff9640 10 monclient: _finish_command 2 = system:0 2024-03-25T18:34:19.171 INFO:teuthology.misc.health.smithi001.stdout: 2024-03-25T18:34:19.171 INFO:teuthology.misc.health.smithi001.stdout:{"epoch":10,"fsid":"c8ea72cf-9e99-4383-9b52-6c23d365fdfc","created":"2024-03-25T18:26:38.856638+0000","modified":"2024-03-25T18:26:44.224208+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T18:26:44.224208+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"b9f34922-3a84-49be-8426-807c1eceaefc","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"42bf306f-55e3-4634-9874-8b4b73ac6644","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"292cb23a-18a9-4207-a7cc-4d5727feaec0","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"45e0e9ab-898d-4673-9590-b4117bc5e84a","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"831a5a09-73ab-4a61-9872-088e76bdbc66","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"88cd903d-3b55-4840-9396-6e3cfe005282","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"89db7355-bc9d-46f4-8ab0-e8d9e744e026","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"cac47bee-0914-4f08-896f-348838882866","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T18:34:19.173 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:34:19.171+0000 7fa002296640 10 librados: watch_flush enter 2024-03-25T18:34:19.173 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:34:19.171+0000 7fa002296640 10 librados: watch_flush exit 2024-03-25T18:34:19.173 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:34:19.171+0000 7fa002296640 20 client.4558.objecter shutdown clearing up homeless session... 2024-03-25T18:34:19.173 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:34:19.171+0000 7fa002296640 10 client.4558.objecter successfully canceled tick 2024-03-25T18:34:19.173 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:34:19.171+0000 7fa002296640 1 -- 172.21.15.1:0/659640812 >> v1:172.21.15.115:6800/4237379176 conn(0x7f9fe806dd00 legacy=0x7f9fe80701c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:34:19.173 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:34:19.171+0000 7fa002296640 10 monclient: shutdown 2024-03-25T18:34:19.174 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:34:19.171+0000 7fa002296640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:34:19.174 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:34:19.171+0000 7fa002296640 1 -- 172.21.15.1:0/659640812 >> v1:172.21.15.115:6789/0 conn(0x7f9ffc13c480 legacy=0x7f9ffc13d580 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:34:19.174 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:34:19.171+0000 7fa002296640 1 -- 172.21.15.1:0/659640812 shutdown_connections 2024-03-25T18:34:19.174 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:34:19.171+0000 7fa002296640 1 -- 172.21.15.1:0/659640812 >> 172.21.15.1:0/659640812 conn(0x7f9ffc0617a0 msgr2=0x7f9ffc1407f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:34:19.174 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:34:19.171+0000 7fa002296640 1 -- 172.21.15.1:0/659640812 shutdown_connections 2024-03-25T18:34:19.174 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:34:19.171+0000 7fa002296640 1 -- 172.21.15.1:0/659640812 wait complete. 2024-03-25T18:34:19.174 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:34:19.171+0000 7fa002296640 1 librados: shutdown 2024-03-25T18:34:19.184 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T18:34:22.383 INFO:tasks.ceph.mon.a.smithi001.stderr:==00:00:08:00.941 34876== Warning: unimplemented fcntl command: 1036 2024-03-25T18:34:22.388 INFO:tasks.ceph.mon.a.smithi001.stderr:==00:00:08:00.946 34876== Warning: unimplemented fcntl command: 1036 2024-03-25T18:34:22.427 INFO:tasks.ceph.mon.a.smithi001.stderr:==00:00:08:00.985 34876== Warning: unimplemented fcntl command: 1036 2024-03-25T18:34:22.428 INFO:tasks.ceph.mon.b.smithi115.stderr:==00:00:08:00.911 34836== Warning: unimplemented fcntl command: 1036 2024-03-25T18:34:22.442 INFO:tasks.ceph.mon.b.smithi115.stderr:==00:00:08:00.926 34836== Warning: unimplemented fcntl command: 1036 2024-03-25T18:34:22.446 INFO:tasks.ceph.mon.c.smithi001.stderr:==00:00:08:01.005 34877== Warning: unimplemented fcntl command: 1036 2024-03-25T18:34:22.458 INFO:tasks.ceph.mon.c.smithi001.stderr:==00:00:08:01.017 34877== Warning: unimplemented fcntl command: 1036 2024-03-25T18:34:22.464 INFO:tasks.ceph.mon.b.smithi115.stderr:==00:00:08:00.947 34836== Warning: unimplemented fcntl command: 1036 2024-03-25T18:34:22.498 INFO:tasks.ceph.mon.c.smithi001.stderr:==00:00:08:01.057 34877== Warning: unimplemented fcntl command: 1036 2024-03-25T18:34:25.992 DEBUG:teuthology.orchestra.run.smithi001:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T18:34:26.108 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:34:26.106+0000 7f128101e640 10 monclient: get_monmap_and_config 2024-03-25T18:34:26.109 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:34:26.106+0000 7f128101e640 10 monclient: build_initial_monmap 2024-03-25T18:34:26.109 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:34:26.106+0000 7f128101e640 10 monclient: monmap: 2024-03-25T18:34:26.109 INFO:teuthology.misc.health.smithi001.stderr:epoch 0 2024-03-25T18:34:26.109 INFO:teuthology.misc.health.smithi001.stderr:fsid c8ea72cf-9e99-4383-9b52-6c23d365fdfc 2024-03-25T18:34:26.109 INFO:teuthology.misc.health.smithi001.stderr:last_changed 2024-03-25T18:34:26.108389+0000 2024-03-25T18:34:26.109 INFO:teuthology.misc.health.smithi001.stderr:created 2024-03-25T18:34:26.108389+0000 2024-03-25T18:34:26.109 INFO:teuthology.misc.health.smithi001.stderr:min_mon_release 0 (unknown) 2024-03-25T18:34:26.109 INFO:teuthology.misc.health.smithi001.stderr:election_strategy: 3 2024-03-25T18:34:26.110 INFO:teuthology.misc.health.smithi001.stderr:0: v1:172.21.15.1:6789/0 mon.noname-a 2024-03-25T18:34:26.110 INFO:teuthology.misc.health.smithi001.stderr:1: v1:172.21.15.115:6789/0 mon.noname-c 2024-03-25T18:34:26.110 INFO:teuthology.misc.health.smithi001.stderr:2: v1:172.21.15.1:6790/0 mon.noname-b 2024-03-25T18:34:26.110 INFO:teuthology.misc.health.smithi001.stderr: 2024-03-25T18:34:26.111 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:34:26.107+0000 7f128101e640 1 Processor -- start 2024-03-25T18:34:26.112 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:34:26.107+0000 7f128101e640 1 -- start start 2024-03-25T18:34:26.112 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:34:26.107+0000 7f128101e640 10 monclient: init 2024-03-25T18:34:26.112 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:34:26.108+0000 7f128101e640 10 monclient: _reopen_session rank -1 2024-03-25T18:34:26.112 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:34:26.108+0000 7f128101e640 10 monclient: _add_conns ranks=[0,2,1] 2024-03-25T18:34:26.112 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:34:26.108+0000 7f128101e640 10 monclient(hunting): picked mon.noname-a con 0x7f127c0b4e70 addr v1:172.21.15.1:6789/0 2024-03-25T18:34:26.112 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:34:26.108+0000 7f128101e640 10 monclient(hunting): picked mon.noname-b con 0x7f127c0b44b0 addr v1:172.21.15.1:6790/0 2024-03-25T18:34:26.112 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:34:26.108+0000 7f128101e640 10 monclient(hunting): picked mon.noname-c con 0x7f127c0b1b00 addr v1:172.21.15.115:6789/0 2024-03-25T18:34:26.112 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:34:26.108+0000 7f128101e640 1 -- --> v1:172.21.15.1:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f127c0a96f0 con 0x7f127c0b4e70 2024-03-25T18:34:26.112 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:34:26.108+0000 7f128101e640 1 -- --> v1:172.21.15.115:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f127c0ab0e0 con 0x7f127c0b1b00 2024-03-25T18:34:26.112 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:34:26.108+0000 7f128101e640 1 -- --> v1:172.21.15.1:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f127c145f70 con 0x7f127c0b44b0 2024-03-25T18:34:26.112 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:34:26.108+0000 7f128101e640 10 monclient(hunting): _renew_subs 2024-03-25T18:34:26.113 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:34:26.108+0000 7f128101e640 10 monclient(hunting): authenticate will time out at 7946.579226s 2024-03-25T18:34:26.122 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:34:26.120+0000 7f1273fff640 1 --1- >> v1:172.21.15.1:6790/0 conn(0x7f127c0b44b0 0x7f127c145860 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.1:6790/0 says I am v1:172.21.15.1:55180/0 (socket says 172.21.15.1:55180) 2024-03-25T18:34:26.122 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:34:26.120+0000 7f1273fff640 1 -- 172.21.15.1:0/296985045 learned_addr learned my addr 172.21.15.1:0/296985045 (peer_addr_for_me v1:172.21.15.1:0/0) 2024-03-25T18:34:26.131 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:34:26.129+0000 7f127a59c640 1 -- 172.21.15.1:0/296985045 <== mon.0 v1:172.21.15.1:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1521730879 0 0) 0x7f127c0a96f0 con 0x7f127c0b4e70 2024-03-25T18:34:26.131 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:34:26.129+0000 7f127a59c640 10 monclient(hunting): _negotiate 2024-03-25T18:34:26.132 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:34:26.129+0000 7f127a59c640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:34:26.132 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:34:26.129+0000 7f127a59c640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:34:26.132 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:34:26.129+0000 7f127a59c640 10 monclient(hunting): my global_id is 4554 2024-03-25T18:34:26.132 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:34:26.129+0000 7f127a59c640 1 -- 172.21.15.1:0/296985045 --> v1:172.21.15.1:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f1268003520 con 0x7f127c0b4e70 2024-03-25T18:34:26.139 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:34:26.137+0000 7f127a59c640 1 -- 172.21.15.1:0/296985045 <== mon.2 v1:172.21.15.1:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 249329472 0 0) 0x7f127c145f70 con 0x7f127c0b44b0 2024-03-25T18:34:26.139 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:34:26.137+0000 7f127a59c640 10 monclient(hunting): _negotiate 2024-03-25T18:34:26.139 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:34:26.137+0000 7f127a59c640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:34:26.139 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:34:26.137+0000 7f127a59c640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:34:26.139 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:34:26.137+0000 7f127a59c640 10 monclient(hunting): my global_id is 4562 2024-03-25T18:34:26.139 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:34:26.137+0000 7f127a59c640 1 -- 172.21.15.1:0/296985045 --> v1:172.21.15.1:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f127c0a96f0 con 0x7f127c0b44b0 2024-03-25T18:34:26.144 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:34:26.141+0000 7f127a59c640 1 -- 172.21.15.1:0/296985045 <== mon.0 v1:172.21.15.1:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1510620476 0 0) 0x7f1268003520 con 0x7f127c0b4e70 2024-03-25T18:34:26.144 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:34:26.141+0000 7f127a59c640 1 -- 172.21.15.1:0/296985045 >> v1:172.21.15.1:6790/0 conn(0x7f127c0b44b0 legacy=0x7f127c145860 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:34:26.144 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:34:26.141+0000 7f127a59c640 1 -- 172.21.15.1:0/296985045 >> v1:172.21.15.115:6789/0 conn(0x7f127c0b1b00 legacy=0x7f127c14dc00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:34:26.144 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:34:26.141+0000 7f127a59c640 10 monclient: _finish_hunting 0 2024-03-25T18:34:26.144 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:34:26.141+0000 7f127a59c640 1 monclient: found mon.noname-a 2024-03-25T18:34:26.144 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:34:26.141+0000 7f127a59c640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:34:26.144 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:34:26.141+0000 7f127a59c640 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.1:6789/0 2024-03-25T18:34:26.144 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:34:26.141+0000 7f127a59c640 1 -- 172.21.15.1:0/296985045 --> v1:172.21.15.1:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f127c0a9b80 con 0x7f127c0b4e70 2024-03-25T18:34:26.145 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:34:26.141+0000 7f127a59c640 10 monclient: _finish_auth 0 2024-03-25T18:34:26.145 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:34:26.141+0000 7f127a59c640 10 monclient: _check_auth_tickets 2024-03-25T18:34:26.145 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:34:26.141+0000 7f127a59c640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:34:26.145 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:34:26.141+0000 7f127a59c640 1 -- 172.21.15.1:0/296985045 <== mon.0 v1:172.21.15.1:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1586965025 0 0) 0x7f126c001690 con 0x7f127c0b4e70 2024-03-25T18:34:26.145 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:34:26.142+0000 7f127a59c640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:34:26.145 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:34:26.142+0000 7f127a59c640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-03-25T18:34:26.145 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:34:26.142+0000 7f127a59c640 10 monclient: dump: 2024-03-25T18:34:26.145 INFO:teuthology.misc.health.smithi001.stderr:epoch 1 2024-03-25T18:34:26.145 INFO:teuthology.misc.health.smithi001.stderr:fsid c8ea72cf-9e99-4383-9b52-6c23d365fdfc 2024-03-25T18:34:26.145 INFO:teuthology.misc.health.smithi001.stderr:last_changed 2024-03-25T18:25:50.510519+0000 2024-03-25T18:34:26.145 INFO:teuthology.misc.health.smithi001.stderr:created 2024-03-25T18:25:50.510519+0000 2024-03-25T18:34:26.145 INFO:teuthology.misc.health.smithi001.stderr:min_mon_release 19 (squid) 2024-03-25T18:34:26.146 INFO:teuthology.misc.health.smithi001.stderr:election_strategy: 3 2024-03-25T18:34:26.146 INFO:teuthology.misc.health.smithi001.stderr:0: v1:172.21.15.1:6789/0 mon.a 2024-03-25T18:34:26.146 INFO:teuthology.misc.health.smithi001.stderr:1: v1:172.21.15.115:6789/0 mon.b 2024-03-25T18:34:26.146 INFO:teuthology.misc.health.smithi001.stderr:2: v1:172.21.15.1:6790/0 mon.c 2024-03-25T18:34:26.146 INFO:teuthology.misc.health.smithi001.stderr: 2024-03-25T18:34:26.146 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:34:26.142+0000 7f128101e640 5 monclient: authenticate success, global_id 4554 2024-03-25T18:34:26.146 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:34:26.142+0000 7f128101e640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T18:34:26.156 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:34:26.154+0000 7f127a59c640 1 -- 172.21.15.1:0/296985045 <== mon.0 v1:172.21.15.1:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f126c0017f0 con 0x7f127c0b4e70 2024-03-25T18:34:26.156 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:34:26.154+0000 7f127a59c640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:34:26.156 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:34:26.154+0000 7f127a59c640 1 -- 172.21.15.1:0/296985045 <== mon.0 v1:172.21.15.1:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1586965025 0 0) 0x7f126c001c00 con 0x7f127c0b4e70 2024-03-25T18:34:26.156 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:34:26.154+0000 7f127a59c640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:34:26.156 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:34:26.154+0000 7f127a59c640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-25T18:34:26.157 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:34:26.154+0000 7f127a59c640 10 monclient: dump: 2024-03-25T18:34:26.157 INFO:teuthology.misc.health.smithi001.stderr:epoch 1 2024-03-25T18:34:26.157 INFO:teuthology.misc.health.smithi001.stderr:fsid c8ea72cf-9e99-4383-9b52-6c23d365fdfc 2024-03-25T18:34:26.157 INFO:teuthology.misc.health.smithi001.stderr:last_changed 2024-03-25T18:25:50.510519+0000 2024-03-25T18:34:26.157 INFO:teuthology.misc.health.smithi001.stderr:created 2024-03-25T18:25:50.510519+0000 2024-03-25T18:34:26.157 INFO:teuthology.misc.health.smithi001.stderr:min_mon_release 19 (squid) 2024-03-25T18:34:26.157 INFO:teuthology.misc.health.smithi001.stderr:election_strategy: 3 2024-03-25T18:34:26.157 INFO:teuthology.misc.health.smithi001.stderr:0: v1:172.21.15.1:6789/0 mon.a 2024-03-25T18:34:26.157 INFO:teuthology.misc.health.smithi001.stderr:1: v1:172.21.15.115:6789/0 mon.b 2024-03-25T18:34:26.157 INFO:teuthology.misc.health.smithi001.stderr:2: v1:172.21.15.1:6790/0 mon.c 2024-03-25T18:34:26.157 INFO:teuthology.misc.health.smithi001.stderr: 2024-03-25T18:34:26.157 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:34:26.154+0000 7f128101e640 10 monclient: get_monmap_and_config success 2024-03-25T18:34:26.157 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:34:26.154+0000 7f128101e640 4 set_mon_vals no callback set 2024-03-25T18:34:26.160 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:34:26.155+0000 7f128101e640 10 monclient: shutdown 2024-03-25T18:34:26.160 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:34:26.155+0000 7f128101e640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:34:26.160 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:34:26.155+0000 7f128101e640 1 -- 172.21.15.1:0/296985045 >> v1:172.21.15.1:6789/0 conn(0x7f127c0b4e70 legacy=0x7f127c13f940 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:34:26.160 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:34:26.155+0000 7f128101e640 1 -- 172.21.15.1:0/296985045 shutdown_connections 2024-03-25T18:34:26.160 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:34:26.155+0000 7f128101e640 1 -- 172.21.15.1:0/296985045 >> 172.21.15.1:0/296985045 conn(0x7f127c0617a0 msgr2=0x7f127c061bb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:34:26.161 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:34:26.155+0000 7f128101e640 1 -- 172.21.15.1:0/296985045 shutdown_connections 2024-03-25T18:34:26.161 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:34:26.155+0000 7f128101e640 1 -- 172.21.15.1:0/296985045 wait complete. 2024-03-25T18:34:26.161 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:34:26.156+0000 7f128101e640 10 monclient: build_initial_monmap 2024-03-25T18:34:26.161 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:34:26.156+0000 7f128101e640 10 monclient: monmap: 2024-03-25T18:34:26.161 INFO:teuthology.misc.health.smithi001.stderr:epoch 0 2024-03-25T18:34:26.161 INFO:teuthology.misc.health.smithi001.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T18:34:26.161 INFO:teuthology.misc.health.smithi001.stderr:last_changed 0.000000 2024-03-25T18:34:26.161 INFO:teuthology.misc.health.smithi001.stderr:created 0.000000 2024-03-25T18:34:26.161 INFO:teuthology.misc.health.smithi001.stderr:min_mon_release 0 (unknown) 2024-03-25T18:34:26.161 INFO:teuthology.misc.health.smithi001.stderr:election_strategy: 1 2024-03-25T18:34:26.161 INFO:teuthology.misc.health.smithi001.stderr:0: v1:172.21.15.1:6789/0 mon.noname-a 2024-03-25T18:34:26.162 INFO:teuthology.misc.health.smithi001.stderr:1: v1:172.21.15.115:6789/0 mon.noname-b 2024-03-25T18:34:26.162 INFO:teuthology.misc.health.smithi001.stderr:2: v1:172.21.15.1:6790/0 mon.noname-c 2024-03-25T18:34:26.162 INFO:teuthology.misc.health.smithi001.stderr: 2024-03-25T18:34:26.162 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:34:26.156+0000 7f128101e640 1 librados: starting msgr at 2024-03-25T18:34:26.162 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:34:26.156+0000 7f128101e640 1 librados: starting objecter 2024-03-25T18:34:26.162 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:34:26.156+0000 7f128101e640 1 Processor -- start 2024-03-25T18:34:26.162 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:34:26.157+0000 7f128101e640 1 -- start start 2024-03-25T18:34:26.162 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:34:26.157+0000 7f128101e640 1 librados: setting wanted keys 2024-03-25T18:34:26.162 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:34:26.157+0000 7f128101e640 1 librados: calling monclient init 2024-03-25T18:34:26.162 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:34:26.157+0000 7f128101e640 10 monclient: init 2024-03-25T18:34:26.162 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:34:26.157+0000 7f128101e640 10 monclient: _reopen_session rank -1 2024-03-25T18:34:26.162 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:34:26.157+0000 7f128101e640 10 monclient: _add_conns ranks=[0,2,1] 2024-03-25T18:34:26.163 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:34:26.157+0000 7f128101e640 10 monclient(hunting): picked mon.noname-a con 0x7f127c0b1b00 addr v1:172.21.15.1:6789/0 2024-03-25T18:34:26.163 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:34:26.157+0000 7f128101e640 10 monclient(hunting): picked mon.noname-c con 0x7f127c0b44b0 addr v1:172.21.15.1:6790/0 2024-03-25T18:34:26.163 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:34:26.157+0000 7f128101e640 10 monclient(hunting): picked mon.noname-b con 0x7f127c0b4e70 addr v1:172.21.15.115:6789/0 2024-03-25T18:34:26.163 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:34:26.157+0000 7f128101e640 1 -- --> v1:172.21.15.1:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f127c138030 con 0x7f127c0b1b00 2024-03-25T18:34:26.163 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:34:26.157+0000 7f128101e640 1 -- --> v1:172.21.15.115:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f127c1daeb0 con 0x7f127c0b4e70 2024-03-25T18:34:26.163 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:34:26.157+0000 7f128101e640 1 -- --> v1:172.21.15.1:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f127c136e80 con 0x7f127c0b44b0 2024-03-25T18:34:26.163 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:34:26.157+0000 7f128101e640 10 monclient(hunting): _renew_subs 2024-03-25T18:34:26.163 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:34:26.157+0000 7f128101e640 10 monclient(hunting): authenticate will time out at 7946.628505s 2024-03-25T18:34:26.174 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:34:26.172+0000 7f127b59e640 1 --1- >> v1:172.21.15.115:6789/0 conn(0x7f127c0b4e70 0x7f127c133e80 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.115:6789/0 says I am v1:172.21.15.1:43746/0 (socket says 172.21.15.1:43746) 2024-03-25T18:34:26.174 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:34:26.172+0000 7f127b59e640 1 -- 172.21.15.1:0/1699652248 learned_addr learned my addr 172.21.15.1:0/1699652248 (peer_addr_for_me v1:172.21.15.1:0/0) 2024-03-25T18:34:26.175 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:34:26.173+0000 7f12737fe640 10 client.?.objecter ms_handle_connect 0x7f127c0b4e70 2024-03-25T18:34:26.175 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:34:26.173+0000 7f12737fe640 10 client.?.objecter resend_mon_ops 2024-03-25T18:34:26.187 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:34:26.185+0000 7f12737fe640 10 client.?.objecter ms_handle_connect 0x7f127c0b1b00 2024-03-25T18:34:26.187 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:34:26.185+0000 7f12737fe640 10 client.?.objecter resend_mon_ops 2024-03-25T18:34:26.193 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:34:26.191+0000 7f12737fe640 1 -- 172.21.15.1:0/1699652248 <== mon.1 v1:172.21.15.115:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3276726663 0 0) 0x7f127c1daeb0 con 0x7f127c0b4e70 2024-03-25T18:34:26.193 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:34:26.191+0000 7f12737fe640 10 monclient(hunting): _negotiate 2024-03-25T18:34:26.193 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:34:26.191+0000 7f12737fe640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:34:26.193 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:34:26.191+0000 7f12737fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:34:26.193 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:34:26.191+0000 7f12737fe640 10 monclient(hunting): my global_id is 4564 2024-03-25T18:34:26.193 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:34:26.191+0000 7f12737fe640 1 -- 172.21.15.1:0/1699652248 --> v1:172.21.15.115:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f125c004eb0 con 0x7f127c0b4e70 2024-03-25T18:34:26.196 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:34:26.194+0000 7f12737fe640 10 client.?.objecter ms_handle_connect 0x7f127c0b44b0 2024-03-25T18:34:26.196 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:34:26.194+0000 7f12737fe640 10 client.?.objecter resend_mon_ops 2024-03-25T18:34:26.208 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:34:26.206+0000 7f12737fe640 1 -- 172.21.15.1:0/1699652248 <== mon.0 v1:172.21.15.1:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 349851958 0 0) 0x7f127c138030 con 0x7f127c0b1b00 2024-03-25T18:34:26.209 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:34:26.206+0000 7f12737fe640 10 monclient(hunting): _negotiate 2024-03-25T18:34:26.209 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:34:26.206+0000 7f12737fe640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:34:26.209 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:34:26.206+0000 7f12737fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:34:26.209 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:34:26.206+0000 7f12737fe640 10 monclient(hunting): my global_id is 4557 2024-03-25T18:34:26.209 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:34:26.206+0000 7f12737fe640 1 -- 172.21.15.1:0/1699652248 --> v1:172.21.15.1:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f127c1daeb0 con 0x7f127c0b1b00 2024-03-25T18:34:26.213 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:34:26.211+0000 7f12737fe640 1 -- 172.21.15.1:0/1699652248 <== mon.1 v1:172.21.15.115:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3230674794 0 0) 0x7f125c004eb0 con 0x7f127c0b4e70 2024-03-25T18:34:26.213 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:34:26.211+0000 7f12737fe640 1 -- 172.21.15.1:0/1699652248 --> v1:172.21.15.115:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f127c138030 con 0x7f127c0b4e70 2024-03-25T18:34:26.213 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:34:26.211+0000 7f12737fe640 1 -- 172.21.15.1:0/1699652248 <== mon.1 v1:172.21.15.115:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1586965025 0 0) 0x7f1274001cd0 con 0x7f127c0b4e70 2024-03-25T18:34:26.213 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:34:26.211+0000 7f12737fe640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:34:26.213 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:34:26.211+0000 7f12737fe640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-03-25T18:34:26.214 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:34:26.211+0000 7f12737fe640 10 monclient(hunting): dump: 2024-03-25T18:34:26.214 INFO:teuthology.misc.health.smithi001.stderr:epoch 1 2024-03-25T18:34:26.214 INFO:teuthology.misc.health.smithi001.stderr:fsid c8ea72cf-9e99-4383-9b52-6c23d365fdfc 2024-03-25T18:34:26.214 INFO:teuthology.misc.health.smithi001.stderr:last_changed 2024-03-25T18:25:50.510519+0000 2024-03-25T18:34:26.214 INFO:teuthology.misc.health.smithi001.stderr:created 2024-03-25T18:25:50.510519+0000 2024-03-25T18:34:26.214 INFO:teuthology.misc.health.smithi001.stderr:min_mon_release 19 (squid) 2024-03-25T18:34:26.214 INFO:teuthology.misc.health.smithi001.stderr:election_strategy: 3 2024-03-25T18:34:26.214 INFO:teuthology.misc.health.smithi001.stderr:0: v1:172.21.15.1:6789/0 mon.a 2024-03-25T18:34:26.214 INFO:teuthology.misc.health.smithi001.stderr:1: v1:172.21.15.115:6789/0 mon.b 2024-03-25T18:34:26.214 INFO:teuthology.misc.health.smithi001.stderr:2: v1:172.21.15.1:6790/0 mon.c 2024-03-25T18:34:26.214 INFO:teuthology.misc.health.smithi001.stderr: 2024-03-25T18:34:26.214 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:34:26.211+0000 7f12737fe640 10 monclient(hunting): _finish_auth 0 2024-03-25T18:34:26.215 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:34:26.212+0000 7f12737fe640 1 -- 172.21.15.1:0/1699652248 <== mon.2 v1:172.21.15.1:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 645920444 0 0) 0x7f127c136e80 con 0x7f127c0b44b0 2024-03-25T18:34:26.215 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:34:26.212+0000 7f12737fe640 10 monclient(hunting): _negotiate 2024-03-25T18:34:26.215 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:34:26.212+0000 7f12737fe640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:34:26.215 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:34:26.212+0000 7f12737fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:34:26.215 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:34:26.212+0000 7f12737fe640 10 monclient(hunting): my global_id is 4565 2024-03-25T18:34:26.215 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:34:26.212+0000 7f12737fe640 1 -- 172.21.15.1:0/1699652248 --> v1:172.21.15.1:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f125c004eb0 con 0x7f127c0b44b0 2024-03-25T18:34:26.229 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:34:26.226+0000 7f12737fe640 1 -- 172.21.15.1:0/1699652248 <== mon.1 v1:172.21.15.115:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 3333761801 0 0) 0x7f127c138030 con 0x7f127c0b4e70 2024-03-25T18:34:26.229 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:34:26.226+0000 7f12737fe640 1 -- 172.21.15.1:0/1699652248 >> v1:172.21.15.1:6790/0 conn(0x7f127c0b44b0 legacy=0x7f127c1db3d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:34:26.229 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:34:26.227+0000 7f12737fe640 1 -- 172.21.15.1:0/1699652248 >> v1:172.21.15.1:6789/0 conn(0x7f127c0b1b00 legacy=0x7f127c136770 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:34:26.229 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:34:26.227+0000 7f12737fe640 10 monclient: _finish_hunting 0 2024-03-25T18:34:26.229 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:34:26.227+0000 7f12737fe640 1 monclient: found mon.b 2024-03-25T18:34:26.229 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:34:26.227+0000 7f12737fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:34:26.229 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:34:26.227+0000 7f12737fe640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.115:6789/0 2024-03-25T18:34:26.230 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:34:26.227+0000 7f12737fe640 1 -- 172.21.15.1:0/1699652248 --> v1:172.21.15.115:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f127c137010 con 0x7f127c0b4e70 2024-03-25T18:34:26.230 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:34:26.227+0000 7f12737fe640 10 monclient: _finish_auth 0 2024-03-25T18:34:26.230 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:34:26.227+0000 7f12737fe640 10 monclient: _check_auth_tickets 2024-03-25T18:34:26.230 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:34:26.227+0000 7f12737fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:34:26.230 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:34:26.227+0000 7f128101e640 5 monclient: authenticate success, global_id 4564 2024-03-25T18:34:26.230 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:34:26.227+0000 7f128101e640 10 monclient: _renew_subs 2024-03-25T18:34:26.230 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:34:26.227+0000 7f128101e640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.115:6789/0 2024-03-25T18:34:26.230 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:34:26.227+0000 7f128101e640 1 -- 172.21.15.1:0/1699652248 --> v1:172.21.15.115:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f127c1417d0 con 0x7f127c0b4e70 2024-03-25T18:34:26.230 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:34:26.227+0000 7f128101e640 10 client.4564.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T18:34:26.230 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:34:26.227+0000 7f128101e640 10 monclient: _renew_subs 2024-03-25T18:34:26.230 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:34:26.227+0000 7f128101e640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.115:6789/0 2024-03-25T18:34:26.230 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:34:26.227+0000 7f128101e640 1 -- 172.21.15.1:0/1699652248 --> v1:172.21.15.115:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f127c1dbcc0 con 0x7f127c0b4e70 2024-03-25T18:34:26.230 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:34:26.227+0000 7f128101e640 1 librados: init done 2024-03-25T18:34:26.231 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:34:26.228+0000 7f128101e640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T18:34:26.231 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:34:26.228+0000 7f128101e640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:34:26.231 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:34:26.228+0000 7f128101e640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.115:6789/0 2024-03-25T18:34:26.231 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:34:26.228+0000 7f128101e640 1 -- 172.21.15.1:0/1699652248 --> v1:172.21.15.115:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f127c0b2090 con 0x7f127c0b4e70 2024-03-25T18:34:26.508 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:34:26.507+0000 7f12737fe640 1 -- 172.21.15.1:0/1699652248 <== mon.1 v1:172.21.15.115:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f1274003990 con 0x7f127c0b4e70 2024-03-25T18:34:26.509 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:34:26.507+0000 7f12737fe640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:34:26.509 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:34:26.507+0000 7f12737fe640 1 -- 172.21.15.1:0/1699652248 <== mon.1 v1:172.21.15.115:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1586965025 0 0) 0x7f1274003da0 con 0x7f127c0b4e70 2024-03-25T18:34:26.509 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:34:26.507+0000 7f12737fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:34:26.509 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:34:26.507+0000 7f12737fe640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-25T18:34:26.509 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:34:26.507+0000 7f12737fe640 10 monclient: dump: 2024-03-25T18:34:26.509 INFO:teuthology.misc.health.smithi001.stderr:epoch 1 2024-03-25T18:34:26.509 INFO:teuthology.misc.health.smithi001.stderr:fsid c8ea72cf-9e99-4383-9b52-6c23d365fdfc 2024-03-25T18:34:26.509 INFO:teuthology.misc.health.smithi001.stderr:last_changed 2024-03-25T18:25:50.510519+0000 2024-03-25T18:34:26.509 INFO:teuthology.misc.health.smithi001.stderr:created 2024-03-25T18:25:50.510519+0000 2024-03-25T18:34:26.509 INFO:teuthology.misc.health.smithi001.stderr:min_mon_release 19 (squid) 2024-03-25T18:34:26.509 INFO:teuthology.misc.health.smithi001.stderr:election_strategy: 3 2024-03-25T18:34:26.510 INFO:teuthology.misc.health.smithi001.stderr:0: v1:172.21.15.1:6789/0 mon.a 2024-03-25T18:34:26.510 INFO:teuthology.misc.health.smithi001.stderr:1: v1:172.21.15.115:6789/0 mon.b 2024-03-25T18:34:26.510 INFO:teuthology.misc.health.smithi001.stderr:2: v1:172.21.15.1:6790/0 mon.c 2024-03-25T18:34:26.510 INFO:teuthology.misc.health.smithi001.stderr: 2024-03-25T18:34:26.510 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:34:26.507+0000 7f1279d9b640 4 set_mon_vals no callback set 2024-03-25T18:34:26.511 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:34:26.507+0000 7f12737fe640 1 -- 172.21.15.1:0/1699652248 <== mon.1 v1:172.21.15.115:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 21787975 0 0) 0x7f1274003f80 con 0x7f127c0b4e70 2024-03-25T18:34:26.511 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:34:26.508+0000 7f127ad9d640 10 monclient: get_auth_request con 0x7f125c06de00 auth_method 0 2024-03-25T18:34:26.511 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:34:26.508+0000 7f12737fe640 10 client.4564.objecter ms_handle_connect 0x7f125c06de00 2024-03-25T18:34:26.515 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:34:26.512+0000 7f12737fe640 1 -- 172.21.15.1:0/1699652248 <== mon.1 v1:172.21.15.115:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 996254230 0 0) 0x7f1274054a80 con 0x7f127c0b4e70 2024-03-25T18:34:26.515 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:34:26.512+0000 7f12737fe640 10 client.4564.objecter ms_dispatch 0x7f127c000b90 osd_map(10..10 src has 1..10) v4 2024-03-25T18:34:26.515 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:34:26.512+0000 7f12737fe640 3 client.4564.objecter handle_osd_map got epochs [10,10] > 0 2024-03-25T18:34:26.515 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:34:26.512+0000 7f12737fe640 3 client.4564.objecter handle_osd_map decoding full epoch 10 2024-03-25T18:34:26.515 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:34:26.512+0000 7f12737fe640 20 client.4564.objecter dump_active .. 0 homeless 2024-03-25T18:34:26.515 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:34:26.513+0000 7f12737fe640 1 -- 172.21.15.1:0/1699652248 <== mon.1 v1:172.21.15.115:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f1274050590 con 0x7f127c0b4e70 2024-03-25T18:34:26.515 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:34:26.513+0000 7f12737fe640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:34:26.515 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:34:26.513+0000 7f12737fe640 10 monclient: _finish_command 1 = system:0 2024-03-25T18:34:26.768 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:34:26.766+0000 7f128101e640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:34:26.768 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:34:26.766+0000 7f128101e640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:34:26.768 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:34:26.766+0000 7f128101e640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.115:6789/0 2024-03-25T18:34:26.768 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:34:26.766+0000 7f128101e640 1 -- 172.21.15.1:0/1699652248 --> v1:172.21.15.115:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f127c04f730 con 0x7f127c0b4e70 2024-03-25T18:34:26.932 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:34:26.930+0000 7f12737fe640 1 -- 172.21.15.1:0/1699652248 <== mon.1 v1:172.21.15.115:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 4092213348) 0x7f12740524c0 con 0x7f127c0b4e70 2024-03-25T18:34:26.932 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:34:26.930+0000 7f12737fe640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:34:26.932 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:34:26.930+0000 7f12737fe640 10 monclient: _finish_command 2 = system:0 2024-03-25T18:34:26.932 INFO:teuthology.misc.health.smithi001.stdout: 2024-03-25T18:34:26.932 INFO:teuthology.misc.health.smithi001.stdout:{"epoch":10,"fsid":"c8ea72cf-9e99-4383-9b52-6c23d365fdfc","created":"2024-03-25T18:26:38.856638+0000","modified":"2024-03-25T18:26:44.224208+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T18:26:44.224208+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"b9f34922-3a84-49be-8426-807c1eceaefc","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"42bf306f-55e3-4634-9874-8b4b73ac6644","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"292cb23a-18a9-4207-a7cc-4d5727feaec0","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"45e0e9ab-898d-4673-9590-b4117bc5e84a","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"831a5a09-73ab-4a61-9872-088e76bdbc66","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"88cd903d-3b55-4840-9396-6e3cfe005282","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"89db7355-bc9d-46f4-8ab0-e8d9e744e026","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"cac47bee-0914-4f08-896f-348838882866","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T18:34:26.934 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:34:26.932+0000 7f128101e640 10 librados: watch_flush enter 2024-03-25T18:34:26.934 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:34:26.932+0000 7f128101e640 10 librados: watch_flush exit 2024-03-25T18:34:26.934 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:34:26.932+0000 7f128101e640 20 client.4564.objecter shutdown clearing up homeless session... 2024-03-25T18:34:26.934 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:34:26.932+0000 7f128101e640 10 client.4564.objecter successfully canceled tick 2024-03-25T18:34:26.934 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:34:26.932+0000 7f128101e640 1 -- 172.21.15.1:0/1699652248 >> v1:172.21.15.115:6800/4237379176 conn(0x7f125c06de00 legacy=0x7f125c0702c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:34:26.935 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:34:26.932+0000 7f128101e640 10 monclient: shutdown 2024-03-25T18:34:26.935 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:34:26.932+0000 7f128101e640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:34:26.935 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:34:26.932+0000 7f128101e640 1 -- 172.21.15.1:0/1699652248 >> v1:172.21.15.115:6789/0 conn(0x7f127c0b4e70 legacy=0x7f127c133e80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:34:26.935 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:34:26.932+0000 7f128101e640 1 -- 172.21.15.1:0/1699652248 shutdown_connections 2024-03-25T18:34:26.935 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:34:26.932+0000 7f128101e640 1 -- 172.21.15.1:0/1699652248 >> 172.21.15.1:0/1699652248 conn(0x7f127c0617a0 msgr2=0x7f127c0b59e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:34:26.935 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:34:26.932+0000 7f128101e640 1 -- 172.21.15.1:0/1699652248 shutdown_connections 2024-03-25T18:34:26.935 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:34:26.932+0000 7f128101e640 1 -- 172.21.15.1:0/1699652248 wait complete. 2024-03-25T18:34:26.935 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:34:26.932+0000 7f128101e640 1 librados: shutdown 2024-03-25T18:34:26.946 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T18:34:33.753 DEBUG:teuthology.orchestra.run.smithi001:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T18:34:33.874 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:34:33.872+0000 7f249aa8d640 10 monclient: get_monmap_and_config 2024-03-25T18:34:33.874 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:34:33.872+0000 7f249aa8d640 10 monclient: build_initial_monmap 2024-03-25T18:34:33.874 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:34:33.872+0000 7f249aa8d640 10 monclient: monmap: 2024-03-25T18:34:33.874 INFO:teuthology.misc.health.smithi001.stderr:epoch 0 2024-03-25T18:34:33.874 INFO:teuthology.misc.health.smithi001.stderr:fsid c8ea72cf-9e99-4383-9b52-6c23d365fdfc 2024-03-25T18:34:33.874 INFO:teuthology.misc.health.smithi001.stderr:last_changed 2024-03-25T18:34:33.873895+0000 2024-03-25T18:34:33.874 INFO:teuthology.misc.health.smithi001.stderr:created 2024-03-25T18:34:33.873895+0000 2024-03-25T18:34:33.875 INFO:teuthology.misc.health.smithi001.stderr:min_mon_release 0 (unknown) 2024-03-25T18:34:33.875 INFO:teuthology.misc.health.smithi001.stderr:election_strategy: 3 2024-03-25T18:34:33.875 INFO:teuthology.misc.health.smithi001.stderr:0: v1:172.21.15.1:6789/0 mon.noname-a 2024-03-25T18:34:33.875 INFO:teuthology.misc.health.smithi001.stderr:1: v1:172.21.15.115:6789/0 mon.noname-c 2024-03-25T18:34:33.875 INFO:teuthology.misc.health.smithi001.stderr:2: v1:172.21.15.1:6790/0 mon.noname-b 2024-03-25T18:34:33.875 INFO:teuthology.misc.health.smithi001.stderr: 2024-03-25T18:34:33.876 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:34:33.873+0000 7f249aa8d640 1 Processor -- start 2024-03-25T18:34:33.877 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:34:33.873+0000 7f249aa8d640 1 -- start start 2024-03-25T18:34:33.877 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:34:33.873+0000 7f249aa8d640 10 monclient: init 2024-03-25T18:34:33.877 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:34:33.874+0000 7f249aa8d640 10 monclient: _reopen_session rank -1 2024-03-25T18:34:33.877 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:34:33.874+0000 7f249aa8d640 10 monclient: _add_conns ranks=[2,0,1] 2024-03-25T18:34:33.877 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:34:33.874+0000 7f249aa8d640 10 monclient(hunting): picked mon.noname-b con 0x7f24940b4ff0 addr v1:172.21.15.1:6790/0 2024-03-25T18:34:33.877 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:34:33.874+0000 7f249aa8d640 10 monclient(hunting): picked mon.noname-a con 0x7f24940b4670 addr v1:172.21.15.1:6789/0 2024-03-25T18:34:33.877 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:34:33.874+0000 7f249aa8d640 10 monclient(hunting): picked mon.noname-c con 0x7f249413c480 addr v1:172.21.15.115:6789/0 2024-03-25T18:34:33.877 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:34:33.874+0000 7f249aa8d640 1 -- --> v1:172.21.15.1:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f24940ab0e0 con 0x7f24940b4670 2024-03-25T18:34:33.877 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:34:33.874+0000 7f249aa8d640 1 -- --> v1:172.21.15.115:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f24940ab3c0 con 0x7f249413c480 2024-03-25T18:34:33.877 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:34:33.874+0000 7f249aa8d640 1 -- --> v1:172.21.15.1:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f24940a9f30 con 0x7f24940b4ff0 2024-03-25T18:34:33.877 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:34:33.874+0000 7f249aa8d640 10 monclient(hunting): _renew_subs 2024-03-25T18:34:33.877 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:34:33.874+0000 7f249aa8d640 10 monclient(hunting): authenticate will time out at 7954.344858s 2024-03-25T18:34:33.886 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:34:33.884+0000 7f249902a640 1 --1- >> v1:172.21.15.115:6789/0 conn(0x7f249413c480 0x7f249414dc00 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.115:6789/0 says I am v1:172.21.15.1:58618/0 (socket says 172.21.15.1:58618) 2024-03-25T18:34:33.886 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:34:33.884+0000 7f249902a640 1 -- 172.21.15.1:0/2488264035 learned_addr learned my addr 172.21.15.1:0/2488264035 (peer_addr_for_me v1:172.21.15.1:0/0) 2024-03-25T18:34:33.898 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:34:33.897+0000 7f24937fe640 1 -- 172.21.15.1:0/2488264035 <== mon.2 v1:172.21.15.1:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1554027067 0 0) 0x7f24940a9f30 con 0x7f24940b4ff0 2024-03-25T18:34:33.899 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:34:33.897+0000 7f24937fe640 10 monclient(hunting): _negotiate 2024-03-25T18:34:33.899 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:34:33.897+0000 7f24937fe640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:34:33.899 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:34:33.897+0000 7f24937fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:34:33.899 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:34:33.897+0000 7f24937fe640 10 monclient(hunting): my global_id is 4568 2024-03-25T18:34:33.899 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:34:33.897+0000 7f24937fe640 1 -- 172.21.15.1:0/2488264035 --> v1:172.21.15.1:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f2484003520 con 0x7f24940b4ff0 2024-03-25T18:34:33.899 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:34:33.898+0000 7f24937fe640 1 -- 172.21.15.1:0/2488264035 <== mon.1 v1:172.21.15.115:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 410382259 0 0) 0x7f24940ab3c0 con 0x7f249413c480 2024-03-25T18:34:33.900 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:34:33.898+0000 7f24937fe640 10 monclient(hunting): _negotiate 2024-03-25T18:34:33.900 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:34:33.898+0000 7f24937fe640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:34:33.900 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:34:33.898+0000 7f24937fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:34:33.900 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:34:33.898+0000 7f24937fe640 10 monclient(hunting): my global_id is 4567 2024-03-25T18:34:33.900 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:34:33.898+0000 7f24937fe640 1 -- 172.21.15.1:0/2488264035 --> v1:172.21.15.115:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f24940a9f30 con 0x7f249413c480 2024-03-25T18:34:33.907 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:34:33.906+0000 7f24937fe640 1 -- 172.21.15.1:0/2488264035 <== mon.0 v1:172.21.15.1:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1633593712 0 0) 0x7f24940ab0e0 con 0x7f24940b4670 2024-03-25T18:34:33.908 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:34:33.906+0000 7f24937fe640 10 monclient(hunting): _negotiate 2024-03-25T18:34:33.908 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:34:33.906+0000 7f24937fe640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:34:33.908 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:34:33.906+0000 7f24937fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:34:33.908 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:34:33.906+0000 7f24937fe640 10 monclient(hunting): my global_id is 4560 2024-03-25T18:34:33.908 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:34:33.906+0000 7f24937fe640 1 -- 172.21.15.1:0/2488264035 --> v1:172.21.15.1:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f24940ab3c0 con 0x7f24940b4670 2024-03-25T18:34:33.912 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:34:33.910+0000 7f24937fe640 1 -- 172.21.15.1:0/2488264035 <== mon.2 v1:172.21.15.1:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2299876775 0 0) 0x7f2484003520 con 0x7f24940b4ff0 2024-03-25T18:34:33.912 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:34:33.910+0000 7f24937fe640 1 -- 172.21.15.1:0/2488264035 >> v1:172.21.15.115:6789/0 conn(0x7f249413c480 legacy=0x7f249414dc00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:34:33.913 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:34:33.910+0000 7f24937fe640 1 -- 172.21.15.1:0/2488264035 >> v1:172.21.15.1:6789/0 conn(0x7f24940b4670 legacy=0x7f24941425c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:34:33.913 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:34:33.910+0000 7f24937fe640 10 monclient: _finish_hunting 0 2024-03-25T18:34:33.913 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:34:33.910+0000 7f24937fe640 1 monclient: found mon.noname-b 2024-03-25T18:34:33.913 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:34:33.910+0000 7f24937fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:34:33.913 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:34:33.910+0000 7f24937fe640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.1:6790/0 2024-03-25T18:34:33.913 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:34:33.910+0000 7f24937fe640 1 -- 172.21.15.1:0/2488264035 --> v1:172.21.15.1:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f24940a9b70 con 0x7f24940b4ff0 2024-03-25T18:34:33.913 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:34:33.910+0000 7f24937fe640 10 monclient: _finish_auth 0 2024-03-25T18:34:33.913 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:34:33.910+0000 7f24937fe640 10 monclient: _check_auth_tickets 2024-03-25T18:34:33.913 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:34:33.910+0000 7f24937fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:34:33.913 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:34:33.910+0000 7f24937fe640 1 -- 172.21.15.1:0/2488264035 <== mon.2 v1:172.21.15.1:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1586965025 0 0) 0x7f2488001690 con 0x7f24940b4ff0 2024-03-25T18:34:33.913 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:34:33.910+0000 7f249aa8d640 5 monclient: authenticate success, global_id 4568 2024-03-25T18:34:33.913 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:34:33.910+0000 7f249aa8d640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T18:34:33.914 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:34:33.910+0000 7f24937fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:34:33.914 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:34:33.910+0000 7f24937fe640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-03-25T18:34:33.914 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:34:33.910+0000 7f24937fe640 10 monclient: dump: 2024-03-25T18:34:33.914 INFO:teuthology.misc.health.smithi001.stderr:epoch 1 2024-03-25T18:34:33.914 INFO:teuthology.misc.health.smithi001.stderr:fsid c8ea72cf-9e99-4383-9b52-6c23d365fdfc 2024-03-25T18:34:33.914 INFO:teuthology.misc.health.smithi001.stderr:last_changed 2024-03-25T18:25:50.510519+0000 2024-03-25T18:34:33.914 INFO:teuthology.misc.health.smithi001.stderr:created 2024-03-25T18:25:50.510519+0000 2024-03-25T18:34:33.914 INFO:teuthology.misc.health.smithi001.stderr:min_mon_release 19 (squid) 2024-03-25T18:34:33.914 INFO:teuthology.misc.health.smithi001.stderr:election_strategy: 3 2024-03-25T18:34:33.914 INFO:teuthology.misc.health.smithi001.stderr:0: v1:172.21.15.1:6789/0 mon.a 2024-03-25T18:34:33.914 INFO:teuthology.misc.health.smithi001.stderr:1: v1:172.21.15.115:6789/0 mon.b 2024-03-25T18:34:33.914 INFO:teuthology.misc.health.smithi001.stderr:2: v1:172.21.15.1:6790/0 mon.c 2024-03-25T18:34:33.914 INFO:teuthology.misc.health.smithi001.stderr: 2024-03-25T18:34:33.914 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:34:33.910+0000 7f249aa8d640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T18:34:33.933 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:34:33.931+0000 7f24937fe640 1 -- 172.21.15.1:0/2488264035 <== mon.2 v1:172.21.15.1:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f24880017f0 con 0x7f24940b4ff0 2024-03-25T18:34:33.933 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:34:33.931+0000 7f24937fe640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:34:33.933 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:34:33.931+0000 7f24937fe640 1 -- 172.21.15.1:0/2488264035 <== mon.2 v1:172.21.15.1:6790/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1586965025 0 0) 0x7f2488001c00 con 0x7f24940b4ff0 2024-03-25T18:34:33.933 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:34:33.931+0000 7f24937fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:34:33.933 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:34:33.931+0000 7f24937fe640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-25T18:34:33.933 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:34:33.931+0000 7f24937fe640 10 monclient: dump: 2024-03-25T18:34:33.934 INFO:teuthology.misc.health.smithi001.stderr:epoch 1 2024-03-25T18:34:33.934 INFO:teuthology.misc.health.smithi001.stderr:fsid c8ea72cf-9e99-4383-9b52-6c23d365fdfc 2024-03-25T18:34:33.934 INFO:teuthology.misc.health.smithi001.stderr:last_changed 2024-03-25T18:25:50.510519+0000 2024-03-25T18:34:33.934 INFO:teuthology.misc.health.smithi001.stderr:created 2024-03-25T18:25:50.510519+0000 2024-03-25T18:34:33.934 INFO:teuthology.misc.health.smithi001.stderr:min_mon_release 19 (squid) 2024-03-25T18:34:33.934 INFO:teuthology.misc.health.smithi001.stderr:election_strategy: 3 2024-03-25T18:34:33.934 INFO:teuthology.misc.health.smithi001.stderr:0: v1:172.21.15.1:6789/0 mon.a 2024-03-25T18:34:33.934 INFO:teuthology.misc.health.smithi001.stderr:1: v1:172.21.15.115:6789/0 mon.b 2024-03-25T18:34:33.934 INFO:teuthology.misc.health.smithi001.stderr:2: v1:172.21.15.1:6790/0 mon.c 2024-03-25T18:34:33.934 INFO:teuthology.misc.health.smithi001.stderr: 2024-03-25T18:34:33.934 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:34:33.931+0000 7f249aa8d640 10 monclient: get_monmap_and_config success 2024-03-25T18:34:33.935 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:34:33.931+0000 7f249aa8d640 4 set_mon_vals no callback set 2024-03-25T18:34:33.937 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:34:33.932+0000 7f249aa8d640 10 monclient: shutdown 2024-03-25T18:34:33.937 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:34:33.932+0000 7f249aa8d640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:34:33.937 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:34:33.932+0000 7f249aa8d640 1 -- 172.21.15.1:0/2488264035 >> v1:172.21.15.1:6790/0 conn(0x7f24940b4ff0 legacy=0x7f249413bd70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:34:33.937 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:34:33.932+0000 7f249aa8d640 1 -- 172.21.15.1:0/2488264035 shutdown_connections 2024-03-25T18:34:33.937 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:34:33.932+0000 7f249aa8d640 1 -- 172.21.15.1:0/2488264035 >> 172.21.15.1:0/2488264035 conn(0x7f24940617a0 msgr2=0x7f2494061bb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:34:33.937 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:34:33.932+0000 7f249aa8d640 1 -- 172.21.15.1:0/2488264035 shutdown_connections 2024-03-25T18:34:33.937 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:34:33.932+0000 7f249aa8d640 1 -- 172.21.15.1:0/2488264035 wait complete. 2024-03-25T18:34:33.937 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:34:33.933+0000 7f249aa8d640 10 monclient: build_initial_monmap 2024-03-25T18:34:33.938 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:34:33.933+0000 7f249aa8d640 10 monclient: monmap: 2024-03-25T18:34:33.938 INFO:teuthology.misc.health.smithi001.stderr:epoch 0 2024-03-25T18:34:33.938 INFO:teuthology.misc.health.smithi001.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T18:34:33.938 INFO:teuthology.misc.health.smithi001.stderr:last_changed 0.000000 2024-03-25T18:34:33.938 INFO:teuthology.misc.health.smithi001.stderr:created 0.000000 2024-03-25T18:34:33.938 INFO:teuthology.misc.health.smithi001.stderr:min_mon_release 0 (unknown) 2024-03-25T18:34:33.938 INFO:teuthology.misc.health.smithi001.stderr:election_strategy: 1 2024-03-25T18:34:33.938 INFO:teuthology.misc.health.smithi001.stderr:0: v1:172.21.15.1:6789/0 mon.noname-a 2024-03-25T18:34:33.938 INFO:teuthology.misc.health.smithi001.stderr:1: v1:172.21.15.115:6789/0 mon.noname-b 2024-03-25T18:34:33.938 INFO:teuthology.misc.health.smithi001.stderr:2: v1:172.21.15.1:6790/0 mon.noname-c 2024-03-25T18:34:33.938 INFO:teuthology.misc.health.smithi001.stderr: 2024-03-25T18:34:33.938 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:34:33.933+0000 7f249aa8d640 1 librados: starting msgr at 2024-03-25T18:34:33.939 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:34:33.933+0000 7f249aa8d640 1 librados: starting objecter 2024-03-25T18:34:33.939 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:34:33.933+0000 7f249aa8d640 1 Processor -- start 2024-03-25T18:34:33.939 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:34:33.933+0000 7f249aa8d640 1 -- start start 2024-03-25T18:34:33.939 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:34:33.933+0000 7f249aa8d640 1 librados: setting wanted keys 2024-03-25T18:34:33.939 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:34:33.933+0000 7f249aa8d640 1 librados: calling monclient init 2024-03-25T18:34:33.939 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:34:33.933+0000 7f249aa8d640 10 monclient: init 2024-03-25T18:34:33.939 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:34:33.934+0000 7f249aa8d640 10 monclient: _reopen_session rank -1 2024-03-25T18:34:33.939 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:34:33.934+0000 7f249aa8d640 10 monclient: _add_conns ranks=[0,1,2] 2024-03-25T18:34:33.939 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:34:33.934+0000 7f249aa8d640 10 monclient(hunting): picked mon.noname-a con 0x7f24940b4670 addr v1:172.21.15.1:6789/0 2024-03-25T18:34:33.939 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:34:33.934+0000 7f249aa8d640 10 monclient(hunting): picked mon.noname-b con 0x7f24940b4ff0 addr v1:172.21.15.115:6789/0 2024-03-25T18:34:33.939 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:34:33.935+0000 7f249aa8d640 10 monclient(hunting): picked mon.noname-c con 0x7f249413c480 addr v1:172.21.15.1:6790/0 2024-03-25T18:34:33.939 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:34:33.935+0000 7f249aa8d640 1 -- --> v1:172.21.15.1:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f24940abe60 con 0x7f24940b4670 2024-03-25T18:34:33.939 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:34:33.935+0000 7f249aa8d640 1 -- --> v1:172.21.15.115:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f24940ac140 con 0x7f24940b4ff0 2024-03-25T18:34:33.939 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:34:33.935+0000 7f249aa8d640 1 -- --> v1:172.21.15.1:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f24940ac420 con 0x7f249413c480 2024-03-25T18:34:33.939 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:34:33.935+0000 7f249aa8d640 10 monclient(hunting): _renew_subs 2024-03-25T18:34:33.940 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:34:33.935+0000 7f249aa8d640 10 monclient(hunting): authenticate will time out at 7954.405863s 2024-03-25T18:34:33.940 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:34:33.938+0000 7f2498829640 1 --1- >> v1:172.21.15.1:6789/0 conn(0x7f24940b4670 0x7f249413b600 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.1:6789/0 says I am v1:172.21.15.1:37808/0 (socket says 172.21.15.1:37808) 2024-03-25T18:34:33.940 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:34:33.938+0000 7f2498829640 1 -- 172.21.15.1:0/1825505297 learned_addr learned my addr 172.21.15.1:0/1825505297 (peer_addr_for_me v1:172.21.15.1:0/0) 2024-03-25T18:34:33.941 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:34:33.939+0000 7f24917fa640 10 client.?.objecter ms_handle_connect 0x7f24940b4670 2024-03-25T18:34:33.941 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:34:33.939+0000 7f24917fa640 10 client.?.objecter resend_mon_ops 2024-03-25T18:34:33.951 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:34:33.949+0000 7f24917fa640 10 client.?.objecter ms_handle_connect 0x7f249413c480 2024-03-25T18:34:33.951 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:34:33.949+0000 7f24917fa640 10 client.?.objecter resend_mon_ops 2024-03-25T18:34:33.952 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:34:33.950+0000 7f24917fa640 1 -- 172.21.15.1:0/1825505297 <== mon.0 v1:172.21.15.1:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3743244810 0 0) 0x7f24940abe60 con 0x7f24940b4670 2024-03-25T18:34:33.952 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:34:33.950+0000 7f24917fa640 10 monclient(hunting): _negotiate 2024-03-25T18:34:33.952 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:34:33.950+0000 7f24917fa640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:34:33.952 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:34:33.950+0000 7f24917fa640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:34:33.952 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:34:33.950+0000 7f24917fa640 10 monclient(hunting): my global_id is 4563 2024-03-25T18:34:33.952 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:34:33.950+0000 7f24917fa640 1 -- 172.21.15.1:0/1825505297 --> v1:172.21.15.1:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f2478004eb0 con 0x7f24940b4670 2024-03-25T18:34:33.952 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:34:33.951+0000 7f24917fa640 10 client.?.objecter ms_handle_connect 0x7f24940b4ff0 2024-03-25T18:34:33.953 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:34:33.951+0000 7f24917fa640 10 client.?.objecter resend_mon_ops 2024-03-25T18:34:33.957 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:34:33.955+0000 7f24917fa640 1 -- 172.21.15.1:0/1825505297 <== mon.2 v1:172.21.15.1:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3344182589 0 0) 0x7f24940ac420 con 0x7f249413c480 2024-03-25T18:34:33.957 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:34:33.955+0000 7f24917fa640 10 monclient(hunting): _negotiate 2024-03-25T18:34:33.957 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:34:33.955+0000 7f24917fa640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:34:33.957 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:34:33.955+0000 7f24917fa640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:34:33.957 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:34:33.955+0000 7f24917fa640 10 monclient(hunting): my global_id is 4571 2024-03-25T18:34:33.957 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:34:33.955+0000 7f24917fa640 1 -- 172.21.15.1:0/1825505297 --> v1:172.21.15.1:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f24940abe60 con 0x7f249413c480 2024-03-25T18:34:33.959 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:34:33.958+0000 7f24917fa640 1 -- 172.21.15.1:0/1825505297 <== mon.1 v1:172.21.15.115:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3669941039 0 0) 0x7f24940ac140 con 0x7f24940b4ff0 2024-03-25T18:34:33.960 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:34:33.958+0000 7f24917fa640 10 monclient(hunting): _negotiate 2024-03-25T18:34:33.960 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:34:33.958+0000 7f24917fa640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:34:33.960 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:34:33.958+0000 7f24917fa640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:34:33.960 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:34:33.958+0000 7f24917fa640 10 monclient(hunting): my global_id is 4570 2024-03-25T18:34:33.960 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:34:33.958+0000 7f24917fa640 1 -- 172.21.15.1:0/1825505297 --> v1:172.21.15.115:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f24940ac420 con 0x7f24940b4ff0 2024-03-25T18:34:33.968 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:34:33.966+0000 7f24917fa640 1 -- 172.21.15.1:0/1825505297 <== mon.2 v1:172.21.15.1:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3713948599 0 0) 0x7f24940abe60 con 0x7f249413c480 2024-03-25T18:34:33.968 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:34:33.966+0000 7f24917fa640 1 -- 172.21.15.1:0/1825505297 --> v1:172.21.15.1:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f24940ac140 con 0x7f249413c480 2024-03-25T18:34:33.968 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:34:33.966+0000 7f24917fa640 1 -- 172.21.15.1:0/1825505297 <== mon.2 v1:172.21.15.1:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1586965025 0 0) 0x7f248c001b50 con 0x7f249413c480 2024-03-25T18:34:33.968 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:34:33.966+0000 7f24917fa640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:34:33.969 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:34:33.966+0000 7f24917fa640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-03-25T18:34:33.969 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:34:33.966+0000 7f24917fa640 10 monclient(hunting): dump: 2024-03-25T18:34:33.969 INFO:teuthology.misc.health.smithi001.stderr:epoch 1 2024-03-25T18:34:33.969 INFO:teuthology.misc.health.smithi001.stderr:fsid c8ea72cf-9e99-4383-9b52-6c23d365fdfc 2024-03-25T18:34:33.969 INFO:teuthology.misc.health.smithi001.stderr:last_changed 2024-03-25T18:25:50.510519+0000 2024-03-25T18:34:33.969 INFO:teuthology.misc.health.smithi001.stderr:created 2024-03-25T18:25:50.510519+0000 2024-03-25T18:34:33.969 INFO:teuthology.misc.health.smithi001.stderr:min_mon_release 19 (squid) 2024-03-25T18:34:33.969 INFO:teuthology.misc.health.smithi001.stderr:election_strategy: 3 2024-03-25T18:34:33.969 INFO:teuthology.misc.health.smithi001.stderr:0: v1:172.21.15.1:6789/0 mon.a 2024-03-25T18:34:33.969 INFO:teuthology.misc.health.smithi001.stderr:1: v1:172.21.15.115:6789/0 mon.b 2024-03-25T18:34:33.969 INFO:teuthology.misc.health.smithi001.stderr:2: v1:172.21.15.1:6790/0 mon.c 2024-03-25T18:34:33.969 INFO:teuthology.misc.health.smithi001.stderr: 2024-03-25T18:34:33.969 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:34:33.966+0000 7f24917fa640 10 monclient(hunting): _finish_auth 0 2024-03-25T18:34:33.970 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:34:33.966+0000 7f24917fa640 1 -- 172.21.15.1:0/1825505297 <== mon.0 v1:172.21.15.1:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 413624399 0 0) 0x7f2478004eb0 con 0x7f24940b4670 2024-03-25T18:34:33.970 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:34:33.966+0000 7f24917fa640 1 -- 172.21.15.1:0/1825505297 --> v1:172.21.15.1:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f24940abe60 con 0x7f24940b4670 2024-03-25T18:34:33.970 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:34:33.966+0000 7f24917fa640 1 -- 172.21.15.1:0/1825505297 <== mon.0 v1:172.21.15.1:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1586965025 0 0) 0x7f2488001bc0 con 0x7f24940b4670 2024-03-25T18:34:33.970 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:34:33.966+0000 7f24917fa640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:34:33.970 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:34:33.966+0000 7f24917fa640 10 monclient(hunting): got monmap 1 from mon.a (according to old e1) 2024-03-25T18:34:33.970 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:34:33.966+0000 7f24917fa640 10 monclient(hunting): dump: 2024-03-25T18:34:33.970 INFO:teuthology.misc.health.smithi001.stderr:epoch 1 2024-03-25T18:34:33.970 INFO:teuthology.misc.health.smithi001.stderr:fsid c8ea72cf-9e99-4383-9b52-6c23d365fdfc 2024-03-25T18:34:33.970 INFO:teuthology.misc.health.smithi001.stderr:last_changed 2024-03-25T18:25:50.510519+0000 2024-03-25T18:34:33.970 INFO:teuthology.misc.health.smithi001.stderr:created 2024-03-25T18:25:50.510519+0000 2024-03-25T18:34:33.970 INFO:teuthology.misc.health.smithi001.stderr:min_mon_release 19 (squid) 2024-03-25T18:34:33.970 INFO:teuthology.misc.health.smithi001.stderr:election_strategy: 3 2024-03-25T18:34:33.971 INFO:teuthology.misc.health.smithi001.stderr:0: v1:172.21.15.1:6789/0 mon.a 2024-03-25T18:34:33.971 INFO:teuthology.misc.health.smithi001.stderr:1: v1:172.21.15.115:6789/0 mon.b 2024-03-25T18:34:33.971 INFO:teuthology.misc.health.smithi001.stderr:2: v1:172.21.15.1:6790/0 mon.c 2024-03-25T18:34:33.971 INFO:teuthology.misc.health.smithi001.stderr: 2024-03-25T18:34:33.977 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:34:33.975+0000 7f24917fa640 1 -- 172.21.15.1:0/1825505297 <== mon.2 v1:172.21.15.1:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 3782477281 0 0) 0x7f24940ac140 con 0x7f249413c480 2024-03-25T18:34:33.977 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:34:33.975+0000 7f24917fa640 1 -- 172.21.15.1:0/1825505297 >> v1:172.21.15.115:6789/0 conn(0x7f24940b4ff0 legacy=0x7f24941381a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:34:33.977 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:34:33.975+0000 7f24917fa640 1 -- 172.21.15.1:0/1825505297 >> v1:172.21.15.1:6789/0 conn(0x7f24940b4670 legacy=0x7f249413b600 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:34:33.978 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:34:33.975+0000 7f24917fa640 10 monclient: _finish_hunting 0 2024-03-25T18:34:33.978 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:34:33.975+0000 7f24917fa640 1 monclient: found mon.c 2024-03-25T18:34:33.978 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:34:33.975+0000 7f24917fa640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:34:33.978 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:34:33.975+0000 7f24917fa640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.1:6790/0 2024-03-25T18:34:33.978 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:34:33.975+0000 7f24917fa640 1 -- 172.21.15.1:0/1825505297 --> v1:172.21.15.1:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f249413a8f0 con 0x7f249413c480 2024-03-25T18:34:33.978 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:34:33.975+0000 7f24917fa640 10 monclient: _finish_auth 0 2024-03-25T18:34:33.978 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:34:33.975+0000 7f24917fa640 10 monclient: _check_auth_tickets 2024-03-25T18:34:33.978 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:34:33.975+0000 7f24917fa640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:34:33.978 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:34:33.975+0000 7f249aa8d640 5 monclient: authenticate success, global_id 4571 2024-03-25T18:34:33.978 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:34:33.975+0000 7f249aa8d640 10 monclient: _renew_subs 2024-03-25T18:34:33.978 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:34:33.975+0000 7f249aa8d640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.1:6790/0 2024-03-25T18:34:33.978 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:34:33.975+0000 7f249aa8d640 1 -- 172.21.15.1:0/1825505297 --> v1:172.21.15.1:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f24940ac840 con 0x7f249413c480 2024-03-25T18:34:33.978 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:34:33.975+0000 7f249aa8d640 10 client.4571.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T18:34:33.978 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:34:33.975+0000 7f249aa8d640 10 monclient: _renew_subs 2024-03-25T18:34:33.979 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:34:33.975+0000 7f249aa8d640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.1:6790/0 2024-03-25T18:34:33.979 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:34:33.975+0000 7f249aa8d640 1 -- 172.21.15.1:0/1825505297 --> v1:172.21.15.1:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f24940acd20 con 0x7f249413c480 2024-03-25T18:34:33.979 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:34:33.975+0000 7f249aa8d640 1 librados: init done 2024-03-25T18:34:33.979 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:34:33.976+0000 7f249aa8d640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T18:34:33.979 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:34:33.976+0000 7f249aa8d640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:34:33.979 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:34:33.976+0000 7f249aa8d640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.1:6790/0 2024-03-25T18:34:33.980 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:34:33.976+0000 7f249aa8d640 1 -- 172.21.15.1:0/1825505297 --> v1:172.21.15.1:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f249406dbd0 con 0x7f249413c480 2024-03-25T18:34:34.283 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:34:34.281+0000 7f24917fa640 1 -- 172.21.15.1:0/1825505297 <== mon.2 v1:172.21.15.1:6790/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f248c002060 con 0x7f249413c480 2024-03-25T18:34:34.284 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:34:34.281+0000 7f24917fa640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:34:34.284 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:34:34.281+0000 7f24917fa640 1 -- 172.21.15.1:0/1825505297 <== mon.2 v1:172.21.15.1:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1586965025 0 0) 0x7f248c004d00 con 0x7f249413c480 2024-03-25T18:34:34.284 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:34:34.281+0000 7f24917fa640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:34:34.284 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:34:34.281+0000 7f24917fa640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-25T18:34:34.284 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:34:34.281+0000 7f24917fa640 10 monclient: dump: 2024-03-25T18:34:34.284 INFO:teuthology.misc.health.smithi001.stderr:epoch 1 2024-03-25T18:34:34.284 INFO:teuthology.misc.health.smithi001.stderr:fsid c8ea72cf-9e99-4383-9b52-6c23d365fdfc 2024-03-25T18:34:34.284 INFO:teuthology.misc.health.smithi001.stderr:last_changed 2024-03-25T18:25:50.510519+0000 2024-03-25T18:34:34.285 INFO:teuthology.misc.health.smithi001.stderr:created 2024-03-25T18:25:50.510519+0000 2024-03-25T18:34:34.285 INFO:teuthology.misc.health.smithi001.stderr:min_mon_release 19 (squid) 2024-03-25T18:34:34.285 INFO:teuthology.misc.health.smithi001.stderr:election_strategy: 3 2024-03-25T18:34:34.285 INFO:teuthology.misc.health.smithi001.stderr:0: v1:172.21.15.1:6789/0 mon.a 2024-03-25T18:34:34.285 INFO:teuthology.misc.health.smithi001.stderr:1: v1:172.21.15.115:6789/0 mon.b 2024-03-25T18:34:34.285 INFO:teuthology.misc.health.smithi001.stderr:2: v1:172.21.15.1:6790/0 mon.c 2024-03-25T18:34:34.285 INFO:teuthology.misc.health.smithi001.stderr: 2024-03-25T18:34:34.285 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:34:34.281+0000 7f2492ffd640 4 set_mon_vals no callback set 2024-03-25T18:34:34.286 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:34:34.282+0000 7f24917fa640 1 -- 172.21.15.1:0/1825505297 <== mon.2 v1:172.21.15.1:6790/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 21787975 0 0) 0x7f248c01ad10 con 0x7f249413c480 2024-03-25T18:34:34.286 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:34:34.283+0000 7f2498829640 10 monclient: get_auth_request con 0x7f247806de80 auth_method 0 2024-03-25T18:34:34.287 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:34:34.284+0000 7f24917fa640 10 client.4571.objecter ms_handle_connect 0x7f247806de80 2024-03-25T18:34:34.292 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:34:34.290+0000 7f24917fa640 1 -- 172.21.15.1:0/1825505297 <== mon.2 v1:172.21.15.1:6790/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 996254230 0 0) 0x7f248c054a60 con 0x7f249413c480 2024-03-25T18:34:34.292 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:34:34.290+0000 7f24917fa640 10 client.4571.objecter ms_dispatch 0x7f2494000b90 osd_map(10..10 src has 1..10) v4 2024-03-25T18:34:34.292 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:34:34.290+0000 7f24917fa640 3 client.4571.objecter handle_osd_map got epochs [10,10] > 0 2024-03-25T18:34:34.292 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:34:34.290+0000 7f24917fa640 3 client.4571.objecter handle_osd_map decoding full epoch 10 2024-03-25T18:34:34.293 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:34:34.290+0000 7f24917fa640 20 client.4571.objecter dump_active .. 0 homeless 2024-03-25T18:34:34.293 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:34:34.290+0000 7f24917fa640 1 -- 172.21.15.1:0/1825505297 <== mon.2 v1:172.21.15.1:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f248c0503a0 con 0x7f249413c480 2024-03-25T18:34:34.293 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:34:34.290+0000 7f24917fa640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:34:34.293 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:34:34.290+0000 7f24917fa640 10 monclient: _finish_command 1 = system:0 2024-03-25T18:34:34.557 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:34:34.555+0000 7f249aa8d640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:34:34.557 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:34:34.555+0000 7f249aa8d640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:34:34.557 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:34:34.555+0000 7f249aa8d640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.1:6790/0 2024-03-25T18:34:34.558 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:34:34.555+0000 7f249aa8d640 1 -- 172.21.15.1:0/1825505297 --> v1:172.21.15.1:6790/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f249414f100 con 0x7f249413c480 2024-03-25T18:34:34.582 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:34:34.580+0000 7f24917fa640 1 -- 172.21.15.1:0/1825505297 <== mon.2 v1:172.21.15.1:6790/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 4092213348) 0x7f248c0524a0 con 0x7f249413c480 2024-03-25T18:34:34.583 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:34:34.580+0000 7f24917fa640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:34:34.583 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:34:34.580+0000 7f24917fa640 10 monclient: _finish_command 2 = system:0 2024-03-25T18:34:34.584 INFO:teuthology.misc.health.smithi001.stdout: 2024-03-25T18:34:34.584 INFO:teuthology.misc.health.smithi001.stdout:{"epoch":10,"fsid":"c8ea72cf-9e99-4383-9b52-6c23d365fdfc","created":"2024-03-25T18:26:38.856638+0000","modified":"2024-03-25T18:26:44.224208+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T18:26:44.224208+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"b9f34922-3a84-49be-8426-807c1eceaefc","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"42bf306f-55e3-4634-9874-8b4b73ac6644","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"292cb23a-18a9-4207-a7cc-4d5727feaec0","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"45e0e9ab-898d-4673-9590-b4117bc5e84a","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"831a5a09-73ab-4a61-9872-088e76bdbc66","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"88cd903d-3b55-4840-9396-6e3cfe005282","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"89db7355-bc9d-46f4-8ab0-e8d9e744e026","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"cac47bee-0914-4f08-896f-348838882866","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T18:34:34.586 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:34:34.583+0000 7f249aa8d640 10 librados: watch_flush enter 2024-03-25T18:34:34.586 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:34:34.583+0000 7f249aa8d640 10 librados: watch_flush exit 2024-03-25T18:34:34.586 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:34:34.583+0000 7f249aa8d640 20 client.4571.objecter shutdown clearing up homeless session... 2024-03-25T18:34:34.586 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:34:34.583+0000 7f249aa8d640 10 client.4571.objecter successfully canceled tick 2024-03-25T18:34:34.586 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:34:34.583+0000 7f249aa8d640 1 -- 172.21.15.1:0/1825505297 >> v1:172.21.15.115:6800/4237379176 conn(0x7f247806de80 legacy=0x7f2478070340 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:34:34.586 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:34:34.583+0000 7f249aa8d640 10 monclient: shutdown 2024-03-25T18:34:34.586 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:34:34.583+0000 7f249aa8d640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:34:34.586 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:34:34.583+0000 7f249aa8d640 1 -- 172.21.15.1:0/1825505297 >> v1:172.21.15.1:6790/0 conn(0x7f249413c480 legacy=0x7f24941e2380 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:34:34.587 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:34:34.583+0000 7f249aa8d640 1 -- 172.21.15.1:0/1825505297 shutdown_connections 2024-03-25T18:34:34.587 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:34:34.583+0000 7f249aa8d640 1 -- 172.21.15.1:0/1825505297 >> 172.21.15.1:0/1825505297 conn(0x7f24940617a0 msgr2=0x7f2494140760 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:34:34.587 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:34:34.583+0000 7f249aa8d640 1 -- 172.21.15.1:0/1825505297 shutdown_connections 2024-03-25T18:34:34.587 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:34:34.583+0000 7f249aa8d640 1 -- 172.21.15.1:0/1825505297 wait complete. 2024-03-25T18:34:34.587 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:34:34.583+0000 7f249aa8d640 1 librados: shutdown 2024-03-25T18:34:34.599 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T18:34:41.405 DEBUG:teuthology.orchestra.run.smithi001:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T18:34:41.523 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:34:41.522+0000 7f9e8d20f640 10 monclient: get_monmap_and_config 2024-03-25T18:34:41.523 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:34:41.522+0000 7f9e8d20f640 10 monclient: build_initial_monmap 2024-03-25T18:34:41.523 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:34:41.522+0000 7f9e8d20f640 10 monclient: monmap: 2024-03-25T18:34:41.523 INFO:teuthology.misc.health.smithi001.stderr:epoch 0 2024-03-25T18:34:41.523 INFO:teuthology.misc.health.smithi001.stderr:fsid c8ea72cf-9e99-4383-9b52-6c23d365fdfc 2024-03-25T18:34:41.523 INFO:teuthology.misc.health.smithi001.stderr:last_changed 2024-03-25T18:34:41.523132+0000 2024-03-25T18:34:41.524 INFO:teuthology.misc.health.smithi001.stderr:created 2024-03-25T18:34:41.523132+0000 2024-03-25T18:34:41.524 INFO:teuthology.misc.health.smithi001.stderr:min_mon_release 0 (unknown) 2024-03-25T18:34:41.524 INFO:teuthology.misc.health.smithi001.stderr:election_strategy: 3 2024-03-25T18:34:41.524 INFO:teuthology.misc.health.smithi001.stderr:0: v1:172.21.15.1:6789/0 mon.noname-a 2024-03-25T18:34:41.524 INFO:teuthology.misc.health.smithi001.stderr:1: v1:172.21.15.115:6789/0 mon.noname-c 2024-03-25T18:34:41.524 INFO:teuthology.misc.health.smithi001.stderr:2: v1:172.21.15.1:6790/0 mon.noname-b 2024-03-25T18:34:41.524 INFO:teuthology.misc.health.smithi001.stderr: 2024-03-25T18:34:41.525 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:34:41.523+0000 7f9e8d20f640 1 Processor -- start 2024-03-25T18:34:41.525 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:34:41.523+0000 7f9e8d20f640 1 -- start start 2024-03-25T18:34:41.525 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:34:41.523+0000 7f9e8d20f640 10 monclient: init 2024-03-25T18:34:41.526 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:34:41.523+0000 7f9e8d20f640 10 monclient: _reopen_session rank -1 2024-03-25T18:34:41.526 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:34:41.523+0000 7f9e8d20f640 10 monclient: _add_conns ranks=[0,2,1] 2024-03-25T18:34:41.526 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:34:41.523+0000 7f9e8d20f640 10 monclient(hunting): picked mon.noname-a con 0x7f9e880b4e70 addr v1:172.21.15.1:6789/0 2024-03-25T18:34:41.526 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:34:41.523+0000 7f9e8d20f640 10 monclient(hunting): picked mon.noname-b con 0x7f9e880b44b0 addr v1:172.21.15.1:6790/0 2024-03-25T18:34:41.526 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:34:41.523+0000 7f9e8d20f640 10 monclient(hunting): picked mon.noname-c con 0x7f9e880b1b00 addr v1:172.21.15.115:6789/0 2024-03-25T18:34:41.526 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:34:41.523+0000 7f9e8d20f640 1 -- --> v1:172.21.15.1:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9e880a96f0 con 0x7f9e880b4e70 2024-03-25T18:34:41.526 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:34:41.523+0000 7f9e8d20f640 1 -- --> v1:172.21.15.115:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9e880ab0e0 con 0x7f9e880b1b00 2024-03-25T18:34:41.526 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:34:41.523+0000 7f9e8d20f640 1 -- --> v1:172.21.15.1:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9e88145f70 con 0x7f9e880b44b0 2024-03-25T18:34:41.527 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:34:41.523+0000 7f9e8d20f640 10 monclient(hunting): _renew_subs 2024-03-25T18:34:41.527 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:34:41.523+0000 7f9e8d20f640 10 monclient(hunting): authenticate will time out at 7961.993496s 2024-03-25T18:34:41.539 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:34:41.538+0000 7f9e8759e640 1 --1- >> v1:172.21.15.115:6789/0 conn(0x7f9e880b1b00 0x7f9e8814dc00 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.115:6789/0 says I am v1:172.21.15.1:35116/0 (socket says 172.21.15.1:35116) 2024-03-25T18:34:41.539 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:34:41.538+0000 7f9e8759e640 1 -- 172.21.15.1:0/1532578125 learned_addr learned my addr 172.21.15.1:0/1532578125 (peer_addr_for_me v1:172.21.15.1:0/0) 2024-03-25T18:34:41.558 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:34:41.558+0000 7f9e8659c640 1 -- 172.21.15.1:0/1532578125 <== mon.0 v1:172.21.15.1:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2422201230 0 0) 0x7f9e880a96f0 con 0x7f9e880b4e70 2024-03-25T18:34:41.559 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:34:41.558+0000 7f9e8659c640 10 monclient(hunting): _negotiate 2024-03-25T18:34:41.559 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:34:41.558+0000 7f9e8659c640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:34:41.559 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:34:41.558+0000 7f9e8659c640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:34:41.559 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:34:41.558+0000 7f9e8659c640 10 monclient(hunting): my global_id is 4566 2024-03-25T18:34:41.559 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:34:41.558+0000 7f9e8659c640 1 -- 172.21.15.1:0/1532578125 --> v1:172.21.15.1:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f9e74003520 con 0x7f9e880b4e70 2024-03-25T18:34:41.562 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:34:41.560+0000 7f9e8659c640 1 -- 172.21.15.1:0/1532578125 <== mon.2 v1:172.21.15.1:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 435170580 0 0) 0x7f9e88145f70 con 0x7f9e880b44b0 2024-03-25T18:34:41.562 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:34:41.560+0000 7f9e8659c640 10 monclient(hunting): _negotiate 2024-03-25T18:34:41.562 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:34:41.560+0000 7f9e8659c640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:34:41.562 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:34:41.560+0000 7f9e8659c640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:34:41.562 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:34:41.560+0000 7f9e8659c640 10 monclient(hunting): my global_id is 4574 2024-03-25T18:34:41.562 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:34:41.560+0000 7f9e8659c640 1 -- 172.21.15.1:0/1532578125 --> v1:172.21.15.1:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f9e880a96f0 con 0x7f9e880b44b0 2024-03-25T18:34:41.562 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:34:41.561+0000 7f9e8659c640 1 -- 172.21.15.1:0/1532578125 <== mon.1 v1:172.21.15.115:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 875848800 0 0) 0x7f9e880ab0e0 con 0x7f9e880b1b00 2024-03-25T18:34:41.562 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:34:41.561+0000 7f9e8659c640 10 monclient(hunting): _negotiate 2024-03-25T18:34:41.562 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:34:41.561+0000 7f9e8659c640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:34:41.562 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:34:41.561+0000 7f9e8659c640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:34:41.562 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:34:41.561+0000 7f9e8659c640 10 monclient(hunting): my global_id is 4573 2024-03-25T18:34:41.562 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:34:41.561+0000 7f9e8659c640 1 -- 172.21.15.1:0/1532578125 --> v1:172.21.15.115:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f9e88145f70 con 0x7f9e880b1b00 2024-03-25T18:34:41.571 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:34:41.570+0000 7f9e8659c640 1 -- 172.21.15.1:0/1532578125 <== mon.2 v1:172.21.15.1:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2568965733 0 0) 0x7f9e880a96f0 con 0x7f9e880b44b0 2024-03-25T18:34:41.571 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:34:41.570+0000 7f9e8659c640 1 -- 172.21.15.1:0/1532578125 >> v1:172.21.15.115:6789/0 conn(0x7f9e880b1b00 legacy=0x7f9e8814dc00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:34:41.571 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:34:41.570+0000 7f9e8659c640 1 -- 172.21.15.1:0/1532578125 >> v1:172.21.15.1:6789/0 conn(0x7f9e880b4e70 legacy=0x7f9e8813f940 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:34:41.571 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:34:41.570+0000 7f9e8659c640 10 monclient: _finish_hunting 0 2024-03-25T18:34:41.571 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:34:41.570+0000 7f9e8659c640 1 monclient: found mon.noname-b 2024-03-25T18:34:41.571 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:34:41.570+0000 7f9e8659c640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:34:41.571 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:34:41.570+0000 7f9e8659c640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.1:6790/0 2024-03-25T18:34:41.571 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:34:41.570+0000 7f9e8659c640 1 -- 172.21.15.1:0/1532578125 --> v1:172.21.15.1:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9e880a9b80 con 0x7f9e880b44b0 2024-03-25T18:34:41.571 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:34:41.570+0000 7f9e8659c640 10 monclient: _finish_auth 0 2024-03-25T18:34:41.571 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:34:41.570+0000 7f9e8659c640 10 monclient: _check_auth_tickets 2024-03-25T18:34:41.572 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:34:41.570+0000 7f9e8659c640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:34:41.572 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:34:41.570+0000 7f9e8659c640 1 -- 172.21.15.1:0/1532578125 <== mon.2 v1:172.21.15.1:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1586965025 0 0) 0x7f9e70001670 con 0x7f9e880b44b0 2024-03-25T18:34:41.572 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:34:41.570+0000 7f9e8d20f640 5 monclient: authenticate success, global_id 4574 2024-03-25T18:34:41.572 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:34:41.570+0000 7f9e8d20f640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T18:34:41.572 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:34:41.570+0000 7f9e8659c640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:34:41.572 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:34:41.570+0000 7f9e8659c640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-03-25T18:34:41.572 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:34:41.570+0000 7f9e8659c640 10 monclient: dump: 2024-03-25T18:34:41.572 INFO:teuthology.misc.health.smithi001.stderr:epoch 1 2024-03-25T18:34:41.572 INFO:teuthology.misc.health.smithi001.stderr:fsid c8ea72cf-9e99-4383-9b52-6c23d365fdfc 2024-03-25T18:34:41.572 INFO:teuthology.misc.health.smithi001.stderr:last_changed 2024-03-25T18:25:50.510519+0000 2024-03-25T18:34:41.572 INFO:teuthology.misc.health.smithi001.stderr:created 2024-03-25T18:25:50.510519+0000 2024-03-25T18:34:41.572 INFO:teuthology.misc.health.smithi001.stderr:min_mon_release 19 (squid) 2024-03-25T18:34:41.572 INFO:teuthology.misc.health.smithi001.stderr:election_strategy: 3 2024-03-25T18:34:41.572 INFO:teuthology.misc.health.smithi001.stderr:0: v1:172.21.15.1:6789/0 mon.a 2024-03-25T18:34:41.573 INFO:teuthology.misc.health.smithi001.stderr:1: v1:172.21.15.115:6789/0 mon.b 2024-03-25T18:34:41.573 INFO:teuthology.misc.health.smithi001.stderr:2: v1:172.21.15.1:6790/0 mon.c 2024-03-25T18:34:41.573 INFO:teuthology.misc.health.smithi001.stderr: 2024-03-25T18:34:41.573 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:34:41.570+0000 7f9e8d20f640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T18:34:41.617 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:34:41.616+0000 7f9e8659c640 1 -- 172.21.15.1:0/1532578125 <== mon.2 v1:172.21.15.1:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f9e700017d0 con 0x7f9e880b44b0 2024-03-25T18:34:41.617 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:34:41.616+0000 7f9e8659c640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:34:41.617 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:34:41.616+0000 7f9e8659c640 1 -- 172.21.15.1:0/1532578125 <== mon.2 v1:172.21.15.1:6790/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1586965025 0 0) 0x7f9e70001be0 con 0x7f9e880b44b0 2024-03-25T18:34:41.617 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:34:41.616+0000 7f9e8d20f640 10 monclient: get_monmap_and_config success 2024-03-25T18:34:41.617 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:34:41.616+0000 7f9e8d20f640 4 set_mon_vals no callback set 2024-03-25T18:34:41.617 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:34:41.616+0000 7f9e8659c640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:34:41.617 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:34:41.616+0000 7f9e8659c640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-25T18:34:41.617 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:34:41.616+0000 7f9e8659c640 10 monclient: dump: 2024-03-25T18:34:41.617 INFO:teuthology.misc.health.smithi001.stderr:epoch 1 2024-03-25T18:34:41.618 INFO:teuthology.misc.health.smithi001.stderr:fsid c8ea72cf-9e99-4383-9b52-6c23d365fdfc 2024-03-25T18:34:41.618 INFO:teuthology.misc.health.smithi001.stderr:last_changed 2024-03-25T18:25:50.510519+0000 2024-03-25T18:34:41.618 INFO:teuthology.misc.health.smithi001.stderr:created 2024-03-25T18:25:50.510519+0000 2024-03-25T18:34:41.618 INFO:teuthology.misc.health.smithi001.stderr:min_mon_release 19 (squid) 2024-03-25T18:34:41.618 INFO:teuthology.misc.health.smithi001.stderr:election_strategy: 3 2024-03-25T18:34:41.618 INFO:teuthology.misc.health.smithi001.stderr:0: v1:172.21.15.1:6789/0 mon.a 2024-03-25T18:34:41.618 INFO:teuthology.misc.health.smithi001.stderr:1: v1:172.21.15.115:6789/0 mon.b 2024-03-25T18:34:41.618 INFO:teuthology.misc.health.smithi001.stderr:2: v1:172.21.15.1:6790/0 mon.c 2024-03-25T18:34:41.618 INFO:teuthology.misc.health.smithi001.stderr: 2024-03-25T18:34:41.620 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:34:41.617+0000 7f9e8d20f640 10 monclient: shutdown 2024-03-25T18:34:41.620 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:34:41.617+0000 7f9e8d20f640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:34:41.620 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:34:41.617+0000 7f9e8d20f640 1 -- 172.21.15.1:0/1532578125 >> v1:172.21.15.1:6790/0 conn(0x7f9e880b44b0 legacy=0x7f9e88145860 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:34:41.620 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:34:41.617+0000 7f9e8d20f640 1 -- 172.21.15.1:0/1532578125 shutdown_connections 2024-03-25T18:34:41.621 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:34:41.617+0000 7f9e8d20f640 1 -- 172.21.15.1:0/1532578125 >> 172.21.15.1:0/1532578125 conn(0x7f9e880617a0 msgr2=0x7f9e88061bb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:34:41.621 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:34:41.617+0000 7f9e8d20f640 1 -- 172.21.15.1:0/1532578125 shutdown_connections 2024-03-25T18:34:41.621 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:34:41.617+0000 7f9e8d20f640 1 -- 172.21.15.1:0/1532578125 wait complete. 2024-03-25T18:34:41.621 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:34:41.618+0000 7f9e8d20f640 10 monclient: build_initial_monmap 2024-03-25T18:34:41.621 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:34:41.618+0000 7f9e8d20f640 10 monclient: monmap: 2024-03-25T18:34:41.621 INFO:teuthology.misc.health.smithi001.stderr:epoch 0 2024-03-25T18:34:41.621 INFO:teuthology.misc.health.smithi001.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T18:34:41.621 INFO:teuthology.misc.health.smithi001.stderr:last_changed 0.000000 2024-03-25T18:34:41.621 INFO:teuthology.misc.health.smithi001.stderr:created 0.000000 2024-03-25T18:34:41.621 INFO:teuthology.misc.health.smithi001.stderr:min_mon_release 0 (unknown) 2024-03-25T18:34:41.621 INFO:teuthology.misc.health.smithi001.stderr:election_strategy: 1 2024-03-25T18:34:41.621 INFO:teuthology.misc.health.smithi001.stderr:0: v1:172.21.15.1:6789/0 mon.noname-a 2024-03-25T18:34:41.621 INFO:teuthology.misc.health.smithi001.stderr:1: v1:172.21.15.115:6789/0 mon.noname-b 2024-03-25T18:34:41.622 INFO:teuthology.misc.health.smithi001.stderr:2: v1:172.21.15.1:6790/0 mon.noname-c 2024-03-25T18:34:41.622 INFO:teuthology.misc.health.smithi001.stderr: 2024-03-25T18:34:41.622 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:34:41.618+0000 7f9e8d20f640 1 librados: starting msgr at 2024-03-25T18:34:41.622 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:34:41.618+0000 7f9e8d20f640 1 librados: starting objecter 2024-03-25T18:34:41.622 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:34:41.618+0000 7f9e8d20f640 1 Processor -- start 2024-03-25T18:34:41.622 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:34:41.618+0000 7f9e8d20f640 1 -- start start 2024-03-25T18:34:41.622 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:34:41.618+0000 7f9e8d20f640 1 librados: setting wanted keys 2024-03-25T18:34:41.622 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:34:41.618+0000 7f9e8d20f640 1 librados: calling monclient init 2024-03-25T18:34:41.622 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:34:41.618+0000 7f9e8d20f640 10 monclient: init 2024-03-25T18:34:41.622 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:34:41.619+0000 7f9e8d20f640 10 monclient: _reopen_session rank -1 2024-03-25T18:34:41.622 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:34:41.619+0000 7f9e8d20f640 10 monclient: _add_conns ranks=[0,1,2] 2024-03-25T18:34:41.622 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:34:41.619+0000 7f9e8d20f640 10 monclient(hunting): picked mon.noname-a con 0x7f9e880b1b00 addr v1:172.21.15.1:6789/0 2024-03-25T18:34:41.622 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:34:41.619+0000 7f9e8d20f640 10 monclient(hunting): picked mon.noname-b con 0x7f9e880b44b0 addr v1:172.21.15.115:6789/0 2024-03-25T18:34:41.623 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:34:41.619+0000 7f9e8d20f640 10 monclient(hunting): picked mon.noname-c con 0x7f9e880b4e70 addr v1:172.21.15.1:6790/0 2024-03-25T18:34:41.623 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:34:41.619+0000 7f9e8d20f640 1 -- --> v1:172.21.15.1:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9e881d8630 con 0x7f9e880b1b00 2024-03-25T18:34:41.623 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:34:41.619+0000 7f9e8d20f640 1 -- --> v1:172.21.15.115:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9e881daf80 con 0x7f9e880b44b0 2024-03-25T18:34:41.623 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:34:41.619+0000 7f9e8d20f640 1 -- --> v1:172.21.15.1:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9e881dabe0 con 0x7f9e880b4e70 2024-03-25T18:34:41.623 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:34:41.619+0000 7f9e8d20f640 10 monclient(hunting): _renew_subs 2024-03-25T18:34:41.623 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:34:41.619+0000 7f9e8d20f640 10 monclient(hunting): authenticate will time out at 7962.089068s 2024-03-25T18:34:41.629 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:34:41.628+0000 7f9e7e59c640 1 --1- >> v1:172.21.15.115:6789/0 conn(0x7f9e880b44b0 0x7f9e881392c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.115:6789/0 says I am v1:172.21.15.1:35160/0 (socket says 172.21.15.1:35160) 2024-03-25T18:34:41.629 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:34:41.628+0000 7f9e7e59c640 1 -- 172.21.15.1:0/3570983862 learned_addr learned my addr 172.21.15.1:0/3570983862 (peer_addr_for_me v1:172.21.15.1:0/0) 2024-03-25T18:34:41.629 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:34:41.629+0000 7f9e7ffff640 10 client.?.objecter ms_handle_connect 0x7f9e880b44b0 2024-03-25T18:34:41.629 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:34:41.629+0000 7f9e7ffff640 10 client.?.objecter resend_mon_ops 2024-03-25T18:34:41.632 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:34:41.632+0000 7f9e7ffff640 10 client.?.objecter ms_handle_connect 0x7f9e880b4e70 2024-03-25T18:34:41.632 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:34:41.632+0000 7f9e7ffff640 10 client.?.objecter resend_mon_ops 2024-03-25T18:34:41.634 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:34:41.634+0000 7f9e7ffff640 1 -- 172.21.15.1:0/3570983862 <== mon.1 v1:172.21.15.115:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3717186476 0 0) 0x7f9e881daf80 con 0x7f9e880b44b0 2024-03-25T18:34:41.635 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:34:41.634+0000 7f9e7ffff640 10 monclient(hunting): _negotiate 2024-03-25T18:34:41.635 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:34:41.634+0000 7f9e7ffff640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:34:41.635 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:34:41.634+0000 7f9e7ffff640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:34:41.635 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:34:41.634+0000 7f9e7ffff640 10 monclient(hunting): my global_id is 4576 2024-03-25T18:34:41.635 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:34:41.634+0000 7f9e7ffff640 1 -- 172.21.15.1:0/3570983862 --> v1:172.21.15.115:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f9e68004eb0 con 0x7f9e880b44b0 2024-03-25T18:34:41.647 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:34:41.646+0000 7f9e7ffff640 1 -- 172.21.15.1:0/3570983862 <== mon.2 v1:172.21.15.1:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3653733551 0 0) 0x7f9e881dabe0 con 0x7f9e880b4e70 2024-03-25T18:34:41.647 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:34:41.646+0000 7f9e7ffff640 10 monclient(hunting): _negotiate 2024-03-25T18:34:41.647 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:34:41.646+0000 7f9e7ffff640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:34:41.647 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:34:41.646+0000 7f9e7ffff640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:34:41.647 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:34:41.646+0000 7f9e7ffff640 10 monclient(hunting): my global_id is 4577 2024-03-25T18:34:41.647 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:34:41.646+0000 7f9e7ffff640 1 -- 172.21.15.1:0/3570983862 --> v1:172.21.15.1:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f9e881daf80 con 0x7f9e880b4e70 2024-03-25T18:34:41.649 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:34:41.648+0000 7f9e7ffff640 1 -- 172.21.15.1:0/3570983862 <== mon.1 v1:172.21.15.115:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2642759028 0 0) 0x7f9e68004eb0 con 0x7f9e880b44b0 2024-03-25T18:34:41.649 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:34:41.648+0000 7f9e7ffff640 1 -- 172.21.15.1:0/3570983862 --> v1:172.21.15.115:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f9e881dabe0 con 0x7f9e880b44b0 2024-03-25T18:34:41.649 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:34:41.648+0000 7f9e7ffff640 1 -- 172.21.15.1:0/3570983862 <== mon.1 v1:172.21.15.115:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1586965025 0 0) 0x7f9e70001670 con 0x7f9e880b44b0 2024-03-25T18:34:41.649 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:34:41.648+0000 7f9e7ffff640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:34:41.649 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:34:41.648+0000 7f9e7ffff640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-03-25T18:34:41.649 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:34:41.648+0000 7f9e7ffff640 10 monclient(hunting): dump: 2024-03-25T18:34:41.649 INFO:teuthology.misc.health.smithi001.stderr:epoch 1 2024-03-25T18:34:41.649 INFO:teuthology.misc.health.smithi001.stderr:fsid c8ea72cf-9e99-4383-9b52-6c23d365fdfc 2024-03-25T18:34:41.649 INFO:teuthology.misc.health.smithi001.stderr:last_changed 2024-03-25T18:25:50.510519+0000 2024-03-25T18:34:41.649 INFO:teuthology.misc.health.smithi001.stderr:created 2024-03-25T18:25:50.510519+0000 2024-03-25T18:34:41.649 INFO:teuthology.misc.health.smithi001.stderr:min_mon_release 19 (squid) 2024-03-25T18:34:41.650 INFO:teuthology.misc.health.smithi001.stderr:election_strategy: 3 2024-03-25T18:34:41.650 INFO:teuthology.misc.health.smithi001.stderr:0: v1:172.21.15.1:6789/0 mon.a 2024-03-25T18:34:41.650 INFO:teuthology.misc.health.smithi001.stderr:1: v1:172.21.15.115:6789/0 mon.b 2024-03-25T18:34:41.650 INFO:teuthology.misc.health.smithi001.stderr:2: v1:172.21.15.1:6790/0 mon.c 2024-03-25T18:34:41.650 INFO:teuthology.misc.health.smithi001.stderr: 2024-03-25T18:34:41.650 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:34:41.648+0000 7f9e7ffff640 10 monclient(hunting): _finish_auth 0 2024-03-25T18:34:41.654 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:34:41.653+0000 7f9e7ffff640 1 -- 172.21.15.1:0/3570983862 <== mon.1 v1:172.21.15.115:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 4229892604 0 0) 0x7f9e881dabe0 con 0x7f9e880b44b0 2024-03-25T18:34:41.654 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:34:41.653+0000 7f9e7ffff640 1 -- 172.21.15.1:0/3570983862 >> v1:172.21.15.1:6790/0 conn(0x7f9e880b4e70 legacy=0x7f9e881e2f20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:34:41.655 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:34:41.653+0000 7f9e7ffff640 1 -- 172.21.15.1:0/3570983862 >> v1:172.21.15.1:6789/0 conn(0x7f9e880b1b00 legacy=0x7f9e881db480 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:34:41.655 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:34:41.653+0000 7f9e7ffff640 10 monclient: _finish_hunting 0 2024-03-25T18:34:41.655 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:34:41.653+0000 7f9e7ffff640 1 monclient: found mon.b 2024-03-25T18:34:41.655 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:34:41.653+0000 7f9e7ffff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:34:41.655 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:34:41.653+0000 7f9e7ffff640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.115:6789/0 2024-03-25T18:34:41.655 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:34:41.653+0000 7f9e7ffff640 1 -- 172.21.15.1:0/3570983862 --> v1:172.21.15.115:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9e881399d0 con 0x7f9e880b44b0 2024-03-25T18:34:41.655 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:34:41.653+0000 7f9e7ffff640 10 monclient: _finish_auth 0 2024-03-25T18:34:41.655 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:34:41.653+0000 7f9e7ffff640 10 monclient: _check_auth_tickets 2024-03-25T18:34:41.655 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:34:41.653+0000 7f9e7ffff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:34:41.655 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:34:41.653+0000 7f9e8d20f640 5 monclient: authenticate success, global_id 4576 2024-03-25T18:34:41.655 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:34:41.653+0000 7f9e8d20f640 10 monclient: _renew_subs 2024-03-25T18:34:41.655 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:34:41.653+0000 7f9e8d20f640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.115:6789/0 2024-03-25T18:34:41.656 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:34:41.653+0000 7f9e8d20f640 1 -- 172.21.15.1:0/3570983862 --> v1:172.21.15.115:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9e88139c20 con 0x7f9e880b44b0 2024-03-25T18:34:41.656 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:34:41.653+0000 7f9e8d20f640 10 client.4576.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T18:34:41.656 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:34:41.653+0000 7f9e8d20f640 10 monclient: _renew_subs 2024-03-25T18:34:41.656 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:34:41.653+0000 7f9e8d20f640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.115:6789/0 2024-03-25T18:34:41.656 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:34:41.653+0000 7f9e8d20f640 1 -- 172.21.15.1:0/3570983862 --> v1:172.21.15.115:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f9e88135c80 con 0x7f9e880b44b0 2024-03-25T18:34:41.656 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:34:41.653+0000 7f9e8d20f640 1 librados: init done 2024-03-25T18:34:41.657 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:34:41.654+0000 7f9e8d20f640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T18:34:41.657 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:34:41.654+0000 7f9e8d20f640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:34:41.657 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:34:41.654+0000 7f9e8d20f640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.115:6789/0 2024-03-25T18:34:41.657 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:34:41.654+0000 7f9e8d20f640 1 -- 172.21.15.1:0/3570983862 --> v1:172.21.15.115:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f9e881dad70 con 0x7f9e880b44b0 2024-03-25T18:34:41.657 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:34:41.657+0000 7f9e7ffff640 1 -- 172.21.15.1:0/3570983862 <== mon.1 v1:172.21.15.115:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f9e700038a0 con 0x7f9e880b44b0 2024-03-25T18:34:41.657 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:34:41.657+0000 7f9e7ffff640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:34:41.658 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:34:41.657+0000 7f9e85d9b640 4 set_mon_vals no callback set 2024-03-25T18:34:41.916 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:34:41.915+0000 7f9e7ffff640 1 -- 172.21.15.1:0/3570983862 <== mon.1 v1:172.21.15.115:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1586965025 0 0) 0x7f9e70003cb0 con 0x7f9e880b44b0 2024-03-25T18:34:41.916 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:34:41.915+0000 7f9e7ffff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:34:41.916 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:34:41.915+0000 7f9e7ffff640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-25T18:34:41.916 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:34:41.915+0000 7f9e7ffff640 10 monclient: dump: 2024-03-25T18:34:41.916 INFO:teuthology.misc.health.smithi001.stderr:epoch 1 2024-03-25T18:34:41.916 INFO:teuthology.misc.health.smithi001.stderr:fsid c8ea72cf-9e99-4383-9b52-6c23d365fdfc 2024-03-25T18:34:41.916 INFO:teuthology.misc.health.smithi001.stderr:last_changed 2024-03-25T18:25:50.510519+0000 2024-03-25T18:34:41.917 INFO:teuthology.misc.health.smithi001.stderr:created 2024-03-25T18:25:50.510519+0000 2024-03-25T18:34:41.917 INFO:teuthology.misc.health.smithi001.stderr:min_mon_release 19 (squid) 2024-03-25T18:34:41.917 INFO:teuthology.misc.health.smithi001.stderr:election_strategy: 3 2024-03-25T18:34:41.917 INFO:teuthology.misc.health.smithi001.stderr:0: v1:172.21.15.1:6789/0 mon.a 2024-03-25T18:34:41.917 INFO:teuthology.misc.health.smithi001.stderr:1: v1:172.21.15.115:6789/0 mon.b 2024-03-25T18:34:41.917 INFO:teuthology.misc.health.smithi001.stderr:2: v1:172.21.15.1:6790/0 mon.c 2024-03-25T18:34:41.917 INFO:teuthology.misc.health.smithi001.stderr: 2024-03-25T18:34:41.917 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:34:41.916+0000 7f9e7ffff640 1 -- 172.21.15.1:0/3570983862 <== mon.1 v1:172.21.15.115:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 21787975 0 0) 0x7f9e70003e90 con 0x7f9e880b44b0 2024-03-25T18:34:41.918 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:34:41.916+0000 7f9e86d9d640 10 monclient: get_auth_request con 0x7f9e6806dac0 auth_method 0 2024-03-25T18:34:41.918 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:34:41.916+0000 7f9e7ffff640 10 client.4576.objecter ms_handle_connect 0x7f9e6806dac0 2024-03-25T18:34:41.919 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:34:41.919+0000 7f9e7ffff640 1 -- 172.21.15.1:0/3570983862 <== mon.1 v1:172.21.15.115:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 996254230 0 0) 0x7f9e700549a0 con 0x7f9e880b44b0 2024-03-25T18:34:41.920 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:34:41.919+0000 7f9e7ffff640 10 client.4576.objecter ms_dispatch 0x7f9e88000b90 osd_map(10..10 src has 1..10) v4 2024-03-25T18:34:41.920 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:34:41.919+0000 7f9e7ffff640 3 client.4576.objecter handle_osd_map got epochs [10,10] > 0 2024-03-25T18:34:41.920 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:34:41.919+0000 7f9e7ffff640 3 client.4576.objecter handle_osd_map decoding full epoch 10 2024-03-25T18:34:41.920 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:34:41.919+0000 7f9e7ffff640 20 client.4576.objecter dump_active .. 0 homeless 2024-03-25T18:34:41.922 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:34:41.922+0000 7f9e7ffff640 1 -- 172.21.15.1:0/3570983862 <== mon.1 v1:172.21.15.115:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f9e700504b0 con 0x7f9e880b44b0 2024-03-25T18:34:41.922 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:34:41.922+0000 7f9e7ffff640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:34:41.922 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:34:41.922+0000 7f9e7ffff640 10 monclient: _finish_command 1 = system:0 2024-03-25T18:34:42.169 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:34:42.168+0000 7f9e8d20f640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:34:42.169 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:34:42.168+0000 7f9e8d20f640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:34:42.169 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:34:42.168+0000 7f9e8d20f640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.115:6789/0 2024-03-25T18:34:42.169 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:34:42.168+0000 7f9e8d20f640 1 -- 172.21.15.1:0/3570983862 --> v1:172.21.15.115:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f9e8804f730 con 0x7f9e880b44b0 2024-03-25T18:34:42.190 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:34:42.189+0000 7f9e7ffff640 1 -- 172.21.15.1:0/3570983862 <== mon.1 v1:172.21.15.115:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 4092213348) 0x7f9e700523e0 con 0x7f9e880b44b0 2024-03-25T18:34:42.190 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:34:42.189+0000 7f9e7ffff640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:34:42.190 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:34:42.189+0000 7f9e7ffff640 10 monclient: _finish_command 2 = system:0 2024-03-25T18:34:42.190 INFO:teuthology.misc.health.smithi001.stdout: 2024-03-25T18:34:42.190 INFO:teuthology.misc.health.smithi001.stdout:{"epoch":10,"fsid":"c8ea72cf-9e99-4383-9b52-6c23d365fdfc","created":"2024-03-25T18:26:38.856638+0000","modified":"2024-03-25T18:26:44.224208+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T18:26:44.224208+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"b9f34922-3a84-49be-8426-807c1eceaefc","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"42bf306f-55e3-4634-9874-8b4b73ac6644","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"292cb23a-18a9-4207-a7cc-4d5727feaec0","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"45e0e9ab-898d-4673-9590-b4117bc5e84a","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"831a5a09-73ab-4a61-9872-088e76bdbc66","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"88cd903d-3b55-4840-9396-6e3cfe005282","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"89db7355-bc9d-46f4-8ab0-e8d9e744e026","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"cac47bee-0914-4f08-896f-348838882866","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T18:34:42.193 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:34:42.191+0000 7f9e8d20f640 10 librados: watch_flush enter 2024-03-25T18:34:42.193 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:34:42.191+0000 7f9e8d20f640 10 librados: watch_flush exit 2024-03-25T18:34:42.193 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:34:42.191+0000 7f9e8d20f640 20 client.4576.objecter shutdown clearing up homeless session... 2024-03-25T18:34:42.193 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:34:42.191+0000 7f9e8d20f640 10 client.4576.objecter successfully canceled tick 2024-03-25T18:34:42.193 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:34:42.191+0000 7f9e8d20f640 1 -- 172.21.15.1:0/3570983862 >> v1:172.21.15.115:6800/4237379176 conn(0x7f9e6806dac0 legacy=0x7f9e6806ff80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:34:42.193 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:34:42.191+0000 7f9e8d20f640 10 monclient: shutdown 2024-03-25T18:34:42.193 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:34:42.191+0000 7f9e8d20f640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:34:42.193 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:34:42.191+0000 7f9e8d20f640 1 -- 172.21.15.1:0/3570983862 >> v1:172.21.15.115:6789/0 conn(0x7f9e880b44b0 legacy=0x7f9e881392c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:34:42.193 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:34:42.191+0000 7f9e8d20f640 1 -- 172.21.15.1:0/3570983862 shutdown_connections 2024-03-25T18:34:42.193 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:34:42.191+0000 7f9e8d20f640 1 -- 172.21.15.1:0/3570983862 >> 172.21.15.1:0/3570983862 conn(0x7f9e880617a0 msgr2=0x7f9e880b2710 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:34:42.193 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:34:42.192+0000 7f9e8d20f640 1 -- 172.21.15.1:0/3570983862 shutdown_connections 2024-03-25T18:34:42.193 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:34:42.192+0000 7f9e8d20f640 1 -- 172.21.15.1:0/3570983862 wait complete. 2024-03-25T18:34:42.193 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:34:42.192+0000 7f9e8d20f640 1 librados: shutdown 2024-03-25T18:34:42.204 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T18:34:42.607 INFO:tasks.ceph.mon.a.smithi001.stderr:==00:00:08:21.165 34876== Warning: unimplemented fcntl command: 1036 2024-03-25T18:34:42.623 INFO:tasks.ceph.mon.a.smithi001.stderr:==00:00:08:21.181 34876== Warning: unimplemented fcntl command: 1036 2024-03-25T18:34:42.675 INFO:tasks.ceph.mon.a.smithi001.stderr:==00:00:08:21.233 34876== Warning: unimplemented fcntl command: 1036 2024-03-25T18:34:42.688 INFO:tasks.ceph.mon.c.smithi001.stderr:==00:00:08:21.247 34877== Warning: unimplemented fcntl command: 1036 2024-03-25T18:34:42.689 INFO:tasks.ceph.mon.b.smithi115.stderr:==00:00:08:21.172 34836== Warning: unimplemented fcntl command: 1036 2024-03-25T18:34:42.695 INFO:tasks.ceph.mon.c.smithi001.stderr:==00:00:08:21.254 34877== Warning: unimplemented fcntl command: 1036 2024-03-25T18:34:42.702 INFO:tasks.ceph.mon.b.smithi115.stderr:==00:00:08:21.186 34836== Warning: unimplemented fcntl command: 1036 2024-03-25T18:34:42.729 INFO:tasks.ceph.mon.c.smithi001.stderr:==00:00:08:21.287 34877== Warning: unimplemented fcntl command: 1036 2024-03-25T18:34:42.733 INFO:tasks.ceph.mon.b.smithi115.stderr:==00:00:08:21.216 34836== Warning: unimplemented fcntl command: 1036 2024-03-25T18:34:49.011 DEBUG:teuthology.orchestra.run.smithi001:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T18:34:49.103 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:34:49.102+0000 7f4981cfb640 10 monclient: get_monmap_and_config 2024-03-25T18:34:49.103 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:34:49.102+0000 7f4981cfb640 10 monclient: build_initial_monmap 2024-03-25T18:34:49.103 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:34:49.102+0000 7f4981cfb640 10 monclient: monmap: 2024-03-25T18:34:49.103 INFO:teuthology.misc.health.smithi001.stderr:epoch 0 2024-03-25T18:34:49.103 INFO:teuthology.misc.health.smithi001.stderr:fsid c8ea72cf-9e99-4383-9b52-6c23d365fdfc 2024-03-25T18:34:49.103 INFO:teuthology.misc.health.smithi001.stderr:last_changed 2024-03-25T18:34:49.103030+0000 2024-03-25T18:34:49.103 INFO:teuthology.misc.health.smithi001.stderr:created 2024-03-25T18:34:49.103030+0000 2024-03-25T18:34:49.103 INFO:teuthology.misc.health.smithi001.stderr:min_mon_release 0 (unknown) 2024-03-25T18:34:49.104 INFO:teuthology.misc.health.smithi001.stderr:election_strategy: 3 2024-03-25T18:34:49.104 INFO:teuthology.misc.health.smithi001.stderr:0: v1:172.21.15.1:6789/0 mon.noname-a 2024-03-25T18:34:49.104 INFO:teuthology.misc.health.smithi001.stderr:1: v1:172.21.15.115:6789/0 mon.noname-c 2024-03-25T18:34:49.104 INFO:teuthology.misc.health.smithi001.stderr:2: v1:172.21.15.1:6790/0 mon.noname-b 2024-03-25T18:34:49.104 INFO:teuthology.misc.health.smithi001.stderr: 2024-03-25T18:34:49.105 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:34:49.103+0000 7f4981cfb640 1 Processor -- start 2024-03-25T18:34:49.106 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:34:49.103+0000 7f4981cfb640 1 -- start start 2024-03-25T18:34:49.106 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:34:49.103+0000 7f4981cfb640 10 monclient: init 2024-03-25T18:34:49.106 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:34:49.103+0000 7f4981cfb640 10 monclient: _reopen_session rank -1 2024-03-25T18:34:49.106 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:34:49.103+0000 7f4981cfb640 10 monclient: _add_conns ranks=[0,1,2] 2024-03-25T18:34:49.106 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:34:49.103+0000 7f4981cfb640 10 monclient(hunting): picked mon.noname-a con 0x7f497c0b4e70 addr v1:172.21.15.1:6789/0 2024-03-25T18:34:49.106 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:34:49.103+0000 7f4981cfb640 10 monclient(hunting): picked mon.noname-c con 0x7f497c0b44b0 addr v1:172.21.15.115:6789/0 2024-03-25T18:34:49.106 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:34:49.103+0000 7f4981cfb640 10 monclient(hunting): picked mon.noname-b con 0x7f497c0b1b00 addr v1:172.21.15.1:6790/0 2024-03-25T18:34:49.106 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:34:49.103+0000 7f4981cfb640 1 -- --> v1:172.21.15.1:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f497c0a96f0 con 0x7f497c0b4e70 2024-03-25T18:34:49.106 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:34:49.103+0000 7f4981cfb640 1 -- --> v1:172.21.15.115:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f497c0ab0e0 con 0x7f497c0b44b0 2024-03-25T18:34:49.106 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:34:49.103+0000 7f4981cfb640 1 -- --> v1:172.21.15.1:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f497c145f70 con 0x7f497c0b1b00 2024-03-25T18:34:49.106 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:34:49.103+0000 7f4981cfb640 10 monclient(hunting): _renew_subs 2024-03-25T18:34:49.106 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:34:49.103+0000 7f4981cfb640 10 monclient(hunting): authenticate will time out at 7969.573570s 2024-03-25T18:34:49.106 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:34:49.104+0000 7f497affd640 1 --1- >> v1:172.21.15.115:6789/0 conn(0x7f497c0b44b0 0x7f497c145860 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.115:6789/0 says I am v1:172.21.15.1:50460/0 (socket says 172.21.15.1:50460) 2024-03-25T18:34:49.107 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:34:49.104+0000 7f497affd640 1 -- 172.21.15.1:0/1392917039 learned_addr learned my addr 172.21.15.1:0/1392917039 (peer_addr_for_me v1:172.21.15.1:0/0) 2024-03-25T18:34:49.139 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:34:49.138+0000 7f497a7fc640 1 -- 172.21.15.1:0/1392917039 <== mon.0 v1:172.21.15.1:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2303743820 0 0) 0x7f497c0a96f0 con 0x7f497c0b4e70 2024-03-25T18:34:49.139 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:34:49.138+0000 7f497a7fc640 10 monclient(hunting): _negotiate 2024-03-25T18:34:49.139 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:34:49.138+0000 7f497a7fc640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:34:49.139 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:34:49.138+0000 7f497a7fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:34:49.139 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:34:49.138+0000 7f497a7fc640 10 monclient(hunting): my global_id is 4569 2024-03-25T18:34:49.139 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:34:49.138+0000 7f497a7fc640 1 -- 172.21.15.1:0/1392917039 --> v1:172.21.15.1:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f4964003520 con 0x7f497c0b4e70 2024-03-25T18:34:49.163 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:34:49.162+0000 7f497a7fc640 1 -- 172.21.15.1:0/1392917039 <== mon.0 v1:172.21.15.1:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3292298051 0 0) 0x7f4964003520 con 0x7f497c0b4e70 2024-03-25T18:34:49.163 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:34:49.162+0000 7f497a7fc640 1 -- 172.21.15.1:0/1392917039 >> v1:172.21.15.1:6790/0 conn(0x7f497c0b1b00 legacy=0x7f497c14dc00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:34:49.163 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:34:49.162+0000 7f497a7fc640 1 -- 172.21.15.1:0/1392917039 >> v1:172.21.15.115:6789/0 conn(0x7f497c0b44b0 legacy=0x7f497c145860 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:34:49.163 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:34:49.162+0000 7f497a7fc640 10 monclient: _finish_hunting 0 2024-03-25T18:34:49.163 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:34:49.162+0000 7f497a7fc640 1 monclient: found mon.noname-a 2024-03-25T18:34:49.163 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:34:49.162+0000 7f497a7fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:34:49.163 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:34:49.162+0000 7f497a7fc640 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.1:6789/0 2024-03-25T18:34:49.163 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:34:49.162+0000 7f497a7fc640 1 -- 172.21.15.1:0/1392917039 --> v1:172.21.15.1:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f497c0a9b80 con 0x7f497c0b4e70 2024-03-25T18:34:49.163 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:34:49.162+0000 7f497a7fc640 10 monclient: _finish_auth 0 2024-03-25T18:34:49.163 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:34:49.162+0000 7f497a7fc640 10 monclient: _check_auth_tickets 2024-03-25T18:34:49.163 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:34:49.162+0000 7f497a7fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:34:49.164 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:34:49.162+0000 7f4981cfb640 5 monclient: authenticate success, global_id 4569 2024-03-25T18:34:49.164 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:34:49.162+0000 7f4981cfb640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T18:34:49.180 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:34:49.180+0000 7f497a7fc640 1 -- 172.21.15.1:0/1392917039 <== mon.0 v1:172.21.15.1:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1586965025 0 0) 0x7f496c001690 con 0x7f497c0b4e70 2024-03-25T18:34:49.181 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:34:49.180+0000 7f497a7fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:34:49.181 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:34:49.180+0000 7f497a7fc640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-03-25T18:34:49.181 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:34:49.180+0000 7f497a7fc640 10 monclient: dump: 2024-03-25T18:34:49.181 INFO:teuthology.misc.health.smithi001.stderr:epoch 1 2024-03-25T18:34:49.181 INFO:teuthology.misc.health.smithi001.stderr:fsid c8ea72cf-9e99-4383-9b52-6c23d365fdfc 2024-03-25T18:34:49.181 INFO:teuthology.misc.health.smithi001.stderr:last_changed 2024-03-25T18:25:50.510519+0000 2024-03-25T18:34:49.181 INFO:teuthology.misc.health.smithi001.stderr:created 2024-03-25T18:25:50.510519+0000 2024-03-25T18:34:49.181 INFO:teuthology.misc.health.smithi001.stderr:min_mon_release 19 (squid) 2024-03-25T18:34:49.181 INFO:teuthology.misc.health.smithi001.stderr:election_strategy: 3 2024-03-25T18:34:49.181 INFO:teuthology.misc.health.smithi001.stderr:0: v1:172.21.15.1:6789/0 mon.a 2024-03-25T18:34:49.181 INFO:teuthology.misc.health.smithi001.stderr:1: v1:172.21.15.115:6789/0 mon.b 2024-03-25T18:34:49.181 INFO:teuthology.misc.health.smithi001.stderr:2: v1:172.21.15.1:6790/0 mon.c 2024-03-25T18:34:49.182 INFO:teuthology.misc.health.smithi001.stderr: 2024-03-25T18:34:49.182 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:34:49.180+0000 7f4981cfb640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T18:34:49.182 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:34:49.180+0000 7f497a7fc640 1 -- 172.21.15.1:0/1392917039 <== mon.0 v1:172.21.15.1:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f496c001920 con 0x7f497c0b4e70 2024-03-25T18:34:49.182 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:34:49.180+0000 7f497a7fc640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:34:49.182 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:34:49.180+0000 7f497a7fc640 1 -- 172.21.15.1:0/1392917039 <== mon.0 v1:172.21.15.1:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1586965025 0 0) 0x7f496c001c30 con 0x7f497c0b4e70 2024-03-25T18:34:49.182 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:34:49.180+0000 7f4981cfb640 10 monclient: get_monmap_and_config success 2024-03-25T18:34:49.182 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:34:49.180+0000 7f4981cfb640 4 set_mon_vals no callback set 2024-03-25T18:34:49.182 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:34:49.180+0000 7f497a7fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:34:49.182 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:34:49.180+0000 7f497a7fc640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-25T18:34:49.182 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:34:49.180+0000 7f497a7fc640 10 monclient: dump: 2024-03-25T18:34:49.182 INFO:teuthology.misc.health.smithi001.stderr:epoch 1 2024-03-25T18:34:49.183 INFO:teuthology.misc.health.smithi001.stderr:fsid c8ea72cf-9e99-4383-9b52-6c23d365fdfc 2024-03-25T18:34:49.183 INFO:teuthology.misc.health.smithi001.stderr:last_changed 2024-03-25T18:25:50.510519+0000 2024-03-25T18:34:49.183 INFO:teuthology.misc.health.smithi001.stderr:created 2024-03-25T18:25:50.510519+0000 2024-03-25T18:34:49.183 INFO:teuthology.misc.health.smithi001.stderr:min_mon_release 19 (squid) 2024-03-25T18:34:49.183 INFO:teuthology.misc.health.smithi001.stderr:election_strategy: 3 2024-03-25T18:34:49.183 INFO:teuthology.misc.health.smithi001.stderr:0: v1:172.21.15.1:6789/0 mon.a 2024-03-25T18:34:49.183 INFO:teuthology.misc.health.smithi001.stderr:1: v1:172.21.15.115:6789/0 mon.b 2024-03-25T18:34:49.183 INFO:teuthology.misc.health.smithi001.stderr:2: v1:172.21.15.1:6790/0 mon.c 2024-03-25T18:34:49.183 INFO:teuthology.misc.health.smithi001.stderr: 2024-03-25T18:34:49.186 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:34:49.180+0000 7f4981cfb640 10 monclient: shutdown 2024-03-25T18:34:49.186 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:34:49.180+0000 7f4981cfb640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:34:49.186 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:34:49.180+0000 7f4981cfb640 1 -- 172.21.15.1:0/1392917039 >> v1:172.21.15.1:6789/0 conn(0x7f497c0b4e70 legacy=0x7f497c13f940 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:34:49.186 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:34:49.180+0000 7f4981cfb640 1 -- 172.21.15.1:0/1392917039 shutdown_connections 2024-03-25T18:34:49.186 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:34:49.181+0000 7f4981cfb640 1 -- 172.21.15.1:0/1392917039 >> 172.21.15.1:0/1392917039 conn(0x7f497c0617a0 msgr2=0x7f497c061bb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:34:49.186 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:34:49.181+0000 7f4981cfb640 1 -- 172.21.15.1:0/1392917039 shutdown_connections 2024-03-25T18:34:49.186 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:34:49.181+0000 7f4981cfb640 1 -- 172.21.15.1:0/1392917039 wait complete. 2024-03-25T18:34:49.186 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:34:49.181+0000 7f4981cfb640 10 monclient: build_initial_monmap 2024-03-25T18:34:49.186 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:34:49.181+0000 7f4981cfb640 10 monclient: monmap: 2024-03-25T18:34:49.186 INFO:teuthology.misc.health.smithi001.stderr:epoch 0 2024-03-25T18:34:49.186 INFO:teuthology.misc.health.smithi001.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T18:34:49.186 INFO:teuthology.misc.health.smithi001.stderr:last_changed 0.000000 2024-03-25T18:34:49.186 INFO:teuthology.misc.health.smithi001.stderr:created 0.000000 2024-03-25T18:34:49.187 INFO:teuthology.misc.health.smithi001.stderr:min_mon_release 0 (unknown) 2024-03-25T18:34:49.187 INFO:teuthology.misc.health.smithi001.stderr:election_strategy: 1 2024-03-25T18:34:49.187 INFO:teuthology.misc.health.smithi001.stderr:0: v1:172.21.15.1:6789/0 mon.noname-a 2024-03-25T18:34:49.187 INFO:teuthology.misc.health.smithi001.stderr:1: v1:172.21.15.115:6789/0 mon.noname-b 2024-03-25T18:34:49.187 INFO:teuthology.misc.health.smithi001.stderr:2: v1:172.21.15.1:6790/0 mon.noname-c 2024-03-25T18:34:49.187 INFO:teuthology.misc.health.smithi001.stderr: 2024-03-25T18:34:49.187 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:34:49.182+0000 7f4981cfb640 1 librados: starting msgr at 2024-03-25T18:34:49.187 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:34:49.182+0000 7f4981cfb640 1 librados: starting objecter 2024-03-25T18:34:49.187 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:34:49.182+0000 7f4981cfb640 1 Processor -- start 2024-03-25T18:34:49.187 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:34:49.182+0000 7f4981cfb640 1 -- start start 2024-03-25T18:34:49.187 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:34:49.182+0000 7f4981cfb640 1 librados: setting wanted keys 2024-03-25T18:34:49.187 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:34:49.182+0000 7f4981cfb640 1 librados: calling monclient init 2024-03-25T18:34:49.187 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:34:49.182+0000 7f4981cfb640 10 monclient: init 2024-03-25T18:34:49.187 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:34:49.183+0000 7f4981cfb640 10 monclient: _reopen_session rank -1 2024-03-25T18:34:49.188 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:34:49.183+0000 7f4981cfb640 10 monclient: _add_conns ranks=[1,0,2] 2024-03-25T18:34:49.188 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:34:49.183+0000 7f4981cfb640 10 monclient(hunting): picked mon.noname-b con 0x7f497c0b1b00 addr v1:172.21.15.115:6789/0 2024-03-25T18:34:49.188 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:34:49.183+0000 7f4981cfb640 10 monclient(hunting): picked mon.noname-a con 0x7f497c0b44b0 addr v1:172.21.15.1:6789/0 2024-03-25T18:34:49.188 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:34:49.183+0000 7f4981cfb640 10 monclient(hunting): picked mon.noname-c con 0x7f497c0b4e70 addr v1:172.21.15.1:6790/0 2024-03-25T18:34:49.188 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:34:49.183+0000 7f4981cfb640 1 -- --> v1:172.21.15.1:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f497c1d5550 con 0x7f497c0b44b0 2024-03-25T18:34:49.188 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:34:49.183+0000 7f4981cfb640 1 -- --> v1:172.21.15.115:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f497c1d8730 con 0x7f497c0b1b00 2024-03-25T18:34:49.188 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:34:49.183+0000 7f4981cfb640 1 -- --> v1:172.21.15.1:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f497c1d7b00 con 0x7f497c0b4e70 2024-03-25T18:34:49.188 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:34:49.183+0000 7f4981cfb640 10 monclient(hunting): _renew_subs 2024-03-25T18:34:49.188 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:34:49.183+0000 7f4981cfb640 10 monclient(hunting): authenticate will time out at 7969.653080s 2024-03-25T18:34:49.188 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:34:49.183+0000 7f497bfff640 1 --1- >> v1:172.21.15.1:6790/0 conn(0x7f497c0b4e70 0x7f497c1e2140 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.1:6790/0 says I am v1:172.21.15.1:36610/0 (socket says 172.21.15.1:36610) 2024-03-25T18:34:49.188 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:34:49.184+0000 7f497bfff640 1 -- 172.21.15.1:0/2671681760 learned_addr learned my addr 172.21.15.1:0/2671681760 (peer_addr_for_me v1:172.21.15.1:0/0) 2024-03-25T18:34:49.188 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:34:49.184+0000 7f4963fff640 10 client.?.objecter ms_handle_connect 0x7f497c0b4e70 2024-03-25T18:34:49.188 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:34:49.184+0000 7f4963fff640 10 client.?.objecter resend_mon_ops 2024-03-25T18:34:49.188 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:34:49.184+0000 7f4963fff640 10 client.?.objecter ms_handle_connect 0x7f497c0b1b00 2024-03-25T18:34:49.188 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:34:49.184+0000 7f4963fff640 10 client.?.objecter resend_mon_ops 2024-03-25T18:34:49.261 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:34:49.261+0000 7f4963fff640 10 client.?.objecter ms_handle_connect 0x7f497c0b44b0 2024-03-25T18:34:49.261 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:34:49.261+0000 7f4963fff640 10 client.?.objecter resend_mon_ops 2024-03-25T18:34:49.271 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:34:49.271+0000 7f4963fff640 1 -- 172.21.15.1:0/2671681760 <== mon.0 v1:172.21.15.1:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1784450898 0 0) 0x7f497c1d5550 con 0x7f497c0b44b0 2024-03-25T18:34:49.272 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:34:49.271+0000 7f4963fff640 10 monclient(hunting): _negotiate 2024-03-25T18:34:49.272 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:34:49.271+0000 7f4963fff640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:34:49.272 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:34:49.271+0000 7f4963fff640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:34:49.272 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:34:49.271+0000 7f4963fff640 10 monclient(hunting): my global_id is 4572 2024-03-25T18:34:49.272 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:34:49.271+0000 7f4963fff640 1 -- 172.21.15.1:0/2671681760 --> v1:172.21.15.1:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f4968004eb0 con 0x7f497c0b44b0 2024-03-25T18:34:49.291 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:34:49.290+0000 7f4963fff640 1 -- 172.21.15.1:0/2671681760 <== mon.1 v1:172.21.15.115:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2136850763 0 0) 0x7f497c1d8730 con 0x7f497c0b1b00 2024-03-25T18:34:49.291 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:34:49.290+0000 7f4963fff640 10 monclient(hunting): _negotiate 2024-03-25T18:34:49.291 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:34:49.290+0000 7f4963fff640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:34:49.291 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:34:49.290+0000 7f4963fff640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:34:49.291 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:34:49.290+0000 7f4963fff640 10 monclient(hunting): my global_id is 4579 2024-03-25T18:34:49.292 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:34:49.290+0000 7f4963fff640 1 -- 172.21.15.1:0/2671681760 --> v1:172.21.15.115:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f497c1d5550 con 0x7f497c0b1b00 2024-03-25T18:34:49.303 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:34:49.302+0000 7f4963fff640 1 -- 172.21.15.1:0/2671681760 <== mon.2 v1:172.21.15.1:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1414255638 0 0) 0x7f497c1d7b00 con 0x7f497c0b4e70 2024-03-25T18:34:49.303 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:34:49.302+0000 7f4963fff640 10 monclient(hunting): _negotiate 2024-03-25T18:34:49.303 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:34:49.302+0000 7f4963fff640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:34:49.303 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:34:49.302+0000 7f4963fff640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:34:49.303 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:34:49.302+0000 7f4963fff640 10 monclient(hunting): my global_id is 4580 2024-03-25T18:34:49.303 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:34:49.302+0000 7f4963fff640 1 -- 172.21.15.1:0/2671681760 --> v1:172.21.15.1:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f497c1d8730 con 0x7f497c0b4e70 2024-03-25T18:34:49.306 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:34:49.305+0000 7f4963fff640 1 -- 172.21.15.1:0/2671681760 <== mon.0 v1:172.21.15.1:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2330517451 0 0) 0x7f4968004eb0 con 0x7f497c0b44b0 2024-03-25T18:34:49.306 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:34:49.305+0000 7f4963fff640 1 -- 172.21.15.1:0/2671681760 --> v1:172.21.15.1:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f497c1d7b00 con 0x7f497c0b44b0 2024-03-25T18:34:49.306 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:34:49.305+0000 7f4963fff640 1 -- 172.21.15.1:0/2671681760 <== mon.0 v1:172.21.15.1:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1586965025 0 0) 0x7f49700018a0 con 0x7f497c0b44b0 2024-03-25T18:34:49.306 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:34:49.305+0000 7f4963fff640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:34:49.306 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:34:49.305+0000 7f4963fff640 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2024-03-25T18:34:49.306 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:34:49.305+0000 7f4963fff640 10 monclient(hunting): dump: 2024-03-25T18:34:49.307 INFO:teuthology.misc.health.smithi001.stderr:epoch 1 2024-03-25T18:34:49.307 INFO:teuthology.misc.health.smithi001.stderr:fsid c8ea72cf-9e99-4383-9b52-6c23d365fdfc 2024-03-25T18:34:49.307 INFO:teuthology.misc.health.smithi001.stderr:last_changed 2024-03-25T18:25:50.510519+0000 2024-03-25T18:34:49.307 INFO:teuthology.misc.health.smithi001.stderr:created 2024-03-25T18:25:50.510519+0000 2024-03-25T18:34:49.307 INFO:teuthology.misc.health.smithi001.stderr:min_mon_release 19 (squid) 2024-03-25T18:34:49.307 INFO:teuthology.misc.health.smithi001.stderr:election_strategy: 3 2024-03-25T18:34:49.307 INFO:teuthology.misc.health.smithi001.stderr:0: v1:172.21.15.1:6789/0 mon.a 2024-03-25T18:34:49.307 INFO:teuthology.misc.health.smithi001.stderr:1: v1:172.21.15.115:6789/0 mon.b 2024-03-25T18:34:49.307 INFO:teuthology.misc.health.smithi001.stderr:2: v1:172.21.15.1:6790/0 mon.c 2024-03-25T18:34:49.307 INFO:teuthology.misc.health.smithi001.stderr: 2024-03-25T18:34:49.307 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:34:49.305+0000 7f4963fff640 10 monclient(hunting): _finish_auth 0 2024-03-25T18:34:49.321 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:34:49.320+0000 7f4963fff640 1 -- 172.21.15.1:0/2671681760 <== mon.0 v1:172.21.15.1:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 2040010721 0 0) 0x7f497c1d7b00 con 0x7f497c0b44b0 2024-03-25T18:34:49.321 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:34:49.320+0000 7f4963fff640 1 -- 172.21.15.1:0/2671681760 >> v1:172.21.15.1:6790/0 conn(0x7f497c0b4e70 legacy=0x7f497c1e2140 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:34:49.321 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:34:49.320+0000 7f4963fff640 1 -- 172.21.15.1:0/2671681760 >> v1:172.21.15.115:6789/0 conn(0x7f497c0b1b00 legacy=0x7f497c1d8020 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:34:49.322 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:34:49.320+0000 7f4963fff640 10 monclient: _finish_hunting 0 2024-03-25T18:34:49.322 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:34:49.320+0000 7f4963fff640 1 monclient: found mon.a 2024-03-25T18:34:49.322 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:34:49.320+0000 7f4963fff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:34:49.322 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:34:49.320+0000 7f4963fff640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.1:6789/0 2024-03-25T18:34:49.322 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:34:49.320+0000 7f4963fff640 1 -- 172.21.15.1:0/2671681760 --> v1:172.21.15.1:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f497c142300 con 0x7f497c0b44b0 2024-03-25T18:34:49.322 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:34:49.320+0000 7f4963fff640 10 monclient: _finish_auth 0 2024-03-25T18:34:49.322 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:34:49.320+0000 7f4963fff640 10 monclient: _check_auth_tickets 2024-03-25T18:34:49.322 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:34:49.320+0000 7f4963fff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:34:49.322 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:34:49.320+0000 7f4981cfb640 5 monclient: authenticate success, global_id 4572 2024-03-25T18:34:49.322 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:34:49.320+0000 7f4981cfb640 10 monclient: _renew_subs 2024-03-25T18:34:49.322 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:34:49.320+0000 7f4981cfb640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.1:6789/0 2024-03-25T18:34:49.322 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:34:49.320+0000 7f4981cfb640 1 -- 172.21.15.1:0/2671681760 --> v1:172.21.15.1:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f497c1413e0 con 0x7f497c0b44b0 2024-03-25T18:34:49.322 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:34:49.320+0000 7f4981cfb640 10 client.4572.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T18:34:49.322 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:34:49.320+0000 7f4981cfb640 10 monclient: _renew_subs 2024-03-25T18:34:49.323 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:34:49.320+0000 7f4981cfb640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.1:6789/0 2024-03-25T18:34:49.323 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:34:49.320+0000 7f4981cfb640 1 -- 172.21.15.1:0/2671681760 --> v1:172.21.15.1:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f497c1e3860 con 0x7f497c0b44b0 2024-03-25T18:34:49.323 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:34:49.320+0000 7f4981cfb640 1 librados: init done 2024-03-25T18:34:49.323 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:34:49.321+0000 7f4981cfb640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T18:34:49.323 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:34:49.321+0000 7f4981cfb640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:34:49.323 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:34:49.321+0000 7f4981cfb640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.1:6789/0 2024-03-25T18:34:49.323 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:34:49.321+0000 7f4981cfb640 1 -- 172.21.15.1:0/2671681760 --> v1:172.21.15.1:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f497c1d7cf0 con 0x7f497c0b44b0 2024-03-25T18:34:49.626 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:34:49.625+0000 7f4963fff640 1 -- 172.21.15.1:0/2671681760 <== mon.0 v1:172.21.15.1:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f4970001d10 con 0x7f497c0b44b0 2024-03-25T18:34:49.626 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:34:49.625+0000 7f4963fff640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:34:49.626 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:34:49.625+0000 7f4979ffb640 4 set_mon_vals no callback set 2024-03-25T18:34:49.639 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:34:49.639+0000 7f4963fff640 1 -- 172.21.15.1:0/2671681760 <== mon.0 v1:172.21.15.1:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1586965025 0 0) 0x7f4970004a40 con 0x7f497c0b44b0 2024-03-25T18:34:49.639 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:34:49.639+0000 7f4963fff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:34:49.640 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:34:49.639+0000 7f4963fff640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-25T18:34:49.640 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:34:49.639+0000 7f4963fff640 10 monclient: dump: 2024-03-25T18:34:49.640 INFO:teuthology.misc.health.smithi001.stderr:epoch 1 2024-03-25T18:34:49.640 INFO:teuthology.misc.health.smithi001.stderr:fsid c8ea72cf-9e99-4383-9b52-6c23d365fdfc 2024-03-25T18:34:49.640 INFO:teuthology.misc.health.smithi001.stderr:last_changed 2024-03-25T18:25:50.510519+0000 2024-03-25T18:34:49.640 INFO:teuthology.misc.health.smithi001.stderr:created 2024-03-25T18:25:50.510519+0000 2024-03-25T18:34:49.640 INFO:teuthology.misc.health.smithi001.stderr:min_mon_release 19 (squid) 2024-03-25T18:34:49.640 INFO:teuthology.misc.health.smithi001.stderr:election_strategy: 3 2024-03-25T18:34:49.640 INFO:teuthology.misc.health.smithi001.stderr:0: v1:172.21.15.1:6789/0 mon.a 2024-03-25T18:34:49.640 INFO:teuthology.misc.health.smithi001.stderr:1: v1:172.21.15.115:6789/0 mon.b 2024-03-25T18:34:49.640 INFO:teuthology.misc.health.smithi001.stderr:2: v1:172.21.15.1:6790/0 mon.c 2024-03-25T18:34:49.640 INFO:teuthology.misc.health.smithi001.stderr: 2024-03-25T18:34:49.641 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:34:49.639+0000 7f4963fff640 1 -- 172.21.15.1:0/2671681760 <== mon.0 v1:172.21.15.1:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 21787975 0 0) 0x7f4970004c20 con 0x7f497c0b44b0 2024-03-25T18:34:49.641 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:34:49.640+0000 7f497b7fe640 10 monclient: get_auth_request con 0x7f496806dda0 auth_method 0 2024-03-25T18:34:49.641 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:34:49.640+0000 7f4963fff640 10 client.4572.objecter ms_handle_connect 0x7f496806dda0 2024-03-25T18:34:49.647 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:34:49.646+0000 7f4963fff640 1 -- 172.21.15.1:0/2671681760 <== mon.0 v1:172.21.15.1:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 996254230 0 0) 0x7f49700547e0 con 0x7f497c0b44b0 2024-03-25T18:34:49.647 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:34:49.646+0000 7f4963fff640 10 client.4572.objecter ms_dispatch 0x7f497c000b90 osd_map(10..10 src has 1..10) v4 2024-03-25T18:34:49.647 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:34:49.646+0000 7f4963fff640 3 client.4572.objecter handle_osd_map got epochs [10,10] > 0 2024-03-25T18:34:49.648 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:34:49.646+0000 7f4963fff640 3 client.4572.objecter handle_osd_map decoding full epoch 10 2024-03-25T18:34:49.648 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:34:49.646+0000 7f4963fff640 20 client.4572.objecter dump_active .. 0 homeless 2024-03-25T18:34:49.648 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:34:49.646+0000 7f4963fff640 1 -- 172.21.15.1:0/2671681760 <== mon.0 v1:172.21.15.1:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f4970050120 con 0x7f497c0b44b0 2024-03-25T18:34:49.648 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:34:49.646+0000 7f4963fff640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:34:49.648 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:34:49.646+0000 7f4963fff640 10 monclient: _finish_command 1 = system:0 2024-03-25T18:34:49.915 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:34:49.914+0000 7f4981cfb640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:34:49.915 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:34:49.914+0000 7f4981cfb640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:34:49.915 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:34:49.914+0000 7f4981cfb640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.1:6789/0 2024-03-25T18:34:49.915 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:34:49.914+0000 7f4981cfb640 1 -- 172.21.15.1:0/2671681760 --> v1:172.21.15.1:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f497c143360 con 0x7f497c0b44b0 2024-03-25T18:34:49.937 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:34:49.936+0000 7f4963fff640 1 -- 172.21.15.1:0/2671681760 <== mon.0 v1:172.21.15.1:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 4092213348) 0x7f4970052220 con 0x7f497c0b44b0 2024-03-25T18:34:49.937 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:34:49.936+0000 7f4963fff640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:34:49.937 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:34:49.936+0000 7f4963fff640 10 monclient: _finish_command 2 = system:0 2024-03-25T18:34:49.937 INFO:teuthology.misc.health.smithi001.stdout: 2024-03-25T18:34:49.937 INFO:teuthology.misc.health.smithi001.stdout:{"epoch":10,"fsid":"c8ea72cf-9e99-4383-9b52-6c23d365fdfc","created":"2024-03-25T18:26:38.856638+0000","modified":"2024-03-25T18:26:44.224208+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T18:26:44.224208+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"b9f34922-3a84-49be-8426-807c1eceaefc","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"42bf306f-55e3-4634-9874-8b4b73ac6644","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"292cb23a-18a9-4207-a7cc-4d5727feaec0","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"45e0e9ab-898d-4673-9590-b4117bc5e84a","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"831a5a09-73ab-4a61-9872-088e76bdbc66","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"88cd903d-3b55-4840-9396-6e3cfe005282","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"89db7355-bc9d-46f4-8ab0-e8d9e744e026","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"cac47bee-0914-4f08-896f-348838882866","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T18:34:49.940 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:34:49.939+0000 7f4981cfb640 10 librados: watch_flush enter 2024-03-25T18:34:49.940 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:34:49.939+0000 7f4981cfb640 10 librados: watch_flush exit 2024-03-25T18:34:49.940 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:34:49.939+0000 7f4981cfb640 20 client.4572.objecter shutdown clearing up homeless session... 2024-03-25T18:34:49.940 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:34:49.939+0000 7f4981cfb640 10 client.4572.objecter successfully canceled tick 2024-03-25T18:34:49.940 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:34:49.939+0000 7f4981cfb640 1 -- 172.21.15.1:0/2671681760 >> v1:172.21.15.115:6800/4237379176 conn(0x7f496806dda0 legacy=0x7f4968070260 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:34:49.940 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:34:49.939+0000 7f4981cfb640 10 monclient: shutdown 2024-03-25T18:34:49.940 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:34:49.939+0000 7f4981cfb640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:34:49.940 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:34:49.939+0000 7f4981cfb640 1 -- 172.21.15.1:0/2671681760 >> v1:172.21.15.1:6789/0 conn(0x7f497c0b44b0 legacy=0x7f497c141bf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:34:49.941 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:34:49.939+0000 7f4981cfb640 1 -- 172.21.15.1:0/2671681760 shutdown_connections 2024-03-25T18:34:49.941 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:34:49.939+0000 7f4981cfb640 1 -- 172.21.15.1:0/2671681760 >> 172.21.15.1:0/2671681760 conn(0x7f497c0617a0 msgr2=0x7f497c13e970 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:34:49.941 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:34:49.939+0000 7f4981cfb640 1 -- 172.21.15.1:0/2671681760 shutdown_connections 2024-03-25T18:34:49.941 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:34:49.939+0000 7f4981cfb640 1 -- 172.21.15.1:0/2671681760 wait complete. 2024-03-25T18:34:49.941 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:34:49.939+0000 7f4981cfb640 1 librados: shutdown 2024-03-25T18:34:49.954 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T18:34:56.759 DEBUG:teuthology.orchestra.run.smithi001:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T18:34:56.871 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:34:56.870+0000 7fc6fc695640 10 monclient: get_monmap_and_config 2024-03-25T18:34:56.871 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:34:56.870+0000 7fc6fc695640 10 monclient: build_initial_monmap 2024-03-25T18:34:56.871 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:34:56.870+0000 7fc6fc695640 10 monclient: monmap: 2024-03-25T18:34:56.871 INFO:teuthology.misc.health.smithi001.stderr:epoch 0 2024-03-25T18:34:56.871 INFO:teuthology.misc.health.smithi001.stderr:fsid c8ea72cf-9e99-4383-9b52-6c23d365fdfc 2024-03-25T18:34:56.872 INFO:teuthology.misc.health.smithi001.stderr:last_changed 2024-03-25T18:34:56.871383+0000 2024-03-25T18:34:56.872 INFO:teuthology.misc.health.smithi001.stderr:created 2024-03-25T18:34:56.871383+0000 2024-03-25T18:34:56.872 INFO:teuthology.misc.health.smithi001.stderr:min_mon_release 0 (unknown) 2024-03-25T18:34:56.872 INFO:teuthology.misc.health.smithi001.stderr:election_strategy: 3 2024-03-25T18:34:56.872 INFO:teuthology.misc.health.smithi001.stderr:0: v1:172.21.15.1:6789/0 mon.noname-a 2024-03-25T18:34:56.872 INFO:teuthology.misc.health.smithi001.stderr:1: v1:172.21.15.115:6789/0 mon.noname-c 2024-03-25T18:34:56.872 INFO:teuthology.misc.health.smithi001.stderr:2: v1:172.21.15.1:6790/0 mon.noname-b 2024-03-25T18:34:56.872 INFO:teuthology.misc.health.smithi001.stderr: 2024-03-25T18:34:56.873 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:34:56.871+0000 7fc6fc695640 1 Processor -- start 2024-03-25T18:34:56.873 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:34:56.871+0000 7fc6fc695640 1 -- start start 2024-03-25T18:34:56.873 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:34:56.871+0000 7fc6fc695640 10 monclient: init 2024-03-25T18:34:56.874 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:34:56.872+0000 7fc6fc695640 10 monclient: _reopen_session rank -1 2024-03-25T18:34:56.874 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:34:56.872+0000 7fc6fc695640 10 monclient: _add_conns ranks=[1,2,0] 2024-03-25T18:34:56.874 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:34:56.872+0000 7fc6fc695640 10 monclient(hunting): picked mon.noname-c con 0x7fc6f40b49f0 addr v1:172.21.15.115:6789/0 2024-03-25T18:34:56.874 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:34:56.872+0000 7fc6fc695640 10 monclient(hunting): picked mon.noname-b con 0x7fc6f40b4dd0 addr v1:172.21.15.1:6790/0 2024-03-25T18:34:56.874 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:34:56.872+0000 7fc6fc695640 10 monclient(hunting): picked mon.noname-a con 0x7fc6f40a9a90 addr v1:172.21.15.1:6789/0 2024-03-25T18:34:56.874 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:34:56.872+0000 7fc6fc695640 1 -- --> v1:172.21.15.1:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc6f40aa070 con 0x7fc6f40a9a90 2024-03-25T18:34:56.874 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:34:56.872+0000 7fc6fc695640 1 -- --> v1:172.21.15.115:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc6f40aaf40 con 0x7fc6f40b49f0 2024-03-25T18:34:56.874 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:34:56.872+0000 7fc6fc695640 1 -- --> v1:172.21.15.1:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc6f40ab220 con 0x7fc6f40b4dd0 2024-03-25T18:34:56.874 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:34:56.872+0000 7fc6fc695640 10 monclient(hunting): _renew_subs 2024-03-25T18:34:56.874 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:34:56.872+0000 7fc6fc695640 10 monclient(hunting): authenticate will time out at 7977.342275s 2024-03-25T18:34:56.889 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:34:56.888+0000 7fc6fa431640 1 --1- >> v1:172.21.15.115:6789/0 conn(0x7fc6f40b49f0 0x7fc6f413bd00 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.115:6789/0 says I am v1:172.21.15.1:52118/0 (socket says 172.21.15.1:52118) 2024-03-25T18:34:56.889 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:34:56.888+0000 7fc6fa431640 1 -- 172.21.15.1:0/912064702 learned_addr learned my addr 172.21.15.1:0/912064702 (peer_addr_for_me v1:172.21.15.1:0/0) 2024-03-25T18:34:56.898 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:34:56.898+0000 7fc6f942f640 1 -- 172.21.15.1:0/912064702 <== mon.0 v1:172.21.15.1:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1805179265 0 0) 0x7fc6f40aa070 con 0x7fc6f40a9a90 2024-03-25T18:34:56.898 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:34:56.898+0000 7fc6f942f640 10 monclient(hunting): _negotiate 2024-03-25T18:34:56.899 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:34:56.898+0000 7fc6f942f640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:34:56.899 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:34:56.898+0000 7fc6f942f640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:34:56.899 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:34:56.898+0000 7fc6f942f640 10 monclient(hunting): my global_id is 4575 2024-03-25T18:34:56.899 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:34:56.898+0000 7fc6f942f640 1 -- 172.21.15.1:0/912064702 --> v1:172.21.15.1:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fc6e0003520 con 0x7fc6f40a9a90 2024-03-25T18:34:56.912 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:34:56.911+0000 7fc6f942f640 1 -- 172.21.15.1:0/912064702 <== mon.0 v1:172.21.15.1:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1560015277 0 0) 0x7fc6e0003520 con 0x7fc6f40a9a90 2024-03-25T18:34:56.912 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:34:56.911+0000 7fc6f942f640 1 -- 172.21.15.1:0/912064702 >> v1:172.21.15.1:6790/0 conn(0x7fc6f40b4dd0 legacy=0x7fc6f4142520 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:34:56.912 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:34:56.911+0000 7fc6f942f640 1 -- 172.21.15.1:0/912064702 >> v1:172.21.15.115:6789/0 conn(0x7fc6f40b49f0 legacy=0x7fc6f413bd00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:34:56.913 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:34:56.911+0000 7fc6f942f640 10 monclient: _finish_hunting 0 2024-03-25T18:34:56.913 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:34:56.911+0000 7fc6f942f640 1 monclient: found mon.noname-a 2024-03-25T18:34:56.913 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:34:56.911+0000 7fc6f942f640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:34:56.913 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:34:56.911+0000 7fc6f942f640 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.1:6789/0 2024-03-25T18:34:56.913 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:34:56.911+0000 7fc6f942f640 1 -- 172.21.15.1:0/912064702 --> v1:172.21.15.1:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc6f413c410 con 0x7fc6f40a9a90 2024-03-25T18:34:56.913 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:34:56.912+0000 7fc6f942f640 10 monclient: _finish_auth 0 2024-03-25T18:34:56.913 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:34:56.912+0000 7fc6f942f640 10 monclient: _check_auth_tickets 2024-03-25T18:34:56.913 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:34:56.912+0000 7fc6f942f640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:34:56.913 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:34:56.912+0000 7fc6fc695640 5 monclient: authenticate success, global_id 4575 2024-03-25T18:34:56.913 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:34:56.912+0000 7fc6fc695640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T18:34:56.915 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:34:56.914+0000 7fc6f942f640 1 -- 172.21.15.1:0/912064702 <== mon.0 v1:172.21.15.1:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1586965025 0 0) 0x7fc6ec001a10 con 0x7fc6f40a9a90 2024-03-25T18:34:56.915 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:34:56.914+0000 7fc6f942f640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:34:56.915 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:34:56.914+0000 7fc6f942f640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-03-25T18:34:56.915 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:34:56.914+0000 7fc6f942f640 10 monclient: dump: 2024-03-25T18:34:56.915 INFO:teuthology.misc.health.smithi001.stderr:epoch 1 2024-03-25T18:34:56.915 INFO:teuthology.misc.health.smithi001.stderr:fsid c8ea72cf-9e99-4383-9b52-6c23d365fdfc 2024-03-25T18:34:56.915 INFO:teuthology.misc.health.smithi001.stderr:last_changed 2024-03-25T18:25:50.510519+0000 2024-03-25T18:34:56.915 INFO:teuthology.misc.health.smithi001.stderr:created 2024-03-25T18:25:50.510519+0000 2024-03-25T18:34:56.915 INFO:teuthology.misc.health.smithi001.stderr:min_mon_release 19 (squid) 2024-03-25T18:34:56.915 INFO:teuthology.misc.health.smithi001.stderr:election_strategy: 3 2024-03-25T18:34:56.915 INFO:teuthology.misc.health.smithi001.stderr:0: v1:172.21.15.1:6789/0 mon.a 2024-03-25T18:34:56.915 INFO:teuthology.misc.health.smithi001.stderr:1: v1:172.21.15.115:6789/0 mon.b 2024-03-25T18:34:56.915 INFO:teuthology.misc.health.smithi001.stderr:2: v1:172.21.15.1:6790/0 mon.c 2024-03-25T18:34:56.915 INFO:teuthology.misc.health.smithi001.stderr: 2024-03-25T18:34:56.916 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:34:56.914+0000 7fc6fc695640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T18:34:56.937 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:34:56.936+0000 7fc6f942f640 1 -- 172.21.15.1:0/912064702 <== mon.0 v1:172.21.15.1:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fc6ec001ca0 con 0x7fc6f40a9a90 2024-03-25T18:34:56.937 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:34:56.936+0000 7fc6f942f640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:34:56.937 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:34:56.937+0000 7fc6f942f640 1 -- 172.21.15.1:0/912064702 <== mon.0 v1:172.21.15.1:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1586965025 0 0) 0x7fc6ec001fb0 con 0x7fc6f40a9a90 2024-03-25T18:34:56.937 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:34:56.937+0000 7fc6f942f640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:34:56.938 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:34:56.937+0000 7fc6f942f640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-25T18:34:56.938 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:34:56.937+0000 7fc6f942f640 10 monclient: dump: 2024-03-25T18:34:56.938 INFO:teuthology.misc.health.smithi001.stderr:epoch 1 2024-03-25T18:34:56.938 INFO:teuthology.misc.health.smithi001.stderr:fsid c8ea72cf-9e99-4383-9b52-6c23d365fdfc 2024-03-25T18:34:56.938 INFO:teuthology.misc.health.smithi001.stderr:last_changed 2024-03-25T18:25:50.510519+0000 2024-03-25T18:34:56.938 INFO:teuthology.misc.health.smithi001.stderr:created 2024-03-25T18:25:50.510519+0000 2024-03-25T18:34:56.938 INFO:teuthology.misc.health.smithi001.stderr:min_mon_release 19 (squid) 2024-03-25T18:34:56.938 INFO:teuthology.misc.health.smithi001.stderr:election_strategy: 3 2024-03-25T18:34:56.938 INFO:teuthology.misc.health.smithi001.stderr:0: v1:172.21.15.1:6789/0 mon.a 2024-03-25T18:34:56.938 INFO:teuthology.misc.health.smithi001.stderr:1: v1:172.21.15.115:6789/0 mon.b 2024-03-25T18:34:56.938 INFO:teuthology.misc.health.smithi001.stderr:2: v1:172.21.15.1:6790/0 mon.c 2024-03-25T18:34:56.938 INFO:teuthology.misc.health.smithi001.stderr: 2024-03-25T18:34:56.938 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:34:56.937+0000 7fc6fc695640 10 monclient: get_monmap_and_config success 2024-03-25T18:34:56.938 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:34:56.937+0000 7fc6fc695640 4 set_mon_vals no callback set 2024-03-25T18:34:56.941 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:34:56.937+0000 7fc6fc695640 10 monclient: shutdown 2024-03-25T18:34:56.941 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:34:56.937+0000 7fc6fc695640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:34:56.941 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:34:56.937+0000 7fc6fc695640 1 -- 172.21.15.1:0/912064702 >> v1:172.21.15.1:6789/0 conn(0x7fc6f40a9a90 legacy=0x7fc6f4142c30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:34:56.941 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:34:56.938+0000 7fc6fc695640 1 -- 172.21.15.1:0/912064702 shutdown_connections 2024-03-25T18:34:56.941 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:34:56.938+0000 7fc6fc695640 1 -- 172.21.15.1:0/912064702 >> 172.21.15.1:0/912064702 conn(0x7fc6f40615c0 msgr2=0x7fc6f40619a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:34:56.941 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:34:56.938+0000 7fc6fc695640 1 -- 172.21.15.1:0/912064702 shutdown_connections 2024-03-25T18:34:56.941 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:34:56.938+0000 7fc6fc695640 1 -- 172.21.15.1:0/912064702 wait complete. 2024-03-25T18:34:56.941 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:34:56.938+0000 7fc6fc695640 10 monclient: build_initial_monmap 2024-03-25T18:34:56.941 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:34:56.938+0000 7fc6fc695640 10 monclient: monmap: 2024-03-25T18:34:56.941 INFO:teuthology.misc.health.smithi001.stderr:epoch 0 2024-03-25T18:34:56.941 INFO:teuthology.misc.health.smithi001.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T18:34:56.941 INFO:teuthology.misc.health.smithi001.stderr:last_changed 0.000000 2024-03-25T18:34:56.941 INFO:teuthology.misc.health.smithi001.stderr:created 0.000000 2024-03-25T18:34:56.942 INFO:teuthology.misc.health.smithi001.stderr:min_mon_release 0 (unknown) 2024-03-25T18:34:56.942 INFO:teuthology.misc.health.smithi001.stderr:election_strategy: 1 2024-03-25T18:34:56.942 INFO:teuthology.misc.health.smithi001.stderr:0: v1:172.21.15.1:6789/0 mon.noname-a 2024-03-25T18:34:56.942 INFO:teuthology.misc.health.smithi001.stderr:1: v1:172.21.15.115:6789/0 mon.noname-b 2024-03-25T18:34:56.942 INFO:teuthology.misc.health.smithi001.stderr:2: v1:172.21.15.1:6790/0 mon.noname-c 2024-03-25T18:34:56.942 INFO:teuthology.misc.health.smithi001.stderr: 2024-03-25T18:34:56.942 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:34:56.939+0000 7fc6fc695640 1 librados: starting msgr at 2024-03-25T18:34:56.942 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:34:56.939+0000 7fc6fc695640 1 librados: starting objecter 2024-03-25T18:34:56.942 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:34:56.939+0000 7fc6fc695640 1 Processor -- start 2024-03-25T18:34:56.942 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:34:56.939+0000 7fc6fc695640 1 -- start start 2024-03-25T18:34:56.942 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:34:56.939+0000 7fc6fc695640 1 librados: setting wanted keys 2024-03-25T18:34:56.942 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:34:56.939+0000 7fc6fc695640 1 librados: calling monclient init 2024-03-25T18:34:56.942 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:34:56.939+0000 7fc6fc695640 10 monclient: init 2024-03-25T18:34:56.942 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:34:56.939+0000 7fc6fc695640 10 monclient: _reopen_session rank -1 2024-03-25T18:34:56.943 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:34:56.939+0000 7fc6fc695640 10 monclient: _add_conns ranks=[2,0,1] 2024-03-25T18:34:56.943 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:34:56.939+0000 7fc6fc695640 10 monclient(hunting): picked mon.noname-c con 0x7fc6f40a9a90 addr v1:172.21.15.1:6790/0 2024-03-25T18:34:56.943 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:34:56.940+0000 7fc6fc695640 10 monclient(hunting): picked mon.noname-a con 0x7fc6f40b49f0 addr v1:172.21.15.1:6789/0 2024-03-25T18:34:56.943 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:34:56.940+0000 7fc6fc695640 10 monclient(hunting): picked mon.noname-b con 0x7fc6f40b4dd0 addr v1:172.21.15.115:6789/0 2024-03-25T18:34:56.943 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:34:56.940+0000 7fc6fc695640 1 -- --> v1:172.21.15.1:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc6f4137860 con 0x7fc6f40b49f0 2024-03-25T18:34:56.943 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:34:56.940+0000 7fc6fc695640 1 -- --> v1:172.21.15.115:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc6f4137b40 con 0x7fc6f40b4dd0 2024-03-25T18:34:56.943 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:34:56.940+0000 7fc6fc695640 1 -- --> v1:172.21.15.1:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc6f4137e20 con 0x7fc6f40a9a90 2024-03-25T18:34:56.943 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:34:56.940+0000 7fc6fc695640 10 monclient(hunting): _renew_subs 2024-03-25T18:34:56.943 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:34:56.940+0000 7fc6fc695640 10 monclient(hunting): authenticate will time out at 7977.409897s 2024-03-25T18:34:56.943 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:34:56.940+0000 7fc6fac32640 1 --1- >> v1:172.21.15.115:6789/0 conn(0x7fc6f40b4dd0 0x7fc6f41e2300 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.115:6789/0 says I am v1:172.21.15.1:52186/0 (socket says 172.21.15.1:52186) 2024-03-25T18:34:56.943 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:34:56.940+0000 7fc6fac32640 1 -- 172.21.15.1:0/3090126873 learned_addr learned my addr 172.21.15.1:0/3090126873 (peer_addr_for_me v1:172.21.15.1:0/0) 2024-03-25T18:34:56.944 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:34:56.940+0000 7fc6eaffd640 10 client.?.objecter ms_handle_connect 0x7fc6f40b4dd0 2024-03-25T18:34:56.944 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:34:56.940+0000 7fc6eaffd640 10 client.?.objecter resend_mon_ops 2024-03-25T18:34:56.944 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:34:56.941+0000 7fc6eaffd640 10 client.?.objecter ms_handle_connect 0x7fc6f40a9a90 2024-03-25T18:34:56.944 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:34:56.941+0000 7fc6eaffd640 10 client.?.objecter resend_mon_ops 2024-03-25T18:34:56.956 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:34:56.956+0000 7fc6eaffd640 10 client.?.objecter ms_handle_connect 0x7fc6f40b49f0 2024-03-25T18:34:56.956 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:34:56.956+0000 7fc6eaffd640 10 client.?.objecter resend_mon_ops 2024-03-25T18:34:57.023 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:34:57.022+0000 7fc6eaffd640 1 -- 172.21.15.1:0/3090126873 <== mon.0 v1:172.21.15.1:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3910450673 0 0) 0x7fc6f4137860 con 0x7fc6f40b49f0 2024-03-25T18:34:57.023 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:34:57.022+0000 7fc6eaffd640 10 monclient(hunting): _negotiate 2024-03-25T18:34:57.023 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:34:57.022+0000 7fc6eaffd640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:34:57.023 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:34:57.022+0000 7fc6eaffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:34:57.023 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:34:57.022+0000 7fc6eaffd640 10 monclient(hunting): my global_id is 4578 2024-03-25T18:34:57.024 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:34:57.022+0000 7fc6eaffd640 1 -- 172.21.15.1:0/3090126873 --> v1:172.21.15.1:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fc6d8004eb0 con 0x7fc6f40b49f0 2024-03-25T18:34:57.033 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:34:57.032+0000 7fc6eaffd640 1 -- 172.21.15.1:0/3090126873 <== mon.0 v1:172.21.15.1:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2421697461 0 0) 0x7fc6d8004eb0 con 0x7fc6f40b49f0 2024-03-25T18:34:57.033 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:34:57.032+0000 7fc6eaffd640 1 -- 172.21.15.1:0/3090126873 --> v1:172.21.15.1:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fc6f4137860 con 0x7fc6f40b49f0 2024-03-25T18:34:57.033 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:34:57.032+0000 7fc6eaffd640 1 -- 172.21.15.1:0/3090126873 <== mon.0 v1:172.21.15.1:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1586965025 0 0) 0x7fc6e4001820 con 0x7fc6f40b49f0 2024-03-25T18:34:57.033 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:34:57.032+0000 7fc6eaffd640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:34:57.033 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:34:57.032+0000 7fc6eaffd640 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2024-03-25T18:34:57.033 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:34:57.032+0000 7fc6eaffd640 10 monclient(hunting): dump: 2024-03-25T18:34:57.033 INFO:teuthology.misc.health.smithi001.stderr:epoch 1 2024-03-25T18:34:57.033 INFO:teuthology.misc.health.smithi001.stderr:fsid c8ea72cf-9e99-4383-9b52-6c23d365fdfc 2024-03-25T18:34:57.033 INFO:teuthology.misc.health.smithi001.stderr:last_changed 2024-03-25T18:25:50.510519+0000 2024-03-25T18:34:57.034 INFO:teuthology.misc.health.smithi001.stderr:created 2024-03-25T18:25:50.510519+0000 2024-03-25T18:34:57.034 INFO:teuthology.misc.health.smithi001.stderr:min_mon_release 19 (squid) 2024-03-25T18:34:57.034 INFO:teuthology.misc.health.smithi001.stderr:election_strategy: 3 2024-03-25T18:34:57.034 INFO:teuthology.misc.health.smithi001.stderr:0: v1:172.21.15.1:6789/0 mon.a 2024-03-25T18:34:57.034 INFO:teuthology.misc.health.smithi001.stderr:1: v1:172.21.15.115:6789/0 mon.b 2024-03-25T18:34:57.034 INFO:teuthology.misc.health.smithi001.stderr:2: v1:172.21.15.1:6790/0 mon.c 2024-03-25T18:34:57.034 INFO:teuthology.misc.health.smithi001.stderr: 2024-03-25T18:34:57.034 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:34:57.032+0000 7fc6eaffd640 10 monclient(hunting): _finish_auth 0 2024-03-25T18:34:57.050 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:34:57.049+0000 7fc6eaffd640 1 -- 172.21.15.1:0/3090126873 <== mon.0 v1:172.21.15.1:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 1431698710 0 0) 0x7fc6f4137860 con 0x7fc6f40b49f0 2024-03-25T18:34:57.050 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:34:57.049+0000 7fc6eaffd640 1 -- 172.21.15.1:0/3090126873 >> v1:172.21.15.1:6790/0 conn(0x7fc6f40a9a90 legacy=0x7fc6f413eb90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:34:57.050 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:34:57.049+0000 7fc6eaffd640 1 -- 172.21.15.1:0/3090126873 >> v1:172.21.15.115:6789/0 conn(0x7fc6f40b4dd0 legacy=0x7fc6f41e2300 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:34:57.050 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:34:57.049+0000 7fc6eaffd640 10 monclient: _finish_hunting 0 2024-03-25T18:34:57.050 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:34:57.049+0000 7fc6eaffd640 1 monclient: found mon.a 2024-03-25T18:34:57.050 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:34:57.049+0000 7fc6eaffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:34:57.050 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:34:57.049+0000 7fc6eaffd640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.1:6789/0 2024-03-25T18:34:57.050 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:34:57.049+0000 7fc6eaffd640 1 -- 172.21.15.1:0/3090126873 --> v1:172.21.15.1:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc6f413e670 con 0x7fc6f40b49f0 2024-03-25T18:34:57.050 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:34:57.049+0000 7fc6eaffd640 10 monclient: _finish_auth 0 2024-03-25T18:34:57.051 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:34:57.049+0000 7fc6eaffd640 10 monclient: _check_auth_tickets 2024-03-25T18:34:57.051 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:34:57.049+0000 7fc6eaffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:34:57.051 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:34:57.049+0000 7fc6fc695640 5 monclient: authenticate success, global_id 4578 2024-03-25T18:34:57.051 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:34:57.049+0000 7fc6fc695640 10 monclient: _renew_subs 2024-03-25T18:34:57.051 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:34:57.049+0000 7fc6fc695640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.1:6789/0 2024-03-25T18:34:57.051 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:34:57.049+0000 7fc6fc695640 1 -- 172.21.15.1:0/3090126873 --> v1:172.21.15.1:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc6f413e040 con 0x7fc6f40b49f0 2024-03-25T18:34:57.051 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:34:57.049+0000 7fc6fc695640 10 client.4578.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T18:34:57.051 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:34:57.049+0000 7fc6fc695640 10 monclient: _renew_subs 2024-03-25T18:34:57.051 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:34:57.049+0000 7fc6fc695640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.1:6789/0 2024-03-25T18:34:57.051 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:34:57.049+0000 7fc6fc695640 1 -- 172.21.15.1:0/3090126873 --> v1:172.21.15.1:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7fc6f4145550 con 0x7fc6f40b49f0 2024-03-25T18:34:57.051 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:34:57.049+0000 7fc6fc695640 1 librados: init done 2024-03-25T18:34:57.052 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:34:57.050+0000 7fc6fc695640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T18:34:57.052 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:34:57.050+0000 7fc6fc695640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:34:57.052 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:34:57.050+0000 7fc6fc695640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.1:6789/0 2024-03-25T18:34:57.052 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:34:57.050+0000 7fc6fc695640 1 -- 172.21.15.1:0/3090126873 --> v1:172.21.15.1:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc6f413e880 con 0x7fc6f40b49f0 2024-03-25T18:34:57.357 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:34:57.356+0000 7fc6eaffd640 1 -- 172.21.15.1:0/3090126873 <== mon.0 v1:172.21.15.1:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fc6e4001d70 con 0x7fc6f40b49f0 2024-03-25T18:34:57.357 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:34:57.356+0000 7fc6eaffd640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:34:57.357 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:34:57.356+0000 7fc6eaffd640 1 -- 172.21.15.1:0/3090126873 <== mon.0 v1:172.21.15.1:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1586965025 0 0) 0x7fc6e4003790 con 0x7fc6f40b49f0 2024-03-25T18:34:57.357 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:34:57.356+0000 7fc6eaffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:34:57.357 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:34:57.356+0000 7fc6eaffd640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-25T18:34:57.357 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:34:57.356+0000 7fc6eaffd640 10 monclient: dump: 2024-03-25T18:34:57.357 INFO:teuthology.misc.health.smithi001.stderr:epoch 1 2024-03-25T18:34:57.357 INFO:teuthology.misc.health.smithi001.stderr:fsid c8ea72cf-9e99-4383-9b52-6c23d365fdfc 2024-03-25T18:34:57.357 INFO:teuthology.misc.health.smithi001.stderr:last_changed 2024-03-25T18:25:50.510519+0000 2024-03-25T18:34:57.358 INFO:teuthology.misc.health.smithi001.stderr:created 2024-03-25T18:25:50.510519+0000 2024-03-25T18:34:57.358 INFO:teuthology.misc.health.smithi001.stderr:min_mon_release 19 (squid) 2024-03-25T18:34:57.358 INFO:teuthology.misc.health.smithi001.stderr:election_strategy: 3 2024-03-25T18:34:57.358 INFO:teuthology.misc.health.smithi001.stderr:0: v1:172.21.15.1:6789/0 mon.a 2024-03-25T18:34:57.358 INFO:teuthology.misc.health.smithi001.stderr:1: v1:172.21.15.115:6789/0 mon.b 2024-03-25T18:34:57.358 INFO:teuthology.misc.health.smithi001.stderr:2: v1:172.21.15.1:6790/0 mon.c 2024-03-25T18:34:57.358 INFO:teuthology.misc.health.smithi001.stderr: 2024-03-25T18:34:57.358 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:34:57.356+0000 7fc6f8c2e640 4 set_mon_vals no callback set 2024-03-25T18:34:57.358 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:34:57.356+0000 7fc6eaffd640 1 -- 172.21.15.1:0/3090126873 <== mon.0 v1:172.21.15.1:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 21787975 0 0) 0x7fc6e40039b0 con 0x7fc6f40b49f0 2024-03-25T18:34:57.359 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:34:57.357+0000 7fc6fa431640 10 monclient: get_auth_request con 0x7fc6d806dc00 auth_method 0 2024-03-25T18:34:57.359 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:34:57.357+0000 7fc6eaffd640 10 client.4578.objecter ms_handle_connect 0x7fc6d806dc00 2024-03-25T18:34:57.363 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:34:57.362+0000 7fc6eaffd640 1 -- 172.21.15.1:0/3090126873 <== mon.0 v1:172.21.15.1:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 996254230 0 0) 0x7fc6e40534d0 con 0x7fc6f40b49f0 2024-03-25T18:34:57.363 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:34:57.362+0000 7fc6eaffd640 10 client.4578.objecter ms_dispatch 0x7fc6f4000b90 osd_map(10..10 src has 1..10) v4 2024-03-25T18:34:57.363 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:34:57.362+0000 7fc6eaffd640 3 client.4578.objecter handle_osd_map got epochs [10,10] > 0 2024-03-25T18:34:57.363 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:34:57.362+0000 7fc6eaffd640 3 client.4578.objecter handle_osd_map decoding full epoch 10 2024-03-25T18:34:57.363 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:34:57.362+0000 7fc6eaffd640 20 client.4578.objecter dump_active .. 0 homeless 2024-03-25T18:34:57.363 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:34:57.362+0000 7fc6eaffd640 1 -- 172.21.15.1:0/3090126873 <== mon.0 v1:172.21.15.1:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7fc6e404ee10 con 0x7fc6f40b49f0 2024-03-25T18:34:57.363 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:34:57.362+0000 7fc6eaffd640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:34:57.363 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:34:57.362+0000 7fc6eaffd640 10 monclient: _finish_command 1 = system:0 2024-03-25T18:34:57.594 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:34:57.593+0000 7fc6fc695640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:34:57.594 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:34:57.593+0000 7fc6fc695640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:34:57.594 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:34:57.593+0000 7fc6fc695640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.1:6789/0 2024-03-25T18:34:57.594 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:34:57.593+0000 7fc6fc695640 1 -- 172.21.15.1:0/3090126873 --> v1:172.21.15.1:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fc6f413c740 con 0x7fc6f40b49f0 2024-03-25T18:34:57.638 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:34:57.637+0000 7fc6eaffd640 1 -- 172.21.15.1:0/3090126873 <== mon.0 v1:172.21.15.1:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 4092213348) 0x7fc6e4050f10 con 0x7fc6f40b49f0 2024-03-25T18:34:57.638 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:34:57.637+0000 7fc6eaffd640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:34:57.638 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:34:57.637+0000 7fc6eaffd640 10 monclient: _finish_command 2 = system:0 2024-03-25T18:34:57.638 INFO:teuthology.misc.health.smithi001.stdout: 2024-03-25T18:34:57.638 INFO:teuthology.misc.health.smithi001.stdout:{"epoch":10,"fsid":"c8ea72cf-9e99-4383-9b52-6c23d365fdfc","created":"2024-03-25T18:26:38.856638+0000","modified":"2024-03-25T18:26:44.224208+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T18:26:44.224208+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"b9f34922-3a84-49be-8426-807c1eceaefc","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"42bf306f-55e3-4634-9874-8b4b73ac6644","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"292cb23a-18a9-4207-a7cc-4d5727feaec0","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"45e0e9ab-898d-4673-9590-b4117bc5e84a","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"831a5a09-73ab-4a61-9872-088e76bdbc66","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"88cd903d-3b55-4840-9396-6e3cfe005282","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"89db7355-bc9d-46f4-8ab0-e8d9e744e026","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"cac47bee-0914-4f08-896f-348838882866","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T18:34:57.641 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:34:57.639+0000 7fc6fc695640 10 librados: watch_flush enter 2024-03-25T18:34:57.641 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:34:57.639+0000 7fc6fc695640 10 librados: watch_flush exit 2024-03-25T18:34:57.641 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:34:57.639+0000 7fc6fc695640 20 client.4578.objecter shutdown clearing up homeless session... 2024-03-25T18:34:57.641 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:34:57.639+0000 7fc6fc695640 10 client.4578.objecter successfully canceled tick 2024-03-25T18:34:57.641 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:34:57.639+0000 7fc6fc695640 1 -- 172.21.15.1:0/3090126873 >> v1:172.21.15.115:6800/4237379176 conn(0x7fc6d806dc00 legacy=0x7fc6d80700c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:34:57.641 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:34:57.639+0000 7fc6fc695640 10 monclient: shutdown 2024-03-25T18:34:57.641 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:34:57.639+0000 7fc6fc695640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:34:57.641 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:34:57.639+0000 7fc6fc695640 1 -- 172.21.15.1:0/3090126873 >> v1:172.21.15.1:6789/0 conn(0x7fc6f40b49f0 legacy=0x7fc6f4130e70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:34:57.641 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:34:57.639+0000 7fc6fc695640 1 -- 172.21.15.1:0/3090126873 shutdown_connections 2024-03-25T18:34:57.641 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:34:57.639+0000 7fc6fc695640 1 -- 172.21.15.1:0/3090126873 >> 172.21.15.1:0/3090126873 conn(0x7fc6f40615c0 msgr2=0x7fc6f4142be0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:34:57.641 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:34:57.640+0000 7fc6fc695640 1 -- 172.21.15.1:0/3090126873 shutdown_connections 2024-03-25T18:34:57.642 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:34:57.640+0000 7fc6fc695640 1 -- 172.21.15.1:0/3090126873 wait complete. 2024-03-25T18:34:57.642 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:34:57.640+0000 7fc6fc695640 1 librados: shutdown 2024-03-25T18:34:57.651 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T18:35:02.528 INFO:tasks.ceph.mon.a.smithi001.stderr:==00:00:08:41.087 34876== Warning: unimplemented fcntl command: 1036 2024-03-25T18:35:02.533 INFO:tasks.ceph.mon.a.smithi001.stderr:==00:00:08:41.091 34876== Warning: unimplemented fcntl command: 1036 2024-03-25T18:35:02.630 INFO:tasks.ceph.mon.a.smithi001.stderr:==00:00:08:41.188 34876== Warning: unimplemented fcntl command: 1036 2024-03-25T18:35:02.644 INFO:tasks.ceph.mon.b.smithi115.stderr:==00:00:08:41.128 34836== Warning: unimplemented fcntl command: 1036 2024-03-25T18:35:02.653 INFO:tasks.ceph.mon.c.smithi001.stderr:==00:00:08:41.211 34877== Warning: unimplemented fcntl command: 1036 2024-03-25T18:35:02.660 INFO:tasks.ceph.mon.b.smithi115.stderr:==00:00:08:41.143 34836== Warning: unimplemented fcntl command: 1036 2024-03-25T18:35:02.665 INFO:tasks.ceph.mon.c.smithi001.stderr:==00:00:08:41.223 34877== Warning: unimplemented fcntl command: 1036 2024-03-25T18:35:02.684 INFO:tasks.ceph.mon.b.smithi115.stderr:==00:00:08:41.168 34836== Warning: unimplemented fcntl command: 1036 2024-03-25T18:35:02.702 INFO:tasks.ceph.mon.c.smithi001.stderr:==00:00:08:41.261 34877== Warning: unimplemented fcntl command: 1036 2024-03-25T18:35:04.458 DEBUG:teuthology.orchestra.run.smithi001:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T18:35:04.573 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:35:04.572+0000 7f292c22e640 10 monclient: get_monmap_and_config 2024-03-25T18:35:04.573 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:35:04.572+0000 7f292c22e640 10 monclient: build_initial_monmap 2024-03-25T18:35:04.573 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:35:04.572+0000 7f292c22e640 10 monclient: monmap: 2024-03-25T18:35:04.573 INFO:teuthology.misc.health.smithi001.stderr:epoch 0 2024-03-25T18:35:04.573 INFO:teuthology.misc.health.smithi001.stderr:fsid c8ea72cf-9e99-4383-9b52-6c23d365fdfc 2024-03-25T18:35:04.573 INFO:teuthology.misc.health.smithi001.stderr:last_changed 2024-03-25T18:35:04.572959+0000 2024-03-25T18:35:04.573 INFO:teuthology.misc.health.smithi001.stderr:created 2024-03-25T18:35:04.572959+0000 2024-03-25T18:35:04.573 INFO:teuthology.misc.health.smithi001.stderr:min_mon_release 0 (unknown) 2024-03-25T18:35:04.573 INFO:teuthology.misc.health.smithi001.stderr:election_strategy: 3 2024-03-25T18:35:04.574 INFO:teuthology.misc.health.smithi001.stderr:0: v1:172.21.15.1:6789/0 mon.noname-a 2024-03-25T18:35:04.574 INFO:teuthology.misc.health.smithi001.stderr:1: v1:172.21.15.115:6789/0 mon.noname-c 2024-03-25T18:35:04.574 INFO:teuthology.misc.health.smithi001.stderr:2: v1:172.21.15.1:6790/0 mon.noname-b 2024-03-25T18:35:04.574 INFO:teuthology.misc.health.smithi001.stderr: 2024-03-25T18:35:04.575 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:35:04.573+0000 7f292c22e640 1 Processor -- start 2024-03-25T18:35:04.575 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:35:04.573+0000 7f292c22e640 1 -- start start 2024-03-25T18:35:04.575 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:35:04.573+0000 7f292c22e640 10 monclient: init 2024-03-25T18:35:04.575 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:35:04.573+0000 7f292c22e640 10 monclient: _reopen_session rank -1 2024-03-25T18:35:04.575 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:35:04.573+0000 7f292c22e640 10 monclient: _add_conns ranks=[0,1,2] 2024-03-25T18:35:04.575 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:35:04.573+0000 7f292c22e640 10 monclient(hunting): picked mon.noname-a con 0x7f29240b4e70 addr v1:172.21.15.1:6789/0 2024-03-25T18:35:04.575 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:35:04.573+0000 7f292c22e640 10 monclient(hunting): picked mon.noname-c con 0x7f29240b44b0 addr v1:172.21.15.115:6789/0 2024-03-25T18:35:04.575 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:35:04.573+0000 7f292c22e640 10 monclient(hunting): picked mon.noname-b con 0x7f29240b1b00 addr v1:172.21.15.1:6790/0 2024-03-25T18:35:04.576 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:35:04.573+0000 7f292c22e640 1 -- --> v1:172.21.15.1:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f29240a96f0 con 0x7f29240b4e70 2024-03-25T18:35:04.576 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:35:04.573+0000 7f292c22e640 1 -- --> v1:172.21.15.115:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f29240ab0e0 con 0x7f29240b44b0 2024-03-25T18:35:04.576 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:35:04.573+0000 7f292c22e640 1 -- --> v1:172.21.15.1:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2924145f70 con 0x7f29240b1b00 2024-03-25T18:35:04.576 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:35:04.573+0000 7f292c22e640 10 monclient(hunting): _renew_subs 2024-03-25T18:35:04.576 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:35:04.573+0000 7f292c22e640 10 monclient(hunting): authenticate will time out at 7985.043606s 2024-03-25T18:35:04.578 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:35:04.577+0000 7f29297c9640 1 --1- >> v1:172.21.15.115:6789/0 conn(0x7f29240b44b0 0x7f2924145860 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.115:6789/0 says I am v1:172.21.15.1:60624/0 (socket says 172.21.15.1:60624) 2024-03-25T18:35:04.578 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:35:04.577+0000 7f29297c9640 1 -- 172.21.15.1:0/1386668227 learned_addr learned my addr 172.21.15.1:0/1386668227 (peer_addr_for_me v1:172.21.15.1:0/0) 2024-03-25T18:35:04.585 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:35:04.584+0000 7f2928fc8640 1 -- 172.21.15.1:0/1386668227 <== mon.1 v1:172.21.15.115:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2231208848 0 0) 0x7f29240ab0e0 con 0x7f29240b44b0 2024-03-25T18:35:04.585 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:35:04.584+0000 7f2928fc8640 10 monclient(hunting): _negotiate 2024-03-25T18:35:04.585 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:35:04.584+0000 7f2928fc8640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:35:04.585 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:35:04.584+0000 7f2928fc8640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:35:04.585 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:35:04.584+0000 7f2928fc8640 10 monclient(hunting): my global_id is 4585 2024-03-25T18:35:04.585 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:35:04.584+0000 7f2928fc8640 1 -- 172.21.15.1:0/1386668227 --> v1:172.21.15.115:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f2910003520 con 0x7f29240b44b0 2024-03-25T18:35:04.591 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:35:04.589+0000 7f2928fc8640 1 -- 172.21.15.1:0/1386668227 <== mon.1 v1:172.21.15.115:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2796320560 0 0) 0x7f2910003520 con 0x7f29240b44b0 2024-03-25T18:35:04.591 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:35:04.589+0000 7f2928fc8640 1 -- 172.21.15.1:0/1386668227 >> v1:172.21.15.1:6790/0 conn(0x7f29240b1b00 legacy=0x7f292414dc00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-25T18:35:04.591 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:35:04.589+0000 7f2928fc8640 1 -- 172.21.15.1:0/1386668227 >> v1:172.21.15.1:6789/0 conn(0x7f29240b4e70 legacy=0x7f292413f940 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-25T18:35:04.591 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:35:04.589+0000 7f2928fc8640 10 monclient: _finish_hunting 0 2024-03-25T18:35:04.591 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:35:04.589+0000 7f2928fc8640 1 monclient: found mon.noname-c 2024-03-25T18:35:04.591 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:35:04.589+0000 7f2928fc8640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:35:04.591 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:35:04.589+0000 7f2928fc8640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.115:6789/0 2024-03-25T18:35:04.591 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:35:04.589+0000 7f2928fc8640 1 -- 172.21.15.1:0/1386668227 --> v1:172.21.15.115:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f29240a9b80 con 0x7f29240b44b0 2024-03-25T18:35:04.591 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:35:04.589+0000 7f2928fc8640 10 monclient: _finish_auth 0 2024-03-25T18:35:04.591 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:35:04.589+0000 7f2928fc8640 10 monclient: _check_auth_tickets 2024-03-25T18:35:04.591 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:35:04.589+0000 7f2928fc8640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:35:04.591 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:35:04.589+0000 7f2928fc8640 1 -- 172.21.15.1:0/1386668227 <== mon.1 v1:172.21.15.115:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1586965025 0 0) 0x7f2914001690 con 0x7f29240b44b0 2024-03-25T18:35:04.591 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:35:04.589+0000 7f2928fc8640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:35:04.592 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:35:04.589+0000 7f2928fc8640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-03-25T18:35:04.592 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:35:04.589+0000 7f2928fc8640 10 monclient: dump: 2024-03-25T18:35:04.592 INFO:teuthology.misc.health.smithi001.stderr:epoch 1 2024-03-25T18:35:04.592 INFO:teuthology.misc.health.smithi001.stderr:fsid c8ea72cf-9e99-4383-9b52-6c23d365fdfc 2024-03-25T18:35:04.592 INFO:teuthology.misc.health.smithi001.stderr:last_changed 2024-03-25T18:25:50.510519+0000 2024-03-25T18:35:04.592 INFO:teuthology.misc.health.smithi001.stderr:created 2024-03-25T18:25:50.510519+0000 2024-03-25T18:35:04.592 INFO:teuthology.misc.health.smithi001.stderr:min_mon_release 19 (squid) 2024-03-25T18:35:04.592 INFO:teuthology.misc.health.smithi001.stderr:election_strategy: 3 2024-03-25T18:35:04.592 INFO:teuthology.misc.health.smithi001.stderr:0: v1:172.21.15.1:6789/0 mon.a 2024-03-25T18:35:04.592 INFO:teuthology.misc.health.smithi001.stderr:1: v1:172.21.15.115:6789/0 mon.b 2024-03-25T18:35:04.592 INFO:teuthology.misc.health.smithi001.stderr:2: v1:172.21.15.1:6790/0 mon.c 2024-03-25T18:35:04.592 INFO:teuthology.misc.health.smithi001.stderr: 2024-03-25T18:35:04.592 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:35:04.589+0000 7f292c22e640 5 monclient: authenticate success, global_id 4585 2024-03-25T18:35:04.592 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:35:04.589+0000 7f292c22e640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T18:35:04.596 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:35:04.595+0000 7f2928fc8640 1 -- 172.21.15.1:0/1386668227 <== mon.1 v1:172.21.15.115:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f2914001860 con 0x7f29240b44b0 2024-03-25T18:35:04.596 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:35:04.595+0000 7f2928fc8640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:35:04.596 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:35:04.595+0000 7f2928fc8640 1 -- 172.21.15.1:0/1386668227 <== mon.1 v1:172.21.15.115:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1586965025 0 0) 0x7f2914001cb0 con 0x7f29240b44b0 2024-03-25T18:35:04.596 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:35:04.595+0000 7f2928fc8640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:35:04.596 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:35:04.595+0000 7f2928fc8640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-25T18:35:04.596 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:35:04.595+0000 7f2928fc8640 10 monclient: dump: 2024-03-25T18:35:04.596 INFO:teuthology.misc.health.smithi001.stderr:epoch 1 2024-03-25T18:35:04.596 INFO:teuthology.misc.health.smithi001.stderr:fsid c8ea72cf-9e99-4383-9b52-6c23d365fdfc 2024-03-25T18:35:04.596 INFO:teuthology.misc.health.smithi001.stderr:last_changed 2024-03-25T18:25:50.510519+0000 2024-03-25T18:35:04.597 INFO:teuthology.misc.health.smithi001.stderr:created 2024-03-25T18:25:50.510519+0000 2024-03-25T18:35:04.597 INFO:teuthology.misc.health.smithi001.stderr:min_mon_release 19 (squid) 2024-03-25T18:35:04.597 INFO:teuthology.misc.health.smithi001.stderr:election_strategy: 3 2024-03-25T18:35:04.597 INFO:teuthology.misc.health.smithi001.stderr:0: v1:172.21.15.1:6789/0 mon.a 2024-03-25T18:35:04.597 INFO:teuthology.misc.health.smithi001.stderr:1: v1:172.21.15.115:6789/0 mon.b 2024-03-25T18:35:04.597 INFO:teuthology.misc.health.smithi001.stderr:2: v1:172.21.15.1:6790/0 mon.c 2024-03-25T18:35:04.597 INFO:teuthology.misc.health.smithi001.stderr: 2024-03-25T18:35:04.597 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:35:04.595+0000 7f292c22e640 10 monclient: get_monmap_and_config success 2024-03-25T18:35:04.597 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:35:04.595+0000 7f292c22e640 4 set_mon_vals no callback set 2024-03-25T18:35:04.599 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:35:04.596+0000 7f292c22e640 10 monclient: shutdown 2024-03-25T18:35:04.599 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:35:04.596+0000 7f292c22e640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:35:04.599 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:35:04.596+0000 7f292c22e640 1 -- 172.21.15.1:0/1386668227 >> v1:172.21.15.115:6789/0 conn(0x7f29240b44b0 legacy=0x7f2924145860 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:35:04.599 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:35:04.596+0000 7f292c22e640 1 -- 172.21.15.1:0/1386668227 shutdown_connections 2024-03-25T18:35:04.599 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:35:04.596+0000 7f292c22e640 1 -- 172.21.15.1:0/1386668227 >> 172.21.15.1:0/1386668227 conn(0x7f29240617a0 msgr2=0x7f2924061bb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:35:04.599 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:35:04.596+0000 7f292c22e640 1 -- 172.21.15.1:0/1386668227 shutdown_connections 2024-03-25T18:35:04.599 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:35:04.596+0000 7f292c22e640 1 -- 172.21.15.1:0/1386668227 wait complete. 2024-03-25T18:35:04.599 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:35:04.597+0000 7f292c22e640 10 monclient: build_initial_monmap 2024-03-25T18:35:04.599 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:35:04.597+0000 7f292c22e640 10 monclient: monmap: 2024-03-25T18:35:04.599 INFO:teuthology.misc.health.smithi001.stderr:epoch 0 2024-03-25T18:35:04.599 INFO:teuthology.misc.health.smithi001.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T18:35:04.599 INFO:teuthology.misc.health.smithi001.stderr:last_changed 0.000000 2024-03-25T18:35:04.599 INFO:teuthology.misc.health.smithi001.stderr:created 0.000000 2024-03-25T18:35:04.600 INFO:teuthology.misc.health.smithi001.stderr:min_mon_release 0 (unknown) 2024-03-25T18:35:04.600 INFO:teuthology.misc.health.smithi001.stderr:election_strategy: 1 2024-03-25T18:35:04.600 INFO:teuthology.misc.health.smithi001.stderr:0: v1:172.21.15.1:6789/0 mon.noname-a 2024-03-25T18:35:04.600 INFO:teuthology.misc.health.smithi001.stderr:1: v1:172.21.15.115:6789/0 mon.noname-b 2024-03-25T18:35:04.600 INFO:teuthology.misc.health.smithi001.stderr:2: v1:172.21.15.1:6790/0 mon.noname-c 2024-03-25T18:35:04.600 INFO:teuthology.misc.health.smithi001.stderr: 2024-03-25T18:35:04.600 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:35:04.597+0000 7f292c22e640 1 librados: starting msgr at 2024-03-25T18:35:04.600 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:35:04.597+0000 7f292c22e640 1 librados: starting objecter 2024-03-25T18:35:04.600 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:35:04.597+0000 7f292c22e640 1 Processor -- start 2024-03-25T18:35:04.600 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:35:04.598+0000 7f292c22e640 1 -- start start 2024-03-25T18:35:04.600 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:35:04.598+0000 7f292c22e640 1 librados: setting wanted keys 2024-03-25T18:35:04.600 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:35:04.598+0000 7f292c22e640 1 librados: calling monclient init 2024-03-25T18:35:04.600 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:35:04.598+0000 7f292c22e640 10 monclient: init 2024-03-25T18:35:04.601 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:35:04.598+0000 7f292c22e640 10 monclient: _reopen_session rank -1 2024-03-25T18:35:04.601 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:35:04.598+0000 7f292c22e640 10 monclient: _add_conns ranks=[2,0,1] 2024-03-25T18:35:04.601 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:35:04.598+0000 7f292c22e640 10 monclient(hunting): picked mon.noname-c con 0x7f29240b1b00 addr v1:172.21.15.1:6790/0 2024-03-25T18:35:04.602 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:35:04.598+0000 7f292c22e640 10 monclient(hunting): picked mon.noname-a con 0x7f29240b44b0 addr v1:172.21.15.1:6789/0 2024-03-25T18:35:04.602 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:35:04.598+0000 7f292c22e640 10 monclient(hunting): picked mon.noname-b con 0x7f29240b4e70 addr v1:172.21.15.115:6789/0 2024-03-25T18:35:04.602 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:35:04.599+0000 7f292c22e640 1 -- --> v1:172.21.15.1:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f29240ae4f0 con 0x7f29240b44b0 2024-03-25T18:35:04.602 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:35:04.599+0000 7f292c22e640 1 -- --> v1:172.21.15.115:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f29240ad5f0 con 0x7f29240b4e70 2024-03-25T18:35:04.602 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:35:04.599+0000 7f292c22e640 1 -- --> v1:172.21.15.1:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f29240ad0d0 con 0x7f29240b1b00 2024-03-25T18:35:04.602 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:35:04.599+0000 7f292c22e640 10 monclient(hunting): _renew_subs 2024-03-25T18:35:04.602 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:35:04.599+0000 7f292c22e640 10 monclient(hunting): authenticate will time out at 7985.068792s 2024-03-25T18:35:04.608 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:35:04.607+0000 7f2929fca640 1 --1- >> v1:172.21.15.1:6790/0 conn(0x7f29240b1b00 0x7f2924141510 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.1:6790/0 says I am v1:172.21.15.1:33752/0 (socket says 172.21.15.1:33752) 2024-03-25T18:35:04.608 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:35:04.607+0000 7f2929fca640 1 -- 172.21.15.1:0/1469620024 learned_addr learned my addr 172.21.15.1:0/1469620024 (peer_addr_for_me v1:172.21.15.1:0/0) 2024-03-25T18:35:04.611 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:35:04.610+0000 7f291a7fc640 10 client.?.objecter ms_handle_connect 0x7f29240b1b00 2024-03-25T18:35:04.611 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:35:04.610+0000 7f291a7fc640 10 client.?.objecter resend_mon_ops 2024-03-25T18:35:04.613 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:35:04.612+0000 7f291a7fc640 10 client.?.objecter ms_handle_connect 0x7f29240b44b0 2024-03-25T18:35:04.613 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:35:04.612+0000 7f291a7fc640 10 client.?.objecter resend_mon_ops 2024-03-25T18:35:04.629 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:35:04.629+0000 7f291a7fc640 1 -- 172.21.15.1:0/1469620024 <== mon.0 v1:172.21.15.1:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2621611595 0 0) 0x7f29240ae4f0 con 0x7f29240b44b0 2024-03-25T18:35:04.630 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:35:04.629+0000 7f291a7fc640 10 monclient(hunting): _negotiate 2024-03-25T18:35:04.630 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:35:04.629+0000 7f291a7fc640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:35:04.630 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:35:04.629+0000 7f291a7fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:35:04.630 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:35:04.629+0000 7f291a7fc640 10 monclient(hunting): my global_id is 4581 2024-03-25T18:35:04.630 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:35:04.629+0000 7f291a7fc640 1 -- 172.21.15.1:0/1469620024 --> v1:172.21.15.1:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f2908004eb0 con 0x7f29240b44b0 2024-03-25T18:35:04.633 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:35:04.632+0000 7f291a7fc640 1 -- 172.21.15.1:0/1469620024 <== mon.2 v1:172.21.15.1:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2618638379 0 0) 0x7f29240ad0d0 con 0x7f29240b1b00 2024-03-25T18:35:04.633 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:35:04.632+0000 7f291a7fc640 10 monclient(hunting): _negotiate 2024-03-25T18:35:04.633 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:35:04.632+0000 7f291a7fc640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:35:04.633 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:35:04.632+0000 7f291a7fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:35:04.633 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:35:04.632+0000 7f291a7fc640 10 monclient(hunting): my global_id is 4589 2024-03-25T18:35:04.633 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:35:04.632+0000 7f291a7fc640 1 -- 172.21.15.1:0/1469620024 --> v1:172.21.15.1:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f29240ae4f0 con 0x7f29240b1b00 2024-03-25T18:35:04.640 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:35:04.640+0000 7f291a7fc640 1 -- 172.21.15.1:0/1469620024 <== mon.0 v1:172.21.15.1:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 583400410 0 0) 0x7f2908004eb0 con 0x7f29240b44b0 2024-03-25T18:35:04.640 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:35:04.640+0000 7f291a7fc640 1 -- 172.21.15.1:0/1469620024 --> v1:172.21.15.1:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f29240ad0d0 con 0x7f29240b44b0 2024-03-25T18:35:04.641 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:35:04.640+0000 7f291a7fc640 10 client.?.objecter ms_handle_connect 0x7f29240b4e70 2024-03-25T18:35:04.641 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:35:04.640+0000 7f291a7fc640 10 client.?.objecter resend_mon_ops 2024-03-25T18:35:04.641 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:35:04.641+0000 7f291a7fc640 1 -- 172.21.15.1:0/1469620024 <== mon.0 v1:172.21.15.1:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1586965025 0 0) 0x7f2914001bb0 con 0x7f29240b44b0 2024-03-25T18:35:04.641 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:35:04.641+0000 7f291a7fc640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:35:04.641 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:35:04.641+0000 7f291a7fc640 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2024-03-25T18:35:04.641 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:35:04.641+0000 7f291a7fc640 10 monclient(hunting): dump: 2024-03-25T18:35:04.641 INFO:teuthology.misc.health.smithi001.stderr:epoch 1 2024-03-25T18:35:04.642 INFO:teuthology.misc.health.smithi001.stderr:fsid c8ea72cf-9e99-4383-9b52-6c23d365fdfc 2024-03-25T18:35:04.642 INFO:teuthology.misc.health.smithi001.stderr:last_changed 2024-03-25T18:25:50.510519+0000 2024-03-25T18:35:04.642 INFO:teuthology.misc.health.smithi001.stderr:created 2024-03-25T18:25:50.510519+0000 2024-03-25T18:35:04.642 INFO:teuthology.misc.health.smithi001.stderr:min_mon_release 19 (squid) 2024-03-25T18:35:04.642 INFO:teuthology.misc.health.smithi001.stderr:election_strategy: 3 2024-03-25T18:35:04.642 INFO:teuthology.misc.health.smithi001.stderr:0: v1:172.21.15.1:6789/0 mon.a 2024-03-25T18:35:04.642 INFO:teuthology.misc.health.smithi001.stderr:1: v1:172.21.15.115:6789/0 mon.b 2024-03-25T18:35:04.642 INFO:teuthology.misc.health.smithi001.stderr:2: v1:172.21.15.1:6790/0 mon.c 2024-03-25T18:35:04.642 INFO:teuthology.misc.health.smithi001.stderr: 2024-03-25T18:35:04.642 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:35:04.641+0000 7f291a7fc640 10 monclient(hunting): _finish_auth 0 2024-03-25T18:35:04.643 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:35:04.642+0000 7f291a7fc640 1 -- 172.21.15.1:0/1469620024 <== mon.2 v1:172.21.15.1:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2012489727 0 0) 0x7f29240ae4f0 con 0x7f29240b1b00 2024-03-25T18:35:04.643 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:35:04.642+0000 7f291a7fc640 1 -- 172.21.15.1:0/1469620024 --> v1:172.21.15.1:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f2908004eb0 con 0x7f29240b1b00 2024-03-25T18:35:04.643 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:35:04.642+0000 7f291a7fc640 1 -- 172.21.15.1:0/1469620024 <== mon.2 v1:172.21.15.1:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1586965025 0 0) 0x7f2920001790 con 0x7f29240b1b00 2024-03-25T18:35:04.643 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:35:04.642+0000 7f291a7fc640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:35:04.643 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:35:04.642+0000 7f291a7fc640 10 monclient(hunting): got monmap 1 from mon.c (according to old e1) 2024-03-25T18:35:04.643 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:35:04.642+0000 7f291a7fc640 10 monclient(hunting): dump: 2024-03-25T18:35:04.643 INFO:teuthology.misc.health.smithi001.stderr:epoch 1 2024-03-25T18:35:04.643 INFO:teuthology.misc.health.smithi001.stderr:fsid c8ea72cf-9e99-4383-9b52-6c23d365fdfc 2024-03-25T18:35:04.643 INFO:teuthology.misc.health.smithi001.stderr:last_changed 2024-03-25T18:25:50.510519+0000 2024-03-25T18:35:04.643 INFO:teuthology.misc.health.smithi001.stderr:created 2024-03-25T18:25:50.510519+0000 2024-03-25T18:35:04.644 INFO:teuthology.misc.health.smithi001.stderr:min_mon_release 19 (squid) 2024-03-25T18:35:04.644 INFO:teuthology.misc.health.smithi001.stderr:election_strategy: 3 2024-03-25T18:35:04.644 INFO:teuthology.misc.health.smithi001.stderr:0: v1:172.21.15.1:6789/0 mon.a 2024-03-25T18:35:04.644 INFO:teuthology.misc.health.smithi001.stderr:1: v1:172.21.15.115:6789/0 mon.b 2024-03-25T18:35:04.644 INFO:teuthology.misc.health.smithi001.stderr:2: v1:172.21.15.1:6790/0 mon.c 2024-03-25T18:35:04.644 INFO:teuthology.misc.health.smithi001.stderr: 2024-03-25T18:35:04.649 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:35:04.649+0000 7f291a7fc640 1 -- 172.21.15.1:0/1469620024 <== mon.1 v1:172.21.15.115:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2562573164 0 0) 0x7f29240ad5f0 con 0x7f29240b4e70 2024-03-25T18:35:04.649 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:35:04.649+0000 7f291a7fc640 10 monclient(hunting): _negotiate 2024-03-25T18:35:04.650 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:35:04.649+0000 7f291a7fc640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:35:04.650 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:35:04.649+0000 7f291a7fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:35:04.650 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:35:04.649+0000 7f291a7fc640 10 monclient(hunting): my global_id is 4588 2024-03-25T18:35:04.650 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:35:04.649+0000 7f291a7fc640 1 -- 172.21.15.1:0/1469620024 --> v1:172.21.15.115:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f29240ae4f0 con 0x7f29240b4e70 2024-03-25T18:35:04.663 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:35:04.661+0000 7f291a7fc640 1 -- 172.21.15.1:0/1469620024 <== mon.2 v1:172.21.15.1:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 1706201179 0 0) 0x7f2908004eb0 con 0x7f29240b1b00 2024-03-25T18:35:04.663 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:35:04.661+0000 7f291a7fc640 1 -- 172.21.15.1:0/1469620024 >> v1:172.21.15.115:6789/0 conn(0x7f29240b4e70 legacy=0x7f29241e2cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:35:04.663 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:35:04.661+0000 7f291a7fc640 1 -- 172.21.15.1:0/1469620024 >> v1:172.21.15.1:6789/0 conn(0x7f29240b44b0 legacy=0x7f2924141fd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:35:04.663 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:35:04.661+0000 7f291a7fc640 10 monclient: _finish_hunting 0 2024-03-25T18:35:04.663 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:35:04.661+0000 7f291a7fc640 1 monclient: found mon.c 2024-03-25T18:35:04.663 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:35:04.661+0000 7f291a7fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:35:04.663 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:35:04.661+0000 7f291a7fc640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.1:6790/0 2024-03-25T18:35:04.663 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:35:04.661+0000 7f291a7fc640 1 -- 172.21.15.1:0/1469620024 --> v1:172.21.15.1:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f29240ad260 con 0x7f29240b1b00 2024-03-25T18:35:04.663 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:35:04.661+0000 7f291a7fc640 10 monclient: _finish_auth 0 2024-03-25T18:35:04.663 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:35:04.661+0000 7f291a7fc640 10 monclient: _check_auth_tickets 2024-03-25T18:35:04.663 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:35:04.661+0000 7f291a7fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:35:04.663 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:35:04.661+0000 7f292c22e640 5 monclient: authenticate success, global_id 4589 2024-03-25T18:35:04.663 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:35:04.661+0000 7f292c22e640 10 monclient: _renew_subs 2024-03-25T18:35:04.664 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:35:04.661+0000 7f292c22e640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.1:6790/0 2024-03-25T18:35:04.664 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:35:04.661+0000 7f292c22e640 1 -- 172.21.15.1:0/1469620024 --> v1:172.21.15.1:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f29241dac00 con 0x7f29240b1b00 2024-03-25T18:35:04.664 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:35:04.661+0000 7f292c22e640 10 client.4589.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T18:35:04.664 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:35:04.661+0000 7f292c22e640 10 monclient: _renew_subs 2024-03-25T18:35:04.664 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:35:04.661+0000 7f292c22e640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.1:6790/0 2024-03-25T18:35:04.664 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:35:04.661+0000 7f292c22e640 1 -- 172.21.15.1:0/1469620024 --> v1:172.21.15.1:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f29241db440 con 0x7f29240b1b00 2024-03-25T18:35:04.664 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:35:04.661+0000 7f292c22e640 1 librados: init done 2024-03-25T18:35:04.664 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:35:04.662+0000 7f292c22e640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T18:35:04.664 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:35:04.662+0000 7f292c22e640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:35:04.665 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:35:04.662+0000 7f292c22e640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.1:6790/0 2024-03-25T18:35:04.665 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:35:04.662+0000 7f292c22e640 1 -- 172.21.15.1:0/1469620024 --> v1:172.21.15.1:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f292404f730 con 0x7f29240b1b00 2024-03-25T18:35:04.971 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:35:04.970+0000 7f291a7fc640 1 -- 172.21.15.1:0/1469620024 <== mon.2 v1:172.21.15.1:6790/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f2920001ca0 con 0x7f29240b1b00 2024-03-25T18:35:04.971 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:35:04.970+0000 7f291a7fc640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:35:04.971 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:35:04.970+0000 7f291a7fc640 1 -- 172.21.15.1:0/1469620024 <== mon.2 v1:172.21.15.1:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1586965025 0 0) 0x7f2920004a00 con 0x7f29240b1b00 2024-03-25T18:35:04.971 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:35:04.970+0000 7f291a7fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:35:04.971 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:35:04.970+0000 7f291a7fc640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-25T18:35:04.971 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:35:04.970+0000 7f291a7fc640 10 monclient: dump: 2024-03-25T18:35:04.972 INFO:teuthology.misc.health.smithi001.stderr:epoch 1 2024-03-25T18:35:04.972 INFO:teuthology.misc.health.smithi001.stderr:fsid c8ea72cf-9e99-4383-9b52-6c23d365fdfc 2024-03-25T18:35:04.972 INFO:teuthology.misc.health.smithi001.stderr:last_changed 2024-03-25T18:25:50.510519+0000 2024-03-25T18:35:04.972 INFO:teuthology.misc.health.smithi001.stderr:created 2024-03-25T18:25:50.510519+0000 2024-03-25T18:35:04.972 INFO:teuthology.misc.health.smithi001.stderr:min_mon_release 19 (squid) 2024-03-25T18:35:04.972 INFO:teuthology.misc.health.smithi001.stderr:election_strategy: 3 2024-03-25T18:35:04.972 INFO:teuthology.misc.health.smithi001.stderr:0: v1:172.21.15.1:6789/0 mon.a 2024-03-25T18:35:04.972 INFO:teuthology.misc.health.smithi001.stderr:1: v1:172.21.15.115:6789/0 mon.b 2024-03-25T18:35:04.972 INFO:teuthology.misc.health.smithi001.stderr:2: v1:172.21.15.1:6790/0 mon.c 2024-03-25T18:35:04.972 INFO:teuthology.misc.health.smithi001.stderr: 2024-03-25T18:35:04.972 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:35:04.970+0000 7f291bfff640 4 set_mon_vals no callback set 2024-03-25T18:35:04.973 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:35:04.971+0000 7f291a7fc640 1 -- 172.21.15.1:0/1469620024 <== mon.2 v1:172.21.15.1:6790/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 21787975 0 0) 0x7f292001a9d0 con 0x7f29240b1b00 2024-03-25T18:35:04.973 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:35:04.972+0000 7f29297c9640 10 monclient: get_auth_request con 0x7f290806dea0 auth_method 0 2024-03-25T18:35:04.973 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:35:04.972+0000 7f291a7fc640 10 client.4589.objecter ms_handle_connect 0x7f290806dea0 2024-03-25T18:35:04.978 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:35:04.977+0000 7f291a7fc640 1 -- 172.21.15.1:0/1469620024 <== mon.2 v1:172.21.15.1:6790/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 996254230 0 0) 0x7f2920054710 con 0x7f29240b1b00 2024-03-25T18:35:04.978 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:35:04.977+0000 7f291a7fc640 10 client.4589.objecter ms_dispatch 0x7f2924000b90 osd_map(10..10 src has 1..10) v4 2024-03-25T18:35:04.978 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:35:04.977+0000 7f291a7fc640 3 client.4589.objecter handle_osd_map got epochs [10,10] > 0 2024-03-25T18:35:04.978 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:35:04.977+0000 7f291a7fc640 3 client.4589.objecter handle_osd_map decoding full epoch 10 2024-03-25T18:35:04.979 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:35:04.977+0000 7f291a7fc640 20 client.4589.objecter dump_active .. 0 homeless 2024-03-25T18:35:04.979 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:35:04.977+0000 7f291a7fc640 1 -- 172.21.15.1:0/1469620024 <== mon.2 v1:172.21.15.1:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f2920050050 con 0x7f29240b1b00 2024-03-25T18:35:04.979 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:35:04.977+0000 7f291a7fc640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:35:04.979 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:35:04.977+0000 7f291a7fc640 10 monclient: _finish_command 1 = system:0 2024-03-25T18:35:05.201 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:35:05.200+0000 7f292c22e640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:35:05.201 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:35:05.200+0000 7f292c22e640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:35:05.202 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:35:05.200+0000 7f292c22e640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.1:6790/0 2024-03-25T18:35:05.202 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:35:05.200+0000 7f292c22e640 1 -- 172.21.15.1:0/1469620024 --> v1:172.21.15.1:6790/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f29240aa090 con 0x7f29240b1b00 2024-03-25T18:35:05.315 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:35:05.314+0000 7f291a7fc640 1 -- 172.21.15.1:0/1469620024 <== mon.2 v1:172.21.15.1:6790/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 4092213348) 0x7f2920052150 con 0x7f29240b1b00 2024-03-25T18:35:05.315 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:35:05.314+0000 7f291a7fc640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:35:05.315 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:35:05.314+0000 7f291a7fc640 10 monclient: _finish_command 2 = system:0 2024-03-25T18:35:05.315 INFO:teuthology.misc.health.smithi001.stdout: 2024-03-25T18:35:05.315 INFO:teuthology.misc.health.smithi001.stdout:{"epoch":10,"fsid":"c8ea72cf-9e99-4383-9b52-6c23d365fdfc","created":"2024-03-25T18:26:38.856638+0000","modified":"2024-03-25T18:26:44.224208+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T18:26:44.224208+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"b9f34922-3a84-49be-8426-807c1eceaefc","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"42bf306f-55e3-4634-9874-8b4b73ac6644","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"292cb23a-18a9-4207-a7cc-4d5727feaec0","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"45e0e9ab-898d-4673-9590-b4117bc5e84a","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"831a5a09-73ab-4a61-9872-088e76bdbc66","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"88cd903d-3b55-4840-9396-6e3cfe005282","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"89db7355-bc9d-46f4-8ab0-e8d9e744e026","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"cac47bee-0914-4f08-896f-348838882866","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T18:35:05.318 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:35:05.316+0000 7f292c22e640 10 librados: watch_flush enter 2024-03-25T18:35:05.319 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:35:05.317+0000 7f292c22e640 10 librados: watch_flush exit 2024-03-25T18:35:05.319 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:35:05.317+0000 7f292c22e640 20 client.4589.objecter shutdown clearing up homeless session... 2024-03-25T18:35:05.319 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:35:05.317+0000 7f292c22e640 10 client.4589.objecter successfully canceled tick 2024-03-25T18:35:05.319 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:35:05.317+0000 7f292c22e640 1 -- 172.21.15.1:0/1469620024 >> v1:172.21.15.115:6800/4237379176 conn(0x7f290806dea0 legacy=0x7f2908070360 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:35:05.319 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:35:05.317+0000 7f292c22e640 10 monclient: shutdown 2024-03-25T18:35:05.319 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:35:05.317+0000 7f292c22e640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:35:05.319 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:35:05.317+0000 7f292c22e640 1 -- 172.21.15.1:0/1469620024 >> v1:172.21.15.1:6790/0 conn(0x7f29240b1b00 legacy=0x7f2924141510 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:35:05.319 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:35:05.317+0000 7f292c22e640 1 -- 172.21.15.1:0/1469620024 shutdown_connections 2024-03-25T18:35:05.319 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:35:05.317+0000 7f292c22e640 1 -- 172.21.15.1:0/1469620024 >> 172.21.15.1:0/1469620024 conn(0x7f29240617a0 msgr2=0x7f29240b2710 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:35:05.319 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:35:05.317+0000 7f292c22e640 1 -- 172.21.15.1:0/1469620024 shutdown_connections 2024-03-25T18:35:05.319 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:35:05.317+0000 7f292c22e640 1 -- 172.21.15.1:0/1469620024 wait complete. 2024-03-25T18:35:05.320 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:35:05.317+0000 7f292c22e640 1 librados: shutdown 2024-03-25T18:35:05.328 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T18:35:12.135 DEBUG:teuthology.orchestra.run.smithi001:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T18:35:12.250 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:35:12.250+0000 7f869eb4d640 10 monclient: get_monmap_and_config 2024-03-25T18:35:12.250 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:35:12.250+0000 7f869eb4d640 10 monclient: build_initial_monmap 2024-03-25T18:35:12.251 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:35:12.250+0000 7f869eb4d640 10 monclient: monmap: 2024-03-25T18:35:12.251 INFO:teuthology.misc.health.smithi001.stderr:epoch 0 2024-03-25T18:35:12.251 INFO:teuthology.misc.health.smithi001.stderr:fsid c8ea72cf-9e99-4383-9b52-6c23d365fdfc 2024-03-25T18:35:12.251 INFO:teuthology.misc.health.smithi001.stderr:last_changed 2024-03-25T18:35:12.250774+0000 2024-03-25T18:35:12.251 INFO:teuthology.misc.health.smithi001.stderr:created 2024-03-25T18:35:12.250774+0000 2024-03-25T18:35:12.251 INFO:teuthology.misc.health.smithi001.stderr:min_mon_release 0 (unknown) 2024-03-25T18:35:12.251 INFO:teuthology.misc.health.smithi001.stderr:election_strategy: 3 2024-03-25T18:35:12.251 INFO:teuthology.misc.health.smithi001.stderr:0: v1:172.21.15.1:6789/0 mon.noname-a 2024-03-25T18:35:12.251 INFO:teuthology.misc.health.smithi001.stderr:1: v1:172.21.15.115:6789/0 mon.noname-c 2024-03-25T18:35:12.251 INFO:teuthology.misc.health.smithi001.stderr:2: v1:172.21.15.1:6790/0 mon.noname-b 2024-03-25T18:35:12.251 INFO:teuthology.misc.health.smithi001.stderr: 2024-03-25T18:35:12.252 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:35:12.250+0000 7f869eb4d640 1 Processor -- start 2024-03-25T18:35:12.252 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:35:12.251+0000 7f869eb4d640 1 -- start start 2024-03-25T18:35:12.252 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:35:12.251+0000 7f869eb4d640 10 monclient: init 2024-03-25T18:35:12.252 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:35:12.251+0000 7f869eb4d640 10 monclient: _reopen_session rank -1 2024-03-25T18:35:12.252 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:35:12.251+0000 7f869eb4d640 10 monclient: _add_conns ranks=[2,1,0] 2024-03-25T18:35:12.253 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:35:12.251+0000 7f869eb4d640 10 monclient(hunting): picked mon.noname-b con 0x7f86980b4e70 addr v1:172.21.15.1:6790/0 2024-03-25T18:35:12.253 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:35:12.251+0000 7f869eb4d640 10 monclient(hunting): picked mon.noname-c con 0x7f86980b44b0 addr v1:172.21.15.115:6789/0 2024-03-25T18:35:12.253 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:35:12.251+0000 7f869eb4d640 10 monclient(hunting): picked mon.noname-a con 0x7f86980b1b00 addr v1:172.21.15.1:6789/0 2024-03-25T18:35:12.253 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:35:12.251+0000 7f869eb4d640 1 -- --> v1:172.21.15.1:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f86980a96f0 con 0x7f86980b1b00 2024-03-25T18:35:12.253 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:35:12.251+0000 7f869eb4d640 1 -- --> v1:172.21.15.115:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f86980ab0e0 con 0x7f86980b44b0 2024-03-25T18:35:12.253 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:35:12.251+0000 7f869eb4d640 1 -- --> v1:172.21.15.1:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8698145f70 con 0x7f86980b4e70 2024-03-25T18:35:12.253 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:35:12.251+0000 7f869eb4d640 10 monclient(hunting): _renew_subs 2024-03-25T18:35:12.253 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:35:12.251+0000 7f869eb4d640 10 monclient(hunting): authenticate will time out at 7992.721399s 2024-03-25T18:35:12.259 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:35:12.259+0000 7f869c8e9640 1 --1- >> v1:172.21.15.1:6790/0 conn(0x7f86980b4e70 0x7f869813f940 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.1:6790/0 says I am v1:172.21.15.1:38628/0 (socket says 172.21.15.1:38628) 2024-03-25T18:35:12.260 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:35:12.259+0000 7f869c8e9640 1 -- 172.21.15.1:0/3216333857 learned_addr learned my addr 172.21.15.1:0/3216333857 (peer_addr_for_me v1:172.21.15.1:0/0) 2024-03-25T18:35:12.278 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:35:12.277+0000 7f86977fe640 1 -- 172.21.15.1:0/3216333857 <== mon.0 v1:172.21.15.1:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 491155088 0 0) 0x7f86980a96f0 con 0x7f86980b1b00 2024-03-25T18:35:12.278 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:35:12.277+0000 7f86977fe640 10 monclient(hunting): _negotiate 2024-03-25T18:35:12.278 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:35:12.277+0000 7f86977fe640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:35:12.278 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:35:12.277+0000 7f86977fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:35:12.278 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:35:12.277+0000 7f86977fe640 10 monclient(hunting): my global_id is 4584 2024-03-25T18:35:12.278 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:35:12.277+0000 7f86977fe640 1 -- 172.21.15.1:0/3216333857 --> v1:172.21.15.1:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f8688003520 con 0x7f86980b1b00 2024-03-25T18:35:12.282 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:35:12.281+0000 7f86977fe640 1 -- 172.21.15.1:0/3216333857 <== mon.2 v1:172.21.15.1:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3650938798 0 0) 0x7f8698145f70 con 0x7f86980b4e70 2024-03-25T18:35:12.282 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:35:12.281+0000 7f86977fe640 10 monclient(hunting): _negotiate 2024-03-25T18:35:12.282 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:35:12.281+0000 7f86977fe640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:35:12.282 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:35:12.281+0000 7f86977fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:35:12.283 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:35:12.281+0000 7f86977fe640 10 monclient(hunting): my global_id is 4592 2024-03-25T18:35:12.283 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:35:12.281+0000 7f86977fe640 1 -- 172.21.15.1:0/3216333857 --> v1:172.21.15.1:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f86980a96f0 con 0x7f86980b4e70 2024-03-25T18:35:12.295 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:35:12.294+0000 7f86977fe640 1 -- 172.21.15.1:0/3216333857 <== mon.1 v1:172.21.15.115:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3847140243 0 0) 0x7f86980ab0e0 con 0x7f86980b44b0 2024-03-25T18:35:12.295 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:35:12.294+0000 7f86977fe640 10 monclient(hunting): _negotiate 2024-03-25T18:35:12.295 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:35:12.294+0000 7f86977fe640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:35:12.295 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:35:12.294+0000 7f86977fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:35:12.295 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:35:12.294+0000 7f86977fe640 10 monclient(hunting): my global_id is 4591 2024-03-25T18:35:12.295 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:35:12.294+0000 7f86977fe640 1 -- 172.21.15.1:0/3216333857 --> v1:172.21.15.115:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f8698145f70 con 0x7f86980b44b0 2024-03-25T18:35:12.306 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:35:12.304+0000 7f86977fe640 1 -- 172.21.15.1:0/3216333857 <== mon.0 v1:172.21.15.1:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 4059205426 0 0) 0x7f8688003520 con 0x7f86980b1b00 2024-03-25T18:35:12.306 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:35:12.305+0000 7f86977fe640 1 -- 172.21.15.1:0/3216333857 >> v1:172.21.15.1:6790/0 conn(0x7f86980b4e70 legacy=0x7f869813f940 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:35:12.306 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:35:12.305+0000 7f86977fe640 1 -- 172.21.15.1:0/3216333857 >> v1:172.21.15.115:6789/0 conn(0x7f86980b44b0 legacy=0x7f8698145860 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:35:12.306 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:35:12.305+0000 7f86977fe640 10 monclient: _finish_hunting 0 2024-03-25T18:35:12.306 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:35:12.305+0000 7f86977fe640 1 monclient: found mon.noname-a 2024-03-25T18:35:12.306 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:35:12.305+0000 7f86977fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:35:12.306 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:35:12.305+0000 7f86977fe640 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.1:6789/0 2024-03-25T18:35:12.306 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:35:12.305+0000 7f86977fe640 1 -- 172.21.15.1:0/3216333857 --> v1:172.21.15.1:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f86980a9b80 con 0x7f86980b1b00 2024-03-25T18:35:12.306 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:35:12.305+0000 7f86977fe640 10 monclient: _finish_auth 0 2024-03-25T18:35:12.306 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:35:12.305+0000 7f86977fe640 10 monclient: _check_auth_tickets 2024-03-25T18:35:12.307 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:35:12.305+0000 7f86977fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:35:12.307 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:35:12.305+0000 7f86977fe640 1 -- 172.21.15.1:0/3216333857 <== mon.0 v1:172.21.15.1:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1586965025 0 0) 0x7f8690001a10 con 0x7f86980b1b00 2024-03-25T18:35:12.307 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:35:12.305+0000 7f86977fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:35:12.307 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:35:12.305+0000 7f86977fe640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-03-25T18:35:12.307 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:35:12.305+0000 7f86977fe640 10 monclient: dump: 2024-03-25T18:35:12.307 INFO:teuthology.misc.health.smithi001.stderr:epoch 1 2024-03-25T18:35:12.307 INFO:teuthology.misc.health.smithi001.stderr:fsid c8ea72cf-9e99-4383-9b52-6c23d365fdfc 2024-03-25T18:35:12.307 INFO:teuthology.misc.health.smithi001.stderr:last_changed 2024-03-25T18:25:50.510519+0000 2024-03-25T18:35:12.307 INFO:teuthology.misc.health.smithi001.stderr:created 2024-03-25T18:25:50.510519+0000 2024-03-25T18:35:12.307 INFO:teuthology.misc.health.smithi001.stderr:min_mon_release 19 (squid) 2024-03-25T18:35:12.307 INFO:teuthology.misc.health.smithi001.stderr:election_strategy: 3 2024-03-25T18:35:12.307 INFO:teuthology.misc.health.smithi001.stderr:0: v1:172.21.15.1:6789/0 mon.a 2024-03-25T18:35:12.307 INFO:teuthology.misc.health.smithi001.stderr:1: v1:172.21.15.115:6789/0 mon.b 2024-03-25T18:35:12.308 INFO:teuthology.misc.health.smithi001.stderr:2: v1:172.21.15.1:6790/0 mon.c 2024-03-25T18:35:12.308 INFO:teuthology.misc.health.smithi001.stderr: 2024-03-25T18:35:12.308 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:35:12.305+0000 7f869eb4d640 5 monclient: authenticate success, global_id 4584 2024-03-25T18:35:12.308 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:35:12.305+0000 7f869eb4d640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T18:35:12.326 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:35:12.325+0000 7f86977fe640 1 -- 172.21.15.1:0/3216333857 <== mon.0 v1:172.21.15.1:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f8690001b70 con 0x7f86980b1b00 2024-03-25T18:35:12.326 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:35:12.325+0000 7f86977fe640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:35:12.326 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:35:12.325+0000 7f86977fe640 1 -- 172.21.15.1:0/3216333857 <== mon.0 v1:172.21.15.1:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1586965025 0 0) 0x7f8690001f80 con 0x7f86980b1b00 2024-03-25T18:35:12.326 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:35:12.325+0000 7f86977fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:35:12.327 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:35:12.325+0000 7f86977fe640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-25T18:35:12.327 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:35:12.325+0000 7f86977fe640 10 monclient: dump: 2024-03-25T18:35:12.327 INFO:teuthology.misc.health.smithi001.stderr:epoch 1 2024-03-25T18:35:12.327 INFO:teuthology.misc.health.smithi001.stderr:fsid c8ea72cf-9e99-4383-9b52-6c23d365fdfc 2024-03-25T18:35:12.327 INFO:teuthology.misc.health.smithi001.stderr:last_changed 2024-03-25T18:25:50.510519+0000 2024-03-25T18:35:12.327 INFO:teuthology.misc.health.smithi001.stderr:created 2024-03-25T18:25:50.510519+0000 2024-03-25T18:35:12.327 INFO:teuthology.misc.health.smithi001.stderr:min_mon_release 19 (squid) 2024-03-25T18:35:12.327 INFO:teuthology.misc.health.smithi001.stderr:election_strategy: 3 2024-03-25T18:35:12.327 INFO:teuthology.misc.health.smithi001.stderr:0: v1:172.21.15.1:6789/0 mon.a 2024-03-25T18:35:12.327 INFO:teuthology.misc.health.smithi001.stderr:1: v1:172.21.15.115:6789/0 mon.b 2024-03-25T18:35:12.327 INFO:teuthology.misc.health.smithi001.stderr:2: v1:172.21.15.1:6790/0 mon.c 2024-03-25T18:35:12.327 INFO:teuthology.misc.health.smithi001.stderr: 2024-03-25T18:35:12.328 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:35:12.326+0000 7f869eb4d640 10 monclient: get_monmap_and_config success 2024-03-25T18:35:12.328 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:35:12.326+0000 7f869eb4d640 4 set_mon_vals no callback set 2024-03-25T18:35:12.329 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:35:12.327+0000 7f869eb4d640 10 monclient: shutdown 2024-03-25T18:35:12.329 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:35:12.327+0000 7f869eb4d640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:35:12.329 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:35:12.327+0000 7f869eb4d640 1 -- 172.21.15.1:0/3216333857 >> v1:172.21.15.1:6789/0 conn(0x7f86980b1b00 legacy=0x7f869814dc00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:35:12.329 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:35:12.327+0000 7f869eb4d640 1 -- 172.21.15.1:0/3216333857 shutdown_connections 2024-03-25T18:35:12.329 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:35:12.327+0000 7f869eb4d640 1 -- 172.21.15.1:0/3216333857 >> 172.21.15.1:0/3216333857 conn(0x7f86980617a0 msgr2=0x7f8698061bb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:35:12.329 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:35:12.327+0000 7f869eb4d640 1 -- 172.21.15.1:0/3216333857 shutdown_connections 2024-03-25T18:35:12.329 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:35:12.327+0000 7f869eb4d640 1 -- 172.21.15.1:0/3216333857 wait complete. 2024-03-25T18:35:12.329 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:35:12.328+0000 7f869eb4d640 10 monclient: build_initial_monmap 2024-03-25T18:35:12.329 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:35:12.328+0000 7f869eb4d640 10 monclient: monmap: 2024-03-25T18:35:12.329 INFO:teuthology.misc.health.smithi001.stderr:epoch 0 2024-03-25T18:35:12.329 INFO:teuthology.misc.health.smithi001.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T18:35:12.329 INFO:teuthology.misc.health.smithi001.stderr:last_changed 0.000000 2024-03-25T18:35:12.329 INFO:teuthology.misc.health.smithi001.stderr:created 0.000000 2024-03-25T18:35:12.330 INFO:teuthology.misc.health.smithi001.stderr:min_mon_release 0 (unknown) 2024-03-25T18:35:12.330 INFO:teuthology.misc.health.smithi001.stderr:election_strategy: 1 2024-03-25T18:35:12.330 INFO:teuthology.misc.health.smithi001.stderr:0: v1:172.21.15.1:6789/0 mon.noname-a 2024-03-25T18:35:12.330 INFO:teuthology.misc.health.smithi001.stderr:1: v1:172.21.15.115:6789/0 mon.noname-b 2024-03-25T18:35:12.330 INFO:teuthology.misc.health.smithi001.stderr:2: v1:172.21.15.1:6790/0 mon.noname-c 2024-03-25T18:35:12.330 INFO:teuthology.misc.health.smithi001.stderr: 2024-03-25T18:35:12.331 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:35:12.328+0000 7f869eb4d640 1 librados: starting msgr at 2024-03-25T18:35:12.332 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:35:12.328+0000 7f869eb4d640 1 librados: starting objecter 2024-03-25T18:35:12.332 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:35:12.328+0000 7f869eb4d640 1 Processor -- start 2024-03-25T18:35:12.332 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:35:12.329+0000 7f869eb4d640 1 -- start start 2024-03-25T18:35:12.332 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:35:12.329+0000 7f869eb4d640 1 librados: setting wanted keys 2024-03-25T18:35:12.332 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:35:12.329+0000 7f869eb4d640 1 librados: calling monclient init 2024-03-25T18:35:12.332 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:35:12.329+0000 7f869eb4d640 10 monclient: init 2024-03-25T18:35:12.332 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:35:12.329+0000 7f869eb4d640 10 monclient: _reopen_session rank -1 2024-03-25T18:35:12.332 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:35:12.329+0000 7f869eb4d640 10 monclient: _add_conns ranks=[0,2,1] 2024-03-25T18:35:12.332 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:35:12.329+0000 7f869eb4d640 10 monclient(hunting): picked mon.noname-a con 0x7f86980b1b00 addr v1:172.21.15.1:6789/0 2024-03-25T18:35:12.332 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:35:12.329+0000 7f869eb4d640 10 monclient(hunting): picked mon.noname-c con 0x7f86980b44b0 addr v1:172.21.15.1:6790/0 2024-03-25T18:35:12.332 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:35:12.330+0000 7f869eb4d640 10 monclient(hunting): picked mon.noname-b con 0x7f86980b4e70 addr v1:172.21.15.115:6789/0 2024-03-25T18:35:12.332 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:35:12.330+0000 7f869eb4d640 1 -- --> v1:172.21.15.1:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8698142d40 con 0x7f86980b1b00 2024-03-25T18:35:12.333 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:35:12.330+0000 7f869eb4d640 1 -- --> v1:172.21.15.115:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f86981438e0 con 0x7f86980b4e70 2024-03-25T18:35:12.333 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:35:12.330+0000 7f869eb4d640 1 -- --> v1:172.21.15.1:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8698131920 con 0x7f86980b44b0 2024-03-25T18:35:12.333 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:35:12.330+0000 7f869eb4d640 10 monclient(hunting): _renew_subs 2024-03-25T18:35:12.333 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:35:12.330+0000 7f869eb4d640 10 monclient(hunting): authenticate will time out at 7992.800089s 2024-03-25T18:35:12.348 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:35:12.347+0000 7f8697fff640 1 --1- >> v1:172.21.15.1:6790/0 conn(0x7f86980b44b0 0x7f8698130b00 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.1:6790/0 says I am v1:172.21.15.1:43882/0 (socket says 172.21.15.1:43882) 2024-03-25T18:35:12.348 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:35:12.347+0000 7f8697fff640 1 -- 172.21.15.1:0/1560800607 learned_addr learned my addr 172.21.15.1:0/1560800607 (peer_addr_for_me v1:172.21.15.1:0/0) 2024-03-25T18:35:12.351 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:35:12.350+0000 7f86957fa640 10 client.?.objecter ms_handle_connect 0x7f86980b44b0 2024-03-25T18:35:12.351 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:35:12.350+0000 7f86957fa640 10 client.?.objecter resend_mon_ops 2024-03-25T18:35:12.351 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:35:12.351+0000 7f86957fa640 10 client.?.objecter ms_handle_connect 0x7f86980b1b00 2024-03-25T18:35:12.352 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:35:12.351+0000 7f86957fa640 10 client.?.objecter resend_mon_ops 2024-03-25T18:35:12.363 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:35:12.362+0000 7f86957fa640 1 -- 172.21.15.1:0/1560800607 <== mon.0 v1:172.21.15.1:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1179452747 0 0) 0x7f8698142d40 con 0x7f86980b1b00 2024-03-25T18:35:12.363 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:35:12.362+0000 7f86957fa640 10 monclient(hunting): _negotiate 2024-03-25T18:35:12.363 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:35:12.362+0000 7f86957fa640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:35:12.363 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:35:12.362+0000 7f86957fa640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:35:12.363 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:35:12.362+0000 7f86957fa640 10 monclient(hunting): my global_id is 4587 2024-03-25T18:35:12.363 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:35:12.362+0000 7f86957fa640 1 -- 172.21.15.1:0/1560800607 --> v1:172.21.15.1:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f867c004eb0 con 0x7f86980b1b00 2024-03-25T18:35:12.369 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:35:12.368+0000 7f86957fa640 1 -- 172.21.15.1:0/1560800607 <== mon.2 v1:172.21.15.1:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1081534255 0 0) 0x7f8698131920 con 0x7f86980b44b0 2024-03-25T18:35:12.369 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:35:12.368+0000 7f86957fa640 10 monclient(hunting): _negotiate 2024-03-25T18:35:12.369 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:35:12.368+0000 7f86957fa640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:35:12.369 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:35:12.368+0000 7f86957fa640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:35:12.369 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:35:12.368+0000 7f86957fa640 10 monclient(hunting): my global_id is 4595 2024-03-25T18:35:12.369 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:35:12.368+0000 7f86957fa640 1 -- 172.21.15.1:0/1560800607 --> v1:172.21.15.1:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f8698142d40 con 0x7f86980b44b0 2024-03-25T18:35:12.370 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:35:12.369+0000 7f86957fa640 10 client.?.objecter ms_handle_connect 0x7f86980b4e70 2024-03-25T18:35:12.370 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:35:12.369+0000 7f86957fa640 10 client.?.objecter resend_mon_ops 2024-03-25T18:35:12.381 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:35:12.380+0000 7f86957fa640 1 -- 172.21.15.1:0/1560800607 <== mon.1 v1:172.21.15.115:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4021979246 0 0) 0x7f86981438e0 con 0x7f86980b4e70 2024-03-25T18:35:12.381 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:35:12.380+0000 7f86957fa640 10 monclient(hunting): _negotiate 2024-03-25T18:35:12.382 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:35:12.380+0000 7f86957fa640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:35:12.382 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:35:12.380+0000 7f86957fa640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:35:12.382 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:35:12.380+0000 7f86957fa640 10 monclient(hunting): my global_id is 4594 2024-03-25T18:35:12.382 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:35:12.380+0000 7f86957fa640 1 -- 172.21.15.1:0/1560800607 --> v1:172.21.15.115:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f8698131920 con 0x7f86980b4e70 2024-03-25T18:35:12.382 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:35:12.380+0000 7f86957fa640 1 -- 172.21.15.1:0/1560800607 <== mon.0 v1:172.21.15.1:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3492625133 0 0) 0x7f867c004eb0 con 0x7f86980b1b00 2024-03-25T18:35:12.382 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:35:12.380+0000 7f86957fa640 1 -- 172.21.15.1:0/1560800607 --> v1:172.21.15.1:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f86981438e0 con 0x7f86980b1b00 2024-03-25T18:35:12.382 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:35:12.380+0000 7f86957fa640 1 -- 172.21.15.1:0/1560800607 <== mon.0 v1:172.21.15.1:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1586965025 0 0) 0x7f868c001910 con 0x7f86980b1b00 2024-03-25T18:35:12.382 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:35:12.380+0000 7f86957fa640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:35:12.382 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:35:12.380+0000 7f86957fa640 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2024-03-25T18:35:12.382 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:35:12.380+0000 7f86957fa640 10 monclient(hunting): dump: 2024-03-25T18:35:12.382 INFO:teuthology.misc.health.smithi001.stderr:epoch 1 2024-03-25T18:35:12.382 INFO:teuthology.misc.health.smithi001.stderr:fsid c8ea72cf-9e99-4383-9b52-6c23d365fdfc 2024-03-25T18:35:12.382 INFO:teuthology.misc.health.smithi001.stderr:last_changed 2024-03-25T18:25:50.510519+0000 2024-03-25T18:35:12.382 INFO:teuthology.misc.health.smithi001.stderr:created 2024-03-25T18:25:50.510519+0000 2024-03-25T18:35:12.382 INFO:teuthology.misc.health.smithi001.stderr:min_mon_release 19 (squid) 2024-03-25T18:35:12.383 INFO:teuthology.misc.health.smithi001.stderr:election_strategy: 3 2024-03-25T18:35:12.383 INFO:teuthology.misc.health.smithi001.stderr:0: v1:172.21.15.1:6789/0 mon.a 2024-03-25T18:35:12.383 INFO:teuthology.misc.health.smithi001.stderr:1: v1:172.21.15.115:6789/0 mon.b 2024-03-25T18:35:12.383 INFO:teuthology.misc.health.smithi001.stderr:2: v1:172.21.15.1:6790/0 mon.c 2024-03-25T18:35:12.383 INFO:teuthology.misc.health.smithi001.stderr: 2024-03-25T18:35:12.383 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:35:12.380+0000 7f86957fa640 10 monclient(hunting): _finish_auth 0 2024-03-25T18:35:12.392 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:35:12.390+0000 7f86957fa640 1 -- 172.21.15.1:0/1560800607 <== mon.0 v1:172.21.15.1:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 580593903 0 0) 0x7f86981438e0 con 0x7f86980b1b00 2024-03-25T18:35:12.392 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:35:12.390+0000 7f86957fa640 1 -- 172.21.15.1:0/1560800607 >> v1:172.21.15.1:6790/0 conn(0x7f86980b44b0 legacy=0x7f8698130b00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:35:12.392 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:35:12.390+0000 7f86957fa640 1 -- 172.21.15.1:0/1560800607 >> v1:172.21.15.115:6789/0 conn(0x7f86980b4e70 legacy=0x7f8698131210 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:35:12.392 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:35:12.391+0000 7f86957fa640 10 monclient: _finish_hunting 0 2024-03-25T18:35:12.392 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:35:12.391+0000 7f86957fa640 1 monclient: found mon.a 2024-03-25T18:35:12.392 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:35:12.391+0000 7f86957fa640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:35:12.392 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:35:12.391+0000 7f86957fa640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.1:6789/0 2024-03-25T18:35:12.392 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:35:12.391+0000 7f86957fa640 1 -- 172.21.15.1:0/1560800607 --> v1:172.21.15.1:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8698144340 con 0x7f86980b1b00 2024-03-25T18:35:12.392 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:35:12.391+0000 7f86957fa640 10 monclient: _finish_auth 0 2024-03-25T18:35:12.392 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:35:12.391+0000 7f86957fa640 10 monclient: _check_auth_tickets 2024-03-25T18:35:12.392 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:35:12.391+0000 7f86957fa640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:35:12.393 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:35:12.391+0000 7f869eb4d640 5 monclient: authenticate success, global_id 4587 2024-03-25T18:35:12.393 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:35:12.391+0000 7f869eb4d640 10 monclient: _renew_subs 2024-03-25T18:35:12.393 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:35:12.391+0000 7f869eb4d640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.1:6789/0 2024-03-25T18:35:12.393 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:35:12.391+0000 7f869eb4d640 1 -- 172.21.15.1:0/1560800607 --> v1:172.21.15.1:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8698144680 con 0x7f86980b1b00 2024-03-25T18:35:12.393 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:35:12.391+0000 7f869eb4d640 10 client.4587.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T18:35:12.393 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:35:12.391+0000 7f869eb4d640 10 monclient: _renew_subs 2024-03-25T18:35:12.393 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:35:12.391+0000 7f869eb4d640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.1:6789/0 2024-03-25T18:35:12.393 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:35:12.391+0000 7f869eb4d640 1 -- 172.21.15.1:0/1560800607 --> v1:172.21.15.1:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f8698142960 con 0x7f86980b1b00 2024-03-25T18:35:12.393 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:35:12.391+0000 7f869eb4d640 1 librados: init done 2024-03-25T18:35:12.394 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:35:12.392+0000 7f869eb4d640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T18:35:12.394 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:35:12.392+0000 7f869eb4d640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:35:12.394 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:35:12.392+0000 7f869eb4d640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.1:6789/0 2024-03-25T18:35:12.394 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:35:12.392+0000 7f869eb4d640 1 -- 172.21.15.1:0/1560800607 --> v1:172.21.15.1:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f869804f730 con 0x7f86980b1b00 2024-03-25T18:35:12.616 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:35:12.616+0000 7f86957fa640 1 -- 172.21.15.1:0/1560800607 <== mon.0 v1:172.21.15.1:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f868c0047e0 con 0x7f86980b1b00 2024-03-25T18:35:12.617 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:35:12.616+0000 7f86957fa640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:35:12.617 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:35:12.616+0000 7f8696ffd640 4 set_mon_vals no callback set 2024-03-25T18:35:12.724 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:35:12.723+0000 7f86957fa640 1 -- 172.21.15.1:0/1560800607 <== mon.0 v1:172.21.15.1:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1586965025 0 0) 0x7f868c004af0 con 0x7f86980b1b00 2024-03-25T18:35:12.724 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:35:12.723+0000 7f86957fa640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:35:12.724 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:35:12.723+0000 7f86957fa640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-25T18:35:12.724 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:35:12.723+0000 7f86957fa640 10 monclient: dump: 2024-03-25T18:35:12.724 INFO:teuthology.misc.health.smithi001.stderr:epoch 1 2024-03-25T18:35:12.724 INFO:teuthology.misc.health.smithi001.stderr:fsid c8ea72cf-9e99-4383-9b52-6c23d365fdfc 2024-03-25T18:35:12.724 INFO:teuthology.misc.health.smithi001.stderr:last_changed 2024-03-25T18:25:50.510519+0000 2024-03-25T18:35:12.724 INFO:teuthology.misc.health.smithi001.stderr:created 2024-03-25T18:25:50.510519+0000 2024-03-25T18:35:12.725 INFO:teuthology.misc.health.smithi001.stderr:min_mon_release 19 (squid) 2024-03-25T18:35:12.725 INFO:teuthology.misc.health.smithi001.stderr:election_strategy: 3 2024-03-25T18:35:12.725 INFO:teuthology.misc.health.smithi001.stderr:0: v1:172.21.15.1:6789/0 mon.a 2024-03-25T18:35:12.725 INFO:teuthology.misc.health.smithi001.stderr:1: v1:172.21.15.115:6789/0 mon.b 2024-03-25T18:35:12.725 INFO:teuthology.misc.health.smithi001.stderr:2: v1:172.21.15.1:6790/0 mon.c 2024-03-25T18:35:12.725 INFO:teuthology.misc.health.smithi001.stderr: 2024-03-25T18:35:12.726 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:35:12.724+0000 7f86957fa640 1 -- 172.21.15.1:0/1560800607 <== mon.0 v1:172.21.15.1:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 21787975 0 0) 0x7f868c01aac0 con 0x7f86980b1b00 2024-03-25T18:35:12.726 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:35:12.725+0000 7f8697fff640 10 monclient: get_auth_request con 0x7f867c06dda0 auth_method 0 2024-03-25T18:35:12.726 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:35:12.725+0000 7f86957fa640 10 client.4587.objecter ms_handle_connect 0x7f867c06dda0 2024-03-25T18:35:12.734 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:35:12.733+0000 7f86957fa640 1 -- 172.21.15.1:0/1560800607 <== mon.0 v1:172.21.15.1:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 996254230 0 0) 0x7f868c0547c0 con 0x7f86980b1b00 2024-03-25T18:35:12.734 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:35:12.733+0000 7f86957fa640 10 client.4587.objecter ms_dispatch 0x7f8698000b90 osd_map(10..10 src has 1..10) v4 2024-03-25T18:35:12.734 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:35:12.733+0000 7f86957fa640 3 client.4587.objecter handle_osd_map got epochs [10,10] > 0 2024-03-25T18:35:12.734 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:35:12.733+0000 7f86957fa640 3 client.4587.objecter handle_osd_map decoding full epoch 10 2024-03-25T18:35:12.734 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:35:12.733+0000 7f86957fa640 20 client.4587.objecter dump_active .. 0 homeless 2024-03-25T18:35:12.734 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:35:12.733+0000 7f86957fa640 1 -- 172.21.15.1:0/1560800607 <== mon.0 v1:172.21.15.1:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f868c050100 con 0x7f86980b1b00 2024-03-25T18:35:12.734 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:35:12.733+0000 7f86957fa640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:35:12.735 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:35:12.733+0000 7f86957fa640 10 monclient: _finish_command 1 = system:0 2024-03-25T18:35:12.979 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:35:12.978+0000 7f869eb4d640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:35:12.979 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:35:12.978+0000 7f869eb4d640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:35:12.979 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:35:12.978+0000 7f869eb4d640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.1:6789/0 2024-03-25T18:35:12.979 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:35:12.978+0000 7f869eb4d640 1 -- 172.21.15.1:0/1560800607 --> v1:172.21.15.1:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f8698144860 con 0x7f86980b1b00 2024-03-25T18:35:13.015 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:35:13.014+0000 7f86957fa640 1 -- 172.21.15.1:0/1560800607 <== mon.0 v1:172.21.15.1:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 4092213348) 0x7f868c052200 con 0x7f86980b1b00 2024-03-25T18:35:13.015 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:35:13.014+0000 7f86957fa640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:35:13.015 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:35:13.014+0000 7f86957fa640 10 monclient: _finish_command 2 = system:0 2024-03-25T18:35:13.015 INFO:teuthology.misc.health.smithi001.stdout: 2024-03-25T18:35:13.016 INFO:teuthology.misc.health.smithi001.stdout:{"epoch":10,"fsid":"c8ea72cf-9e99-4383-9b52-6c23d365fdfc","created":"2024-03-25T18:26:38.856638+0000","modified":"2024-03-25T18:26:44.224208+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T18:26:44.224208+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"b9f34922-3a84-49be-8426-807c1eceaefc","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"42bf306f-55e3-4634-9874-8b4b73ac6644","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"292cb23a-18a9-4207-a7cc-4d5727feaec0","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"45e0e9ab-898d-4673-9590-b4117bc5e84a","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"831a5a09-73ab-4a61-9872-088e76bdbc66","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"88cd903d-3b55-4840-9396-6e3cfe005282","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"89db7355-bc9d-46f4-8ab0-e8d9e744e026","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"cac47bee-0914-4f08-896f-348838882866","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T18:35:13.018 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:35:13.016+0000 7f869eb4d640 10 librados: watch_flush enter 2024-03-25T18:35:13.018 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:35:13.016+0000 7f869eb4d640 10 librados: watch_flush exit 2024-03-25T18:35:13.018 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:35:13.016+0000 7f869eb4d640 20 client.4587.objecter shutdown clearing up homeless session... 2024-03-25T18:35:13.018 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:35:13.016+0000 7f869eb4d640 10 client.4587.objecter successfully canceled tick 2024-03-25T18:35:13.018 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:35:13.016+0000 7f869eb4d640 1 -- 172.21.15.1:0/1560800607 >> v1:172.21.15.115:6800/4237379176 conn(0x7f867c06dda0 legacy=0x7f867c070260 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:35:13.018 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:35:13.016+0000 7f869eb4d640 10 monclient: shutdown 2024-03-25T18:35:13.018 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:35:13.016+0000 7f869eb4d640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:35:13.018 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:35:13.016+0000 7f869eb4d640 1 -- 172.21.15.1:0/1560800607 >> v1:172.21.15.1:6789/0 conn(0x7f86980b1b00 legacy=0x7f86981431d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:35:13.019 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:35:13.016+0000 7f869eb4d640 1 -- 172.21.15.1:0/1560800607 shutdown_connections 2024-03-25T18:35:13.019 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:35:13.016+0000 7f869eb4d640 1 -- 172.21.15.1:0/1560800607 >> 172.21.15.1:0/1560800607 conn(0x7f86980617a0 msgr2=0x7f86980b2710 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:35:13.019 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:35:13.017+0000 7f869eb4d640 1 -- 172.21.15.1:0/1560800607 shutdown_connections 2024-03-25T18:35:13.019 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:35:13.017+0000 7f869eb4d640 1 -- 172.21.15.1:0/1560800607 wait complete. 2024-03-25T18:35:13.019 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:35:13.017+0000 7f869eb4d640 1 librados: shutdown 2024-03-25T18:35:13.029 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T18:35:19.834 DEBUG:teuthology.orchestra.run.smithi001:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T18:35:19.937 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:35:19.936+0000 7ff8b3c0f640 10 monclient: get_monmap_and_config 2024-03-25T18:35:19.937 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:35:19.936+0000 7ff8b3c0f640 10 monclient: build_initial_monmap 2024-03-25T18:35:19.937 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:35:19.936+0000 7ff8b3c0f640 10 monclient: monmap: 2024-03-25T18:35:19.937 INFO:teuthology.misc.health.smithi001.stderr:epoch 0 2024-03-25T18:35:19.937 INFO:teuthology.misc.health.smithi001.stderr:fsid c8ea72cf-9e99-4383-9b52-6c23d365fdfc 2024-03-25T18:35:19.937 INFO:teuthology.misc.health.smithi001.stderr:last_changed 2024-03-25T18:35:19.936966+0000 2024-03-25T18:35:19.937 INFO:teuthology.misc.health.smithi001.stderr:created 2024-03-25T18:35:19.936966+0000 2024-03-25T18:35:19.937 INFO:teuthology.misc.health.smithi001.stderr:min_mon_release 0 (unknown) 2024-03-25T18:35:19.937 INFO:teuthology.misc.health.smithi001.stderr:election_strategy: 3 2024-03-25T18:35:19.937 INFO:teuthology.misc.health.smithi001.stderr:0: v1:172.21.15.1:6789/0 mon.noname-a 2024-03-25T18:35:19.937 INFO:teuthology.misc.health.smithi001.stderr:1: v1:172.21.15.115:6789/0 mon.noname-c 2024-03-25T18:35:19.938 INFO:teuthology.misc.health.smithi001.stderr:2: v1:172.21.15.1:6790/0 mon.noname-b 2024-03-25T18:35:19.938 INFO:teuthology.misc.health.smithi001.stderr: 2024-03-25T18:35:19.939 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:35:19.936+0000 7ff8b3c0f640 1 Processor -- start 2024-03-25T18:35:19.939 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:35:19.937+0000 7ff8b3c0f640 1 -- start start 2024-03-25T18:35:19.939 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:35:19.937+0000 7ff8b3c0f640 10 monclient: init 2024-03-25T18:35:19.939 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:35:19.937+0000 7ff8b3c0f640 10 monclient: _reopen_session rank -1 2024-03-25T18:35:19.939 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:35:19.937+0000 7ff8b3c0f640 10 monclient: _add_conns ranks=[2,0,1] 2024-03-25T18:35:19.939 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:35:19.937+0000 7ff8b3c0f640 10 monclient(hunting): picked mon.noname-b con 0x7ff8ac0b4e70 addr v1:172.21.15.1:6790/0 2024-03-25T18:35:19.939 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:35:19.937+0000 7ff8b3c0f640 10 monclient(hunting): picked mon.noname-a con 0x7ff8ac0b44b0 addr v1:172.21.15.1:6789/0 2024-03-25T18:35:19.939 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:35:19.937+0000 7ff8b3c0f640 10 monclient(hunting): picked mon.noname-c con 0x7ff8ac0b1b00 addr v1:172.21.15.115:6789/0 2024-03-25T18:35:19.939 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:35:19.937+0000 7ff8b3c0f640 1 -- --> v1:172.21.15.1:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff8ac0a96f0 con 0x7ff8ac0b44b0 2024-03-25T18:35:19.939 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:35:19.937+0000 7ff8b3c0f640 1 -- --> v1:172.21.15.115:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff8ac0ab0e0 con 0x7ff8ac0b1b00 2024-03-25T18:35:19.940 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:35:19.937+0000 7ff8b3c0f640 1 -- --> v1:172.21.15.1:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff8ac145f70 con 0x7ff8ac0b4e70 2024-03-25T18:35:19.940 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:35:19.937+0000 7ff8b3c0f640 10 monclient(hunting): _renew_subs 2024-03-25T18:35:19.940 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:35:19.937+0000 7ff8b3c0f640 10 monclient(hunting): authenticate will time out at 8000.407446s 2024-03-25T18:35:19.946 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:35:19.945+0000 7ff8b21ac640 1 --1- >> v1:172.21.15.115:6789/0 conn(0x7ff8ac0b1b00 0x7ff8ac14dc00 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.115:6789/0 says I am v1:172.21.15.1:35668/0 (socket says 172.21.15.1:35668) 2024-03-25T18:35:19.946 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:35:19.945+0000 7ff8b21ac640 1 -- 172.21.15.1:0/1287188621 learned_addr learned my addr 172.21.15.1:0/1287188621 (peer_addr_for_me v1:172.21.15.1:0/0) 2024-03-25T18:35:19.968 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:35:19.968+0000 7ff8b09a9640 1 -- 172.21.15.1:0/1287188621 <== mon.1 v1:172.21.15.115:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1523112453 0 0) 0x7ff8ac0ab0e0 con 0x7ff8ac0b1b00 2024-03-25T18:35:19.969 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:35:19.968+0000 7ff8b09a9640 10 monclient(hunting): _negotiate 2024-03-25T18:35:19.969 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:35:19.968+0000 7ff8b09a9640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:35:19.969 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:35:19.968+0000 7ff8b09a9640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:35:19.969 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:35:19.968+0000 7ff8b09a9640 10 monclient(hunting): my global_id is 4597 2024-03-25T18:35:19.969 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:35:19.968+0000 7ff8b09a9640 1 -- 172.21.15.1:0/1287188621 --> v1:172.21.15.115:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ff898003520 con 0x7ff8ac0b1b00 2024-03-25T18:35:19.975 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:35:19.974+0000 7ff8b09a9640 1 -- 172.21.15.1:0/1287188621 <== mon.2 v1:172.21.15.1:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3192361488 0 0) 0x7ff8ac145f70 con 0x7ff8ac0b4e70 2024-03-25T18:35:19.975 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:35:19.974+0000 7ff8b09a9640 10 monclient(hunting): _negotiate 2024-03-25T18:35:19.975 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:35:19.974+0000 7ff8b09a9640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:35:19.975 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:35:19.974+0000 7ff8b09a9640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:35:19.975 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:35:19.974+0000 7ff8b09a9640 10 monclient(hunting): my global_id is 4598 2024-03-25T18:35:19.975 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:35:19.974+0000 7ff8b09a9640 1 -- 172.21.15.1:0/1287188621 --> v1:172.21.15.1:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ff8ac0ab0e0 con 0x7ff8ac0b4e70 2024-03-25T18:35:19.991 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:35:19.990+0000 7ff8b09a9640 1 -- 172.21.15.1:0/1287188621 <== mon.0 v1:172.21.15.1:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3487114942 0 0) 0x7ff8ac0a96f0 con 0x7ff8ac0b44b0 2024-03-25T18:35:19.991 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:35:19.990+0000 7ff8b09a9640 10 monclient(hunting): _negotiate 2024-03-25T18:35:19.991 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:35:19.990+0000 7ff8b09a9640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:35:19.991 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:35:19.990+0000 7ff8b09a9640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:35:19.991 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:35:19.990+0000 7ff8b09a9640 10 monclient(hunting): my global_id is 4590 2024-03-25T18:35:19.991 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:35:19.990+0000 7ff8b09a9640 1 -- 172.21.15.1:0/1287188621 --> v1:172.21.15.1:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ff8ac145f70 con 0x7ff8ac0b44b0 2024-03-25T18:35:19.996 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:35:19.995+0000 7ff8b09a9640 1 -- 172.21.15.1:0/1287188621 <== mon.1 v1:172.21.15.115:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 170557596 0 0) 0x7ff898003520 con 0x7ff8ac0b1b00 2024-03-25T18:35:19.996 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:35:19.995+0000 7ff8b09a9640 1 -- 172.21.15.1:0/1287188621 >> v1:172.21.15.1:6790/0 conn(0x7ff8ac0b4e70 legacy=0x7ff8ac13f940 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:35:19.996 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:35:19.995+0000 7ff8b09a9640 1 -- 172.21.15.1:0/1287188621 >> v1:172.21.15.1:6789/0 conn(0x7ff8ac0b44b0 legacy=0x7ff8ac145860 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:35:19.996 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:35:19.995+0000 7ff8b09a9640 10 monclient: _finish_hunting 0 2024-03-25T18:35:19.997 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:35:19.995+0000 7ff8b09a9640 1 monclient: found mon.noname-c 2024-03-25T18:35:19.997 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:35:19.995+0000 7ff8b09a9640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:35:19.997 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:35:19.995+0000 7ff8b09a9640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.115:6789/0 2024-03-25T18:35:19.997 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:35:19.995+0000 7ff8b09a9640 1 -- 172.21.15.1:0/1287188621 --> v1:172.21.15.115:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff8ac0a9b80 con 0x7ff8ac0b1b00 2024-03-25T18:35:19.997 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:35:19.995+0000 7ff8b09a9640 10 monclient: _finish_auth 0 2024-03-25T18:35:19.997 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:35:19.995+0000 7ff8b09a9640 10 monclient: _check_auth_tickets 2024-03-25T18:35:19.997 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:35:19.995+0000 7ff8b09a9640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:35:19.997 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:35:19.995+0000 7ff8b09a9640 1 -- 172.21.15.1:0/1287188621 <== mon.1 v1:172.21.15.115:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1586965025 0 0) 0x7ff8a4001a10 con 0x7ff8ac0b1b00 2024-03-25T18:35:19.997 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:35:19.995+0000 7ff8b09a9640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:35:19.997 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:35:19.995+0000 7ff8b09a9640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-03-25T18:35:19.997 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:35:19.995+0000 7ff8b09a9640 10 monclient: dump: 2024-03-25T18:35:19.997 INFO:teuthology.misc.health.smithi001.stderr:epoch 1 2024-03-25T18:35:19.997 INFO:teuthology.misc.health.smithi001.stderr:fsid c8ea72cf-9e99-4383-9b52-6c23d365fdfc 2024-03-25T18:35:19.997 INFO:teuthology.misc.health.smithi001.stderr:last_changed 2024-03-25T18:25:50.510519+0000 2024-03-25T18:35:19.997 INFO:teuthology.misc.health.smithi001.stderr:created 2024-03-25T18:25:50.510519+0000 2024-03-25T18:35:19.998 INFO:teuthology.misc.health.smithi001.stderr:min_mon_release 19 (squid) 2024-03-25T18:35:19.998 INFO:teuthology.misc.health.smithi001.stderr:election_strategy: 3 2024-03-25T18:35:19.998 INFO:teuthology.misc.health.smithi001.stderr:0: v1:172.21.15.1:6789/0 mon.a 2024-03-25T18:35:19.998 INFO:teuthology.misc.health.smithi001.stderr:1: v1:172.21.15.115:6789/0 mon.b 2024-03-25T18:35:19.998 INFO:teuthology.misc.health.smithi001.stderr:2: v1:172.21.15.1:6790/0 mon.c 2024-03-25T18:35:19.998 INFO:teuthology.misc.health.smithi001.stderr: 2024-03-25T18:35:19.998 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:35:19.995+0000 7ff8b3c0f640 5 monclient: authenticate success, global_id 4597 2024-03-25T18:35:19.998 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:35:19.995+0000 7ff8b3c0f640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T18:35:20.021 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:35:20.020+0000 7ff8b09a9640 1 -- 172.21.15.1:0/1287188621 <== mon.1 v1:172.21.15.115:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7ff8a4001b70 con 0x7ff8ac0b1b00 2024-03-25T18:35:20.021 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:35:20.020+0000 7ff8b09a9640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:35:20.021 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:35:20.020+0000 7ff8b09a9640 1 -- 172.21.15.1:0/1287188621 <== mon.1 v1:172.21.15.115:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1586965025 0 0) 0x7ff8a4001f80 con 0x7ff8ac0b1b00 2024-03-25T18:35:20.021 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:35:20.020+0000 7ff8b09a9640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:35:20.021 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:35:20.020+0000 7ff8b09a9640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-25T18:35:20.021 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:35:20.020+0000 7ff8b09a9640 10 monclient: dump: 2024-03-25T18:35:20.021 INFO:teuthology.misc.health.smithi001.stderr:epoch 1 2024-03-25T18:35:20.021 INFO:teuthology.misc.health.smithi001.stderr:fsid c8ea72cf-9e99-4383-9b52-6c23d365fdfc 2024-03-25T18:35:20.021 INFO:teuthology.misc.health.smithi001.stderr:last_changed 2024-03-25T18:25:50.510519+0000 2024-03-25T18:35:20.021 INFO:teuthology.misc.health.smithi001.stderr:created 2024-03-25T18:25:50.510519+0000 2024-03-25T18:35:20.021 INFO:teuthology.misc.health.smithi001.stderr:min_mon_release 19 (squid) 2024-03-25T18:35:20.022 INFO:teuthology.misc.health.smithi001.stderr:election_strategy: 3 2024-03-25T18:35:20.022 INFO:teuthology.misc.health.smithi001.stderr:0: v1:172.21.15.1:6789/0 mon.a 2024-03-25T18:35:20.022 INFO:teuthology.misc.health.smithi001.stderr:1: v1:172.21.15.115:6789/0 mon.b 2024-03-25T18:35:20.022 INFO:teuthology.misc.health.smithi001.stderr:2: v1:172.21.15.1:6790/0 mon.c 2024-03-25T18:35:20.022 INFO:teuthology.misc.health.smithi001.stderr: 2024-03-25T18:35:20.022 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:35:20.020+0000 7ff8b3c0f640 10 monclient: get_monmap_and_config success 2024-03-25T18:35:20.022 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:35:20.020+0000 7ff8b3c0f640 4 set_mon_vals no callback set 2024-03-25T18:35:20.024 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:35:20.021+0000 7ff8b3c0f640 10 monclient: shutdown 2024-03-25T18:35:20.024 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:35:20.021+0000 7ff8b3c0f640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:35:20.024 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:35:20.021+0000 7ff8b3c0f640 1 -- 172.21.15.1:0/1287188621 >> v1:172.21.15.115:6789/0 conn(0x7ff8ac0b1b00 legacy=0x7ff8ac14dc00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:35:20.024 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:35:20.021+0000 7ff8b3c0f640 1 -- 172.21.15.1:0/1287188621 shutdown_connections 2024-03-25T18:35:20.024 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:35:20.021+0000 7ff8b3c0f640 1 -- 172.21.15.1:0/1287188621 >> 172.21.15.1:0/1287188621 conn(0x7ff8ac0617a0 msgr2=0x7ff8ac061bb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:35:20.024 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:35:20.021+0000 7ff8b3c0f640 1 -- 172.21.15.1:0/1287188621 shutdown_connections 2024-03-25T18:35:20.024 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:35:20.021+0000 7ff8b3c0f640 1 -- 172.21.15.1:0/1287188621 wait complete. 2024-03-25T18:35:20.024 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:35:20.022+0000 7ff8b3c0f640 10 monclient: build_initial_monmap 2024-03-25T18:35:20.024 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:35:20.022+0000 7ff8b3c0f640 10 monclient: monmap: 2024-03-25T18:35:20.024 INFO:teuthology.misc.health.smithi001.stderr:epoch 0 2024-03-25T18:35:20.024 INFO:teuthology.misc.health.smithi001.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T18:35:20.024 INFO:teuthology.misc.health.smithi001.stderr:last_changed 0.000000 2024-03-25T18:35:20.024 INFO:teuthology.misc.health.smithi001.stderr:created 0.000000 2024-03-25T18:35:20.024 INFO:teuthology.misc.health.smithi001.stderr:min_mon_release 0 (unknown) 2024-03-25T18:35:20.025 INFO:teuthology.misc.health.smithi001.stderr:election_strategy: 1 2024-03-25T18:35:20.025 INFO:teuthology.misc.health.smithi001.stderr:0: v1:172.21.15.1:6789/0 mon.noname-a 2024-03-25T18:35:20.025 INFO:teuthology.misc.health.smithi001.stderr:1: v1:172.21.15.115:6789/0 mon.noname-b 2024-03-25T18:35:20.025 INFO:teuthology.misc.health.smithi001.stderr:2: v1:172.21.15.1:6790/0 mon.noname-c 2024-03-25T18:35:20.025 INFO:teuthology.misc.health.smithi001.stderr: 2024-03-25T18:35:20.025 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:35:20.022+0000 7ff8b3c0f640 1 librados: starting msgr at 2024-03-25T18:35:20.025 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:35:20.022+0000 7ff8b3c0f640 1 librados: starting objecter 2024-03-25T18:35:20.025 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:35:20.023+0000 7ff8b3c0f640 1 Processor -- start 2024-03-25T18:35:20.025 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:35:20.023+0000 7ff8b3c0f640 1 -- start start 2024-03-25T18:35:20.025 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:35:20.023+0000 7ff8b3c0f640 1 librados: setting wanted keys 2024-03-25T18:35:20.025 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:35:20.023+0000 7ff8b3c0f640 1 librados: calling monclient init 2024-03-25T18:35:20.025 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:35:20.023+0000 7ff8b3c0f640 10 monclient: init 2024-03-25T18:35:20.026 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:35:20.023+0000 7ff8b3c0f640 10 monclient: _reopen_session rank -1 2024-03-25T18:35:20.026 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:35:20.023+0000 7ff8b3c0f640 10 monclient: _add_conns ranks=[1,0,2] 2024-03-25T18:35:20.026 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:35:20.023+0000 7ff8b3c0f640 10 monclient(hunting): picked mon.noname-b con 0x7ff8ac0b1b00 addr v1:172.21.15.115:6789/0 2024-03-25T18:35:20.026 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:35:20.024+0000 7ff8b3c0f640 10 monclient(hunting): picked mon.noname-a con 0x7ff8ac0b44b0 addr v1:172.21.15.1:6789/0 2024-03-25T18:35:20.026 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:35:20.024+0000 7ff8b3c0f640 10 monclient(hunting): picked mon.noname-c con 0x7ff8ac0b4e70 addr v1:172.21.15.1:6790/0 2024-03-25T18:35:20.026 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:35:20.024+0000 7ff8b3c0f640 1 -- --> v1:172.21.15.1:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff8ac135a60 con 0x7ff8ac0b44b0 2024-03-25T18:35:20.026 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:35:20.024+0000 7ff8b3c0f640 1 -- --> v1:172.21.15.115:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff8ac136e30 con 0x7ff8ac0b1b00 2024-03-25T18:35:20.027 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:35:20.024+0000 7ff8b3c0f640 1 -- --> v1:172.21.15.1:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff8ac135270 con 0x7ff8ac0b4e70 2024-03-25T18:35:20.027 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:35:20.024+0000 7ff8b3c0f640 10 monclient(hunting): _renew_subs 2024-03-25T18:35:20.027 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:35:20.024+0000 7ff8b3c0f640 10 monclient(hunting): authenticate will time out at 8000.493821s 2024-03-25T18:35:20.038 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:35:20.038+0000 7ff8b11aa640 1 --1- >> v1:172.21.15.1:6789/0 conn(0x7ff8ac0b44b0 0x7ff8ac1dac50 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.1:6789/0 says I am v1:172.21.15.1:42036/0 (socket says 172.21.15.1:42036) 2024-03-25T18:35:20.039 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:35:20.038+0000 7ff8b11aa640 1 -- 172.21.15.1:0/4071709775 learned_addr learned my addr 172.21.15.1:0/4071709775 (peer_addr_for_me v1:172.21.15.1:0/0) 2024-03-25T18:35:20.039 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:35:20.039+0000 7ff8a27fc640 10 client.?.objecter ms_handle_connect 0x7ff8ac0b44b0 2024-03-25T18:35:20.039 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:35:20.039+0000 7ff8a27fc640 10 client.?.objecter resend_mon_ops 2024-03-25T18:35:20.042 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:35:20.041+0000 7ff8a27fc640 10 client.?.objecter ms_handle_connect 0x7ff8ac0b4e70 2024-03-25T18:35:20.042 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:35:20.041+0000 7ff8a27fc640 10 client.?.objecter resend_mon_ops 2024-03-25T18:35:20.044 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:35:20.044+0000 7ff8a27fc640 10 client.?.objecter ms_handle_connect 0x7ff8ac0b1b00 2024-03-25T18:35:20.044 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:35:20.044+0000 7ff8a27fc640 10 client.?.objecter resend_mon_ops 2024-03-25T18:35:20.054 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:35:20.053+0000 7ff8a27fc640 1 -- 172.21.15.1:0/4071709775 <== mon.0 v1:172.21.15.1:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1580307386 0 0) 0x7ff8ac135a60 con 0x7ff8ac0b44b0 2024-03-25T18:35:20.054 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:35:20.053+0000 7ff8a27fc640 10 monclient(hunting): _negotiate 2024-03-25T18:35:20.054 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:35:20.053+0000 7ff8a27fc640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:35:20.054 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:35:20.053+0000 7ff8a27fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:35:20.054 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:35:20.053+0000 7ff8a27fc640 10 monclient(hunting): my global_id is 4593 2024-03-25T18:35:20.054 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:35:20.053+0000 7ff8a27fc640 1 -- 172.21.15.1:0/4071709775 --> v1:172.21.15.1:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ff890004eb0 con 0x7ff8ac0b44b0 2024-03-25T18:35:20.056 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:35:20.056+0000 7ff8a27fc640 1 -- 172.21.15.1:0/4071709775 <== mon.1 v1:172.21.15.115:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4099019625 0 0) 0x7ff8ac136e30 con 0x7ff8ac0b1b00 2024-03-25T18:35:20.057 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:35:20.056+0000 7ff8a27fc640 10 monclient(hunting): _negotiate 2024-03-25T18:35:20.057 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:35:20.056+0000 7ff8a27fc640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:35:20.057 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:35:20.056+0000 7ff8a27fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:35:20.057 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:35:20.056+0000 7ff8a27fc640 10 monclient(hunting): my global_id is 4600 2024-03-25T18:35:20.057 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:35:20.056+0000 7ff8a27fc640 1 -- 172.21.15.1:0/4071709775 --> v1:172.21.15.115:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ff8ac135a60 con 0x7ff8ac0b1b00 2024-03-25T18:35:20.063 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:35:20.062+0000 7ff8a27fc640 1 -- 172.21.15.1:0/4071709775 <== mon.2 v1:172.21.15.1:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1765484743 0 0) 0x7ff8ac135270 con 0x7ff8ac0b4e70 2024-03-25T18:35:20.063 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:35:20.062+0000 7ff8a27fc640 10 monclient(hunting): _negotiate 2024-03-25T18:35:20.063 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:35:20.062+0000 7ff8a27fc640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:35:20.063 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:35:20.062+0000 7ff8a27fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:35:20.063 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:35:20.062+0000 7ff8a27fc640 10 monclient(hunting): my global_id is 4601 2024-03-25T18:35:20.063 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:35:20.062+0000 7ff8a27fc640 1 -- 172.21.15.1:0/4071709775 --> v1:172.21.15.1:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ff8ac136e30 con 0x7ff8ac0b4e70 2024-03-25T18:35:20.078 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:35:20.078+0000 7ff8a27fc640 1 -- 172.21.15.1:0/4071709775 <== mon.0 v1:172.21.15.1:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2087298729 0 0) 0x7ff890004eb0 con 0x7ff8ac0b44b0 2024-03-25T18:35:20.079 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:35:20.078+0000 7ff8a27fc640 1 -- 172.21.15.1:0/4071709775 --> v1:172.21.15.1:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7ff8ac135270 con 0x7ff8ac0b44b0 2024-03-25T18:35:20.079 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:35:20.078+0000 7ff8a27fc640 1 -- 172.21.15.1:0/4071709775 <== mon.0 v1:172.21.15.1:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1586965025 0 0) 0x7ff89c001a80 con 0x7ff8ac0b44b0 2024-03-25T18:35:20.079 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:35:20.078+0000 7ff8a27fc640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:35:20.079 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:35:20.078+0000 7ff8a27fc640 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2024-03-25T18:35:20.079 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:35:20.078+0000 7ff8a27fc640 10 monclient(hunting): dump: 2024-03-25T18:35:20.079 INFO:teuthology.misc.health.smithi001.stderr:epoch 1 2024-03-25T18:35:20.079 INFO:teuthology.misc.health.smithi001.stderr:fsid c8ea72cf-9e99-4383-9b52-6c23d365fdfc 2024-03-25T18:35:20.079 INFO:teuthology.misc.health.smithi001.stderr:last_changed 2024-03-25T18:25:50.510519+0000 2024-03-25T18:35:20.079 INFO:teuthology.misc.health.smithi001.stderr:created 2024-03-25T18:25:50.510519+0000 2024-03-25T18:35:20.079 INFO:teuthology.misc.health.smithi001.stderr:min_mon_release 19 (squid) 2024-03-25T18:35:20.079 INFO:teuthology.misc.health.smithi001.stderr:election_strategy: 3 2024-03-25T18:35:20.079 INFO:teuthology.misc.health.smithi001.stderr:0: v1:172.21.15.1:6789/0 mon.a 2024-03-25T18:35:20.079 INFO:teuthology.misc.health.smithi001.stderr:1: v1:172.21.15.115:6789/0 mon.b 2024-03-25T18:35:20.079 INFO:teuthology.misc.health.smithi001.stderr:2: v1:172.21.15.1:6790/0 mon.c 2024-03-25T18:35:20.080 INFO:teuthology.misc.health.smithi001.stderr: 2024-03-25T18:35:20.080 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:35:20.078+0000 7ff8a27fc640 10 monclient(hunting): _finish_auth 0 2024-03-25T18:35:20.088 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:35:20.087+0000 7ff8a27fc640 1 -- 172.21.15.1:0/4071709775 <== mon.2 v1:172.21.15.1:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 626379790 0 0) 0x7ff8ac136e30 con 0x7ff8ac0b4e70 2024-03-25T18:35:20.088 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:35:20.087+0000 7ff8a27fc640 1 -- 172.21.15.1:0/4071709775 --> v1:172.21.15.1:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7ff890004eb0 con 0x7ff8ac0b4e70 2024-03-25T18:35:20.088 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:35:20.087+0000 7ff8a27fc640 1 -- 172.21.15.1:0/4071709775 <== mon.2 v1:172.21.15.1:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1586965025 0 0) 0x7ff8a4001e80 con 0x7ff8ac0b4e70 2024-03-25T18:35:20.088 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:35:20.087+0000 7ff8a27fc640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:35:20.088 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:35:20.087+0000 7ff8a27fc640 10 monclient(hunting): got monmap 1 from mon.c (according to old e1) 2024-03-25T18:35:20.088 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:35:20.087+0000 7ff8a27fc640 10 monclient(hunting): dump: 2024-03-25T18:35:20.088 INFO:teuthology.misc.health.smithi001.stderr:epoch 1 2024-03-25T18:35:20.088 INFO:teuthology.misc.health.smithi001.stderr:fsid c8ea72cf-9e99-4383-9b52-6c23d365fdfc 2024-03-25T18:35:20.088 INFO:teuthology.misc.health.smithi001.stderr:last_changed 2024-03-25T18:25:50.510519+0000 2024-03-25T18:35:20.088 INFO:teuthology.misc.health.smithi001.stderr:created 2024-03-25T18:25:50.510519+0000 2024-03-25T18:35:20.088 INFO:teuthology.misc.health.smithi001.stderr:min_mon_release 19 (squid) 2024-03-25T18:35:20.088 INFO:teuthology.misc.health.smithi001.stderr:election_strategy: 3 2024-03-25T18:35:20.088 INFO:teuthology.misc.health.smithi001.stderr:0: v1:172.21.15.1:6789/0 mon.a 2024-03-25T18:35:20.088 INFO:teuthology.misc.health.smithi001.stderr:1: v1:172.21.15.115:6789/0 mon.b 2024-03-25T18:35:20.089 INFO:teuthology.misc.health.smithi001.stderr:2: v1:172.21.15.1:6790/0 mon.c 2024-03-25T18:35:20.089 INFO:teuthology.misc.health.smithi001.stderr: 2024-03-25T18:35:20.091 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:35:20.090+0000 7ff8a27fc640 1 -- 172.21.15.1:0/4071709775 <== mon.0 v1:172.21.15.1:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 685855962 0 0) 0x7ff8ac135270 con 0x7ff8ac0b44b0 2024-03-25T18:35:20.091 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:35:20.090+0000 7ff8a27fc640 1 -- 172.21.15.1:0/4071709775 >> v1:172.21.15.1:6790/0 conn(0x7ff8ac0b4e70 legacy=0x7ff8ac1db360 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:35:20.091 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:35:20.090+0000 7ff8a27fc640 1 -- 172.21.15.1:0/4071709775 >> v1:172.21.15.115:6789/0 conn(0x7ff8ac0b1b00 legacy=0x7ff8ac134a40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:35:20.091 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:35:20.090+0000 7ff8a27fc640 10 monclient: _finish_hunting 0 2024-03-25T18:35:20.091 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:35:20.090+0000 7ff8a27fc640 1 monclient: found mon.a 2024-03-25T18:35:20.091 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:35:20.090+0000 7ff8a27fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:35:20.092 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:35:20.090+0000 7ff8a27fc640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.1:6789/0 2024-03-25T18:35:20.092 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:35:20.090+0000 7ff8a27fc640 1 -- 172.21.15.1:0/4071709775 --> v1:172.21.15.1:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff8ac135400 con 0x7ff8ac0b44b0 2024-03-25T18:35:20.092 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:35:20.090+0000 7ff8a27fc640 10 monclient: _finish_auth 0 2024-03-25T18:35:20.092 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:35:20.090+0000 7ff8a27fc640 10 monclient: _check_auth_tickets 2024-03-25T18:35:20.092 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:35:20.090+0000 7ff8a27fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:35:20.092 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:35:20.090+0000 7ff8b3c0f640 5 monclient: authenticate success, global_id 4593 2024-03-25T18:35:20.092 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:35:20.090+0000 7ff8b3c0f640 10 monclient: _renew_subs 2024-03-25T18:35:20.092 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:35:20.090+0000 7ff8b3c0f640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.1:6789/0 2024-03-25T18:35:20.092 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:35:20.090+0000 7ff8b3c0f640 1 -- 172.21.15.1:0/4071709775 --> v1:172.21.15.1:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff8ac135650 con 0x7ff8ac0b44b0 2024-03-25T18:35:20.092 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:35:20.090+0000 7ff8b3c0f640 10 client.4593.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T18:35:20.092 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:35:20.090+0000 7ff8b3c0f640 10 monclient: _renew_subs 2024-03-25T18:35:20.092 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:35:20.090+0000 7ff8b3c0f640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.1:6789/0 2024-03-25T18:35:20.092 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:35:20.090+0000 7ff8b3c0f640 1 -- 172.21.15.1:0/4071709775 --> v1:172.21.15.1:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7ff8ac1e3380 con 0x7ff8ac0b44b0 2024-03-25T18:35:20.092 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:35:20.090+0000 7ff8b3c0f640 1 librados: init done 2024-03-25T18:35:20.093 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:35:20.091+0000 7ff8b3c0f640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T18:35:20.093 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:35:20.091+0000 7ff8b3c0f640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:35:20.093 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:35:20.091+0000 7ff8b3c0f640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.1:6789/0 2024-03-25T18:35:20.093 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:35:20.091+0000 7ff8b3c0f640 1 -- 172.21.15.1:0/4071709775 --> v1:172.21.15.1:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff8ac135840 con 0x7ff8ac0b44b0 2024-03-25T18:35:20.418 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:35:20.417+0000 7ff8a27fc640 1 -- 172.21.15.1:0/4071709775 <== mon.0 v1:172.21.15.1:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7ff89c001ca0 con 0x7ff8ac0b44b0 2024-03-25T18:35:20.418 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:35:20.417+0000 7ff8a27fc640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:35:20.418 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:35:20.417+0000 7ff8a27fc640 1 -- 172.21.15.1:0/4071709775 <== mon.0 v1:172.21.15.1:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1586965025 0 0) 0x7ff89c003a60 con 0x7ff8ac0b44b0 2024-03-25T18:35:20.418 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:35:20.417+0000 7ff8a27fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:35:20.418 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:35:20.417+0000 7ff8a27fc640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-25T18:35:20.419 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:35:20.417+0000 7ff8a27fc640 10 monclient: dump: 2024-03-25T18:35:20.419 INFO:teuthology.misc.health.smithi001.stderr:epoch 1 2024-03-25T18:35:20.419 INFO:teuthology.misc.health.smithi001.stderr:fsid c8ea72cf-9e99-4383-9b52-6c23d365fdfc 2024-03-25T18:35:20.419 INFO:teuthology.misc.health.smithi001.stderr:last_changed 2024-03-25T18:25:50.510519+0000 2024-03-25T18:35:20.419 INFO:teuthology.misc.health.smithi001.stderr:created 2024-03-25T18:25:50.510519+0000 2024-03-25T18:35:20.419 INFO:teuthology.misc.health.smithi001.stderr:min_mon_release 19 (squid) 2024-03-25T18:35:20.419 INFO:teuthology.misc.health.smithi001.stderr:election_strategy: 3 2024-03-25T18:35:20.419 INFO:teuthology.misc.health.smithi001.stderr:0: v1:172.21.15.1:6789/0 mon.a 2024-03-25T18:35:20.419 INFO:teuthology.misc.health.smithi001.stderr:1: v1:172.21.15.115:6789/0 mon.b 2024-03-25T18:35:20.419 INFO:teuthology.misc.health.smithi001.stderr:2: v1:172.21.15.1:6790/0 mon.c 2024-03-25T18:35:20.419 INFO:teuthology.misc.health.smithi001.stderr: 2024-03-25T18:35:20.420 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:35:20.417+0000 7ff8a3fff640 4 set_mon_vals no callback set 2024-03-25T18:35:20.420 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:35:20.418+0000 7ff8a27fc640 1 -- 172.21.15.1:0/4071709775 <== mon.0 v1:172.21.15.1:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 21787975 0 0) 0x7ff89c003c80 con 0x7ff8ac0b44b0 2024-03-25T18:35:20.420 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:35:20.419+0000 7ff8b19ab640 10 monclient: get_auth_request con 0x7ff89006df70 auth_method 0 2024-03-25T18:35:20.420 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:35:20.419+0000 7ff8a27fc640 10 client.4593.objecter ms_handle_connect 0x7ff89006df70 2024-03-25T18:35:20.427 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:35:20.426+0000 7ff8a27fc640 1 -- 172.21.15.1:0/4071709775 <== mon.0 v1:172.21.15.1:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 996254230 0 0) 0x7ff89c0548f0 con 0x7ff8ac0b44b0 2024-03-25T18:35:20.427 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:35:20.426+0000 7ff8a27fc640 10 client.4593.objecter ms_dispatch 0x7ff8ac000b90 osd_map(10..10 src has 1..10) v4 2024-03-25T18:35:20.427 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:35:20.426+0000 7ff8a27fc640 3 client.4593.objecter handle_osd_map got epochs [10,10] > 0 2024-03-25T18:35:20.427 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:35:20.426+0000 7ff8a27fc640 3 client.4593.objecter handle_osd_map decoding full epoch 10 2024-03-25T18:35:20.427 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:35:20.426+0000 7ff8a27fc640 20 client.4593.objecter dump_active .. 0 homeless 2024-03-25T18:35:20.427 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:35:20.426+0000 7ff8a27fc640 1 -- 172.21.15.1:0/4071709775 <== mon.0 v1:172.21.15.1:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7ff89c050400 con 0x7ff8ac0b44b0 2024-03-25T18:35:20.427 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:35:20.426+0000 7ff8a27fc640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:35:20.427 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:35:20.426+0000 7ff8a27fc640 10 monclient: _finish_command 1 = system:0 2024-03-25T18:35:20.681 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:35:20.680+0000 7ff8b3c0f640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:35:20.681 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:35:20.680+0000 7ff8b3c0f640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:35:20.682 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:35:20.680+0000 7ff8b3c0f640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.1:6789/0 2024-03-25T18:35:20.682 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:35:20.680+0000 7ff8b3c0f640 1 -- 172.21.15.1:0/4071709775 --> v1:172.21.15.1:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7ff8ac135c10 con 0x7ff8ac0b44b0 2024-03-25T18:35:20.705 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:35:20.704+0000 7ff8a27fc640 1 -- 172.21.15.1:0/4071709775 <== mon.0 v1:172.21.15.1:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 4092213348) 0x7ff8ac135c10 con 0x7ff8ac0b44b0 2024-03-25T18:35:20.705 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:35:20.704+0000 7ff8a27fc640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:35:20.705 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:35:20.704+0000 7ff8a27fc640 10 monclient: _finish_command 2 = system:0 2024-03-25T18:35:20.706 INFO:teuthology.misc.health.smithi001.stdout: 2024-03-25T18:35:20.706 INFO:teuthology.misc.health.smithi001.stdout:{"epoch":10,"fsid":"c8ea72cf-9e99-4383-9b52-6c23d365fdfc","created":"2024-03-25T18:26:38.856638+0000","modified":"2024-03-25T18:26:44.224208+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T18:26:44.224208+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"b9f34922-3a84-49be-8426-807c1eceaefc","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"42bf306f-55e3-4634-9874-8b4b73ac6644","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"292cb23a-18a9-4207-a7cc-4d5727feaec0","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"45e0e9ab-898d-4673-9590-b4117bc5e84a","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"831a5a09-73ab-4a61-9872-088e76bdbc66","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"88cd903d-3b55-4840-9396-6e3cfe005282","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"89db7355-bc9d-46f4-8ab0-e8d9e744e026","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"cac47bee-0914-4f08-896f-348838882866","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T18:35:20.708 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:35:20.707+0000 7ff8b3c0f640 10 librados: watch_flush enter 2024-03-25T18:35:20.708 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:35:20.707+0000 7ff8b3c0f640 10 librados: watch_flush exit 2024-03-25T18:35:20.708 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:35:20.707+0000 7ff8b3c0f640 20 client.4593.objecter shutdown clearing up homeless session... 2024-03-25T18:35:20.708 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:35:20.707+0000 7ff8b3c0f640 10 client.4593.objecter successfully canceled tick 2024-03-25T18:35:20.709 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:35:20.707+0000 7ff8b3c0f640 1 -- 172.21.15.1:0/4071709775 >> v1:172.21.15.115:6800/4237379176 conn(0x7ff89006df70 legacy=0x7ff890070430 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:35:20.709 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:35:20.707+0000 7ff8b3c0f640 10 monclient: shutdown 2024-03-25T18:35:20.709 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:35:20.707+0000 7ff8b3c0f640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:35:20.709 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:35:20.707+0000 7ff8b3c0f640 1 -- 172.21.15.1:0/4071709775 >> v1:172.21.15.1:6789/0 conn(0x7ff8ac0b44b0 legacy=0x7ff8ac1dac50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:35:20.709 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:35:20.707+0000 7ff8b3c0f640 1 -- 172.21.15.1:0/4071709775 shutdown_connections 2024-03-25T18:35:20.709 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:35:20.707+0000 7ff8b3c0f640 1 -- 172.21.15.1:0/4071709775 >> 172.21.15.1:0/4071709775 conn(0x7ff8ac0617a0 msgr2=0x7ff8ac0b2710 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:35:20.709 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:35:20.707+0000 7ff8b3c0f640 1 -- 172.21.15.1:0/4071709775 shutdown_connections 2024-03-25T18:35:20.709 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:35:20.707+0000 7ff8b3c0f640 1 -- 172.21.15.1:0/4071709775 wait complete. 2024-03-25T18:35:20.709 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:35:20.707+0000 7ff8b3c0f640 1 librados: shutdown 2024-03-25T18:35:20.719 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T18:35:22.869 INFO:tasks.ceph.mon.a.smithi001.stderr:==00:00:09:01.428 34876== Warning: unimplemented fcntl command: 1036 2024-03-25T18:35:22.876 INFO:tasks.ceph.mon.a.smithi001.stderr:==00:00:09:01.434 34876== Warning: unimplemented fcntl command: 1036 2024-03-25T18:35:22.898 INFO:tasks.ceph.mon.a.smithi001.stderr:==00:00:09:01.457 34876== Warning: unimplemented fcntl command: 1036 2024-03-25T18:35:22.926 INFO:tasks.ceph.mon.b.smithi115.stderr:==00:00:09:01.410 34836== Warning: unimplemented fcntl command: 1036 2024-03-25T18:35:22.935 INFO:tasks.ceph.mon.b.smithi115.stderr:==00:00:09:01.419 34836== Warning: unimplemented fcntl command: 1036 2024-03-25T18:35:22.936 INFO:tasks.ceph.mon.c.smithi001.stderr:==00:00:09:01.494 34877== Warning: unimplemented fcntl command: 1036 2024-03-25T18:35:22.948 INFO:tasks.ceph.mon.c.smithi001.stderr:==00:00:09:01.506 34877== Warning: unimplemented fcntl command: 1036 2024-03-25T18:35:22.977 INFO:tasks.ceph.mon.b.smithi115.stderr:==00:00:09:01.460 34836== Warning: unimplemented fcntl command: 1036 2024-03-25T18:35:22.982 INFO:tasks.ceph.mon.c.smithi001.stderr:==00:00:09:01.541 34877== Warning: unimplemented fcntl command: 1036 2024-03-25T18:35:27.525 DEBUG:teuthology.orchestra.run.smithi001:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T18:35:27.642 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:35:27.640+0000 7f30b5fc7640 10 monclient: get_monmap_and_config 2024-03-25T18:35:27.642 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:35:27.640+0000 7f30b5fc7640 10 monclient: build_initial_monmap 2024-03-25T18:35:27.642 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:35:27.640+0000 7f30b5fc7640 10 monclient: monmap: 2024-03-25T18:35:27.642 INFO:teuthology.misc.health.smithi001.stderr:epoch 0 2024-03-25T18:35:27.642 INFO:teuthology.misc.health.smithi001.stderr:fsid c8ea72cf-9e99-4383-9b52-6c23d365fdfc 2024-03-25T18:35:27.643 INFO:teuthology.misc.health.smithi001.stderr:last_changed 2024-03-25T18:35:27.642253+0000 2024-03-25T18:35:27.643 INFO:teuthology.misc.health.smithi001.stderr:created 2024-03-25T18:35:27.642253+0000 2024-03-25T18:35:27.643 INFO:teuthology.misc.health.smithi001.stderr:min_mon_release 0 (unknown) 2024-03-25T18:35:27.643 INFO:teuthology.misc.health.smithi001.stderr:election_strategy: 3 2024-03-25T18:35:27.643 INFO:teuthology.misc.health.smithi001.stderr:0: v1:172.21.15.1:6789/0 mon.noname-a 2024-03-25T18:35:27.643 INFO:teuthology.misc.health.smithi001.stderr:1: v1:172.21.15.115:6789/0 mon.noname-c 2024-03-25T18:35:27.643 INFO:teuthology.misc.health.smithi001.stderr:2: v1:172.21.15.1:6790/0 mon.noname-b 2024-03-25T18:35:27.643 INFO:teuthology.misc.health.smithi001.stderr: 2024-03-25T18:35:27.644 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:35:27.641+0000 7f30b5fc7640 1 Processor -- start 2024-03-25T18:35:27.644 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:35:27.641+0000 7f30b5fc7640 1 -- start start 2024-03-25T18:35:27.644 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:35:27.641+0000 7f30b5fc7640 10 monclient: init 2024-03-25T18:35:27.645 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:35:27.642+0000 7f30b5fc7640 10 monclient: _reopen_session rank -1 2024-03-25T18:35:27.645 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:35:27.642+0000 7f30b5fc7640 10 monclient: _add_conns ranks=[0,1,2] 2024-03-25T18:35:27.645 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:35:27.642+0000 7f30b5fc7640 10 monclient(hunting): picked mon.noname-a con 0x7f30b00b4e70 addr v1:172.21.15.1:6789/0 2024-03-25T18:35:27.645 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:35:27.642+0000 7f30b5fc7640 10 monclient(hunting): picked mon.noname-c con 0x7f30b00b44b0 addr v1:172.21.15.115:6789/0 2024-03-25T18:35:27.645 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:35:27.642+0000 7f30b5fc7640 10 monclient(hunting): picked mon.noname-b con 0x7f30b00b1b00 addr v1:172.21.15.1:6790/0 2024-03-25T18:35:27.645 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:35:27.642+0000 7f30b5fc7640 1 -- --> v1:172.21.15.1:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f30b00a96f0 con 0x7f30b00b4e70 2024-03-25T18:35:27.645 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:35:27.642+0000 7f30b5fc7640 1 -- --> v1:172.21.15.115:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f30b00ab0e0 con 0x7f30b00b44b0 2024-03-25T18:35:27.645 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:35:27.642+0000 7f30b5fc7640 1 -- --> v1:172.21.15.1:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f30b0145f70 con 0x7f30b00b1b00 2024-03-25T18:35:27.645 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:35:27.642+0000 7f30b5fc7640 10 monclient(hunting): _renew_subs 2024-03-25T18:35:27.645 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:35:27.642+0000 7f30b5fc7640 10 monclient(hunting): authenticate will time out at 8008.113366s 2024-03-25T18:35:27.664 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:35:27.663+0000 7f30aeffd640 1 --1- >> v1:172.21.15.115:6789/0 conn(0x7f30b00b44b0 0x7f30b0145860 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.115:6789/0 says I am v1:172.21.15.1:46240/0 (socket says 172.21.15.1:46240) 2024-03-25T18:35:27.664 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:35:27.663+0000 7f30aeffd640 1 -- 172.21.15.1:0/3290728214 learned_addr learned my addr 172.21.15.1:0/3290728214 (peer_addr_for_me v1:172.21.15.1:0/0) 2024-03-25T18:35:27.678 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:35:27.676+0000 7f30ae7fc640 1 -- 172.21.15.1:0/3290728214 <== mon.1 v1:172.21.15.115:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1332651224 0 0) 0x7f30b00ab0e0 con 0x7f30b00b44b0 2024-03-25T18:35:27.678 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:35:27.676+0000 7f30ae7fc640 10 monclient(hunting): _negotiate 2024-03-25T18:35:27.678 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:35:27.676+0000 7f30ae7fc640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:35:27.678 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:35:27.676+0000 7f30ae7fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:35:27.678 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:35:27.676+0000 7f30ae7fc640 10 monclient(hunting): my global_id is 4603 2024-03-25T18:35:27.678 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:35:27.676+0000 7f30ae7fc640 1 -- 172.21.15.1:0/3290728214 --> v1:172.21.15.115:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f3098003520 con 0x7f30b00b44b0 2024-03-25T18:35:27.685 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:35:27.683+0000 7f30ae7fc640 1 -- 172.21.15.1:0/3290728214 <== mon.0 v1:172.21.15.1:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3673474901 0 0) 0x7f30b00a96f0 con 0x7f30b00b4e70 2024-03-25T18:35:27.685 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:35:27.683+0000 7f30ae7fc640 10 monclient(hunting): _negotiate 2024-03-25T18:35:27.685 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:35:27.683+0000 7f30ae7fc640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:35:27.685 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:35:27.683+0000 7f30ae7fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:35:27.685 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:35:27.683+0000 7f30ae7fc640 10 monclient(hunting): my global_id is 4596 2024-03-25T18:35:27.685 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:35:27.683+0000 7f30ae7fc640 1 -- 172.21.15.1:0/3290728214 --> v1:172.21.15.1:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f30b00ab0e0 con 0x7f30b00b4e70 2024-03-25T18:35:27.686 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:35:27.684+0000 7f30ae7fc640 1 -- 172.21.15.1:0/3290728214 <== mon.2 v1:172.21.15.1:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3317654556 0 0) 0x7f30b0145f70 con 0x7f30b00b1b00 2024-03-25T18:35:27.686 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:35:27.684+0000 7f30ae7fc640 10 monclient(hunting): _negotiate 2024-03-25T18:35:27.686 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:35:27.684+0000 7f30ae7fc640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:35:27.686 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:35:27.684+0000 7f30ae7fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:35:27.686 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:35:27.684+0000 7f30ae7fc640 10 monclient(hunting): my global_id is 4604 2024-03-25T18:35:27.686 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:35:27.684+0000 7f30ae7fc640 1 -- 172.21.15.1:0/3290728214 --> v1:172.21.15.1:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f30b00a96f0 con 0x7f30b00b1b00 2024-03-25T18:35:27.691 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:35:27.690+0000 7f30ae7fc640 1 -- 172.21.15.1:0/3290728214 <== mon.1 v1:172.21.15.115:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2138089804 0 0) 0x7f3098003520 con 0x7f30b00b44b0 2024-03-25T18:35:27.692 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:35:27.690+0000 7f30ae7fc640 1 -- 172.21.15.1:0/3290728214 >> v1:172.21.15.1:6790/0 conn(0x7f30b00b1b00 legacy=0x7f30b014dc00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:35:27.692 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:35:27.690+0000 7f30ae7fc640 1 -- 172.21.15.1:0/3290728214 >> v1:172.21.15.1:6789/0 conn(0x7f30b00b4e70 legacy=0x7f30b013f940 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:35:27.692 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:35:27.690+0000 7f30ae7fc640 10 monclient: _finish_hunting 0 2024-03-25T18:35:27.692 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:35:27.690+0000 7f30ae7fc640 1 monclient: found mon.noname-c 2024-03-25T18:35:27.692 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:35:27.690+0000 7f30ae7fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:35:27.692 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:35:27.690+0000 7f30ae7fc640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.115:6789/0 2024-03-25T18:35:27.692 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:35:27.690+0000 7f30ae7fc640 1 -- 172.21.15.1:0/3290728214 --> v1:172.21.15.115:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f30b00a9b80 con 0x7f30b00b44b0 2024-03-25T18:35:27.692 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:35:27.690+0000 7f30ae7fc640 10 monclient: _finish_auth 0 2024-03-25T18:35:27.692 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:35:27.690+0000 7f30ae7fc640 10 monclient: _check_auth_tickets 2024-03-25T18:35:27.692 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:35:27.690+0000 7f30ae7fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:35:27.692 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:35:27.690+0000 7f30ae7fc640 1 -- 172.21.15.1:0/3290728214 <== mon.1 v1:172.21.15.115:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1586965025 0 0) 0x7f30a4001690 con 0x7f30b00b44b0 2024-03-25T18:35:27.692 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:35:27.690+0000 7f30ae7fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:35:27.692 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:35:27.690+0000 7f30ae7fc640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-03-25T18:35:27.692 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:35:27.690+0000 7f30ae7fc640 10 monclient: dump: 2024-03-25T18:35:27.692 INFO:teuthology.misc.health.smithi001.stderr:epoch 1 2024-03-25T18:35:27.693 INFO:teuthology.misc.health.smithi001.stderr:fsid c8ea72cf-9e99-4383-9b52-6c23d365fdfc 2024-03-25T18:35:27.693 INFO:teuthology.misc.health.smithi001.stderr:last_changed 2024-03-25T18:25:50.510519+0000 2024-03-25T18:35:27.693 INFO:teuthology.misc.health.smithi001.stderr:created 2024-03-25T18:25:50.510519+0000 2024-03-25T18:35:27.693 INFO:teuthology.misc.health.smithi001.stderr:min_mon_release 19 (squid) 2024-03-25T18:35:27.693 INFO:teuthology.misc.health.smithi001.stderr:election_strategy: 3 2024-03-25T18:35:27.693 INFO:teuthology.misc.health.smithi001.stderr:0: v1:172.21.15.1:6789/0 mon.a 2024-03-25T18:35:27.693 INFO:teuthology.misc.health.smithi001.stderr:1: v1:172.21.15.115:6789/0 mon.b 2024-03-25T18:35:27.693 INFO:teuthology.misc.health.smithi001.stderr:2: v1:172.21.15.1:6790/0 mon.c 2024-03-25T18:35:27.693 INFO:teuthology.misc.health.smithi001.stderr: 2024-03-25T18:35:27.693 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:35:27.690+0000 7f30b5fc7640 5 monclient: authenticate success, global_id 4603 2024-03-25T18:35:27.693 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:35:27.690+0000 7f30b5fc7640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T18:35:27.709 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:35:27.707+0000 7f30ae7fc640 1 -- 172.21.15.1:0/3290728214 <== mon.1 v1:172.21.15.115:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f30a40017f0 con 0x7f30b00b44b0 2024-03-25T18:35:27.709 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:35:27.707+0000 7f30ae7fc640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:35:27.709 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:35:27.707+0000 7f30ae7fc640 1 -- 172.21.15.1:0/3290728214 <== mon.1 v1:172.21.15.115:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1586965025 0 0) 0x7f30a4001c00 con 0x7f30b00b44b0 2024-03-25T18:35:27.710 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:35:27.707+0000 7f30b5fc7640 10 monclient: get_monmap_and_config success 2024-03-25T18:35:27.710 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:35:27.707+0000 7f30b5fc7640 4 set_mon_vals no callback set 2024-03-25T18:35:27.710 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:35:27.707+0000 7f30ae7fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:35:27.710 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:35:27.707+0000 7f30ae7fc640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-25T18:35:27.710 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:35:27.707+0000 7f30ae7fc640 10 monclient: dump: 2024-03-25T18:35:27.710 INFO:teuthology.misc.health.smithi001.stderr:epoch 1 2024-03-25T18:35:27.710 INFO:teuthology.misc.health.smithi001.stderr:fsid c8ea72cf-9e99-4383-9b52-6c23d365fdfc 2024-03-25T18:35:27.710 INFO:teuthology.misc.health.smithi001.stderr:last_changed 2024-03-25T18:25:50.510519+0000 2024-03-25T18:35:27.710 INFO:teuthology.misc.health.smithi001.stderr:created 2024-03-25T18:25:50.510519+0000 2024-03-25T18:35:27.710 INFO:teuthology.misc.health.smithi001.stderr:min_mon_release 19 (squid) 2024-03-25T18:35:27.710 INFO:teuthology.misc.health.smithi001.stderr:election_strategy: 3 2024-03-25T18:35:27.710 INFO:teuthology.misc.health.smithi001.stderr:0: v1:172.21.15.1:6789/0 mon.a 2024-03-25T18:35:27.710 INFO:teuthology.misc.health.smithi001.stderr:1: v1:172.21.15.115:6789/0 mon.b 2024-03-25T18:35:27.710 INFO:teuthology.misc.health.smithi001.stderr:2: v1:172.21.15.1:6790/0 mon.c 2024-03-25T18:35:27.711 INFO:teuthology.misc.health.smithi001.stderr: 2024-03-25T18:35:27.711 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:35:27.707+0000 7f30b5fc7640 10 monclient: shutdown 2024-03-25T18:35:27.711 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:35:27.707+0000 7f30b5fc7640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:35:27.711 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:35:27.707+0000 7f30b5fc7640 1 -- 172.21.15.1:0/3290728214 >> v1:172.21.15.115:6789/0 conn(0x7f30b00b44b0 legacy=0x7f30b0145860 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:35:27.711 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:35:27.707+0000 7f30b5fc7640 1 -- 172.21.15.1:0/3290728214 shutdown_connections 2024-03-25T18:35:27.711 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:35:27.707+0000 7f30b5fc7640 1 -- 172.21.15.1:0/3290728214 >> 172.21.15.1:0/3290728214 conn(0x7f30b00617a0 msgr2=0x7f30b0061bb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:35:27.711 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:35:27.708+0000 7f30b5fc7640 1 -- 172.21.15.1:0/3290728214 shutdown_connections 2024-03-25T18:35:27.711 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:35:27.708+0000 7f30b5fc7640 1 -- 172.21.15.1:0/3290728214 wait complete. 2024-03-25T18:35:27.712 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:35:27.708+0000 7f30b5fc7640 10 monclient: build_initial_monmap 2024-03-25T18:35:27.713 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:35:27.708+0000 7f30b5fc7640 10 monclient: monmap: 2024-03-25T18:35:27.713 INFO:teuthology.misc.health.smithi001.stderr:epoch 0 2024-03-25T18:35:27.713 INFO:teuthology.misc.health.smithi001.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T18:35:27.713 INFO:teuthology.misc.health.smithi001.stderr:last_changed 0.000000 2024-03-25T18:35:27.713 INFO:teuthology.misc.health.smithi001.stderr:created 0.000000 2024-03-25T18:35:27.713 INFO:teuthology.misc.health.smithi001.stderr:min_mon_release 0 (unknown) 2024-03-25T18:35:27.713 INFO:teuthology.misc.health.smithi001.stderr:election_strategy: 1 2024-03-25T18:35:27.713 INFO:teuthology.misc.health.smithi001.stderr:0: v1:172.21.15.1:6789/0 mon.noname-a 2024-03-25T18:35:27.713 INFO:teuthology.misc.health.smithi001.stderr:1: v1:172.21.15.115:6789/0 mon.noname-b 2024-03-25T18:35:27.713 INFO:teuthology.misc.health.smithi001.stderr:2: v1:172.21.15.1:6790/0 mon.noname-c 2024-03-25T18:35:27.713 INFO:teuthology.misc.health.smithi001.stderr: 2024-03-25T18:35:27.713 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:35:27.709+0000 7f30b5fc7640 1 librados: starting msgr at 2024-03-25T18:35:27.713 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:35:27.709+0000 7f30b5fc7640 1 librados: starting objecter 2024-03-25T18:35:27.714 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:35:27.709+0000 7f30b5fc7640 1 Processor -- start 2024-03-25T18:35:27.714 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:35:27.709+0000 7f30b5fc7640 1 -- start start 2024-03-25T18:35:27.714 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:35:27.709+0000 7f30b5fc7640 1 librados: setting wanted keys 2024-03-25T18:35:27.714 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:35:27.709+0000 7f30b5fc7640 1 librados: calling monclient init 2024-03-25T18:35:27.714 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:35:27.709+0000 7f30b5fc7640 10 monclient: init 2024-03-25T18:35:27.714 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:35:27.710+0000 7f30b5fc7640 10 monclient: _reopen_session rank -1 2024-03-25T18:35:27.714 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:35:27.710+0000 7f30b5fc7640 10 monclient: _add_conns ranks=[0,2,1] 2024-03-25T18:35:27.714 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:35:27.710+0000 7f30b5fc7640 10 monclient(hunting): picked mon.noname-a con 0x7f30b00b1b00 addr v1:172.21.15.1:6789/0 2024-03-25T18:35:27.714 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:35:27.710+0000 7f30b5fc7640 10 monclient(hunting): picked mon.noname-c con 0x7f30b00b44b0 addr v1:172.21.15.1:6790/0 2024-03-25T18:35:27.714 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:35:27.710+0000 7f30b5fc7640 10 monclient(hunting): picked mon.noname-b con 0x7f30b00b4e70 addr v1:172.21.15.115:6789/0 2024-03-25T18:35:27.714 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:35:27.710+0000 7f30b5fc7640 1 -- --> v1:172.21.15.1:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f30b013b6c0 con 0x7f30b00b1b00 2024-03-25T18:35:27.714 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:35:27.710+0000 7f30b5fc7640 1 -- --> v1:172.21.15.115:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f30b013c870 con 0x7f30b00b4e70 2024-03-25T18:35:27.714 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:35:27.710+0000 7f30b5fc7640 1 -- --> v1:172.21.15.1:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f30b013b920 con 0x7f30b00b44b0 2024-03-25T18:35:27.714 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:35:27.710+0000 7f30b5fc7640 10 monclient(hunting): _renew_subs 2024-03-25T18:35:27.715 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:35:27.710+0000 7f30b5fc7640 10 monclient(hunting): authenticate will time out at 8008.181049s 2024-03-25T18:35:27.715 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:35:27.712+0000 7f30affff640 1 --1- >> v1:172.21.15.115:6789/0 conn(0x7f30b00b4e70 0x7f30b01e25f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.115:6789/0 says I am v1:172.21.15.1:46288/0 (socket says 172.21.15.1:46288) 2024-03-25T18:35:27.715 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:35:27.712+0000 7f30affff640 1 -- 172.21.15.1:0/132449110 learned_addr learned my addr 172.21.15.1:0/132449110 (peer_addr_for_me v1:172.21.15.1:0/0) 2024-03-25T18:35:27.715 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:35:27.712+0000 7f3097fff640 10 client.?.objecter ms_handle_connect 0x7f30b00b4e70 2024-03-25T18:35:27.715 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:35:27.712+0000 7f3097fff640 10 client.?.objecter resend_mon_ops 2024-03-25T18:35:27.731 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:35:27.730+0000 7f3097fff640 10 client.?.objecter ms_handle_connect 0x7f30b00b44b0 2024-03-25T18:35:27.731 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:35:27.730+0000 7f3097fff640 10 client.?.objecter resend_mon_ops 2024-03-25T18:35:27.733 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:35:27.731+0000 7f3097fff640 10 client.?.objecter ms_handle_connect 0x7f30b00b1b00 2024-03-25T18:35:27.733 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:35:27.731+0000 7f3097fff640 10 client.?.objecter resend_mon_ops 2024-03-25T18:35:27.741 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:35:27.739+0000 7f3097fff640 1 -- 172.21.15.1:0/132449110 <== mon.0 v1:172.21.15.1:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4244463303 0 0) 0x7f30b013b6c0 con 0x7f30b00b1b00 2024-03-25T18:35:27.741 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:35:27.739+0000 7f3097fff640 10 monclient(hunting): _negotiate 2024-03-25T18:35:27.741 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:35:27.739+0000 7f3097fff640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:35:27.741 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:35:27.739+0000 7f3097fff640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:35:27.741 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:35:27.739+0000 7f3097fff640 10 monclient(hunting): my global_id is 4599 2024-03-25T18:35:27.741 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:35:27.739+0000 7f3097fff640 1 -- 172.21.15.1:0/132449110 --> v1:172.21.15.1:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f309c004eb0 con 0x7f30b00b1b00 2024-03-25T18:35:27.747 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:35:27.745+0000 7f3097fff640 1 -- 172.21.15.1:0/132449110 <== mon.2 v1:172.21.15.1:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3310247105 0 0) 0x7f30b013b920 con 0x7f30b00b44b0 2024-03-25T18:35:27.747 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:35:27.745+0000 7f3097fff640 10 monclient(hunting): _negotiate 2024-03-25T18:35:27.747 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:35:27.745+0000 7f3097fff640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:35:27.747 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:35:27.745+0000 7f3097fff640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:35:27.747 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:35:27.745+0000 7f3097fff640 10 monclient(hunting): my global_id is 4607 2024-03-25T18:35:27.747 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:35:27.745+0000 7f3097fff640 1 -- 172.21.15.1:0/132449110 --> v1:172.21.15.1:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f30b013b6c0 con 0x7f30b00b44b0 2024-03-25T18:35:27.755 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:35:27.753+0000 7f3097fff640 1 -- 172.21.15.1:0/132449110 <== mon.0 v1:172.21.15.1:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3792149749 0 0) 0x7f309c004eb0 con 0x7f30b00b1b00 2024-03-25T18:35:27.755 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:35:27.753+0000 7f3097fff640 1 -- 172.21.15.1:0/132449110 --> v1:172.21.15.1:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f30b013b920 con 0x7f30b00b1b00 2024-03-25T18:35:27.756 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:35:27.753+0000 7f3097fff640 1 -- 172.21.15.1:0/132449110 <== mon.0 v1:172.21.15.1:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1586965025 0 0) 0x7f30a0001910 con 0x7f30b00b1b00 2024-03-25T18:35:27.756 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:35:27.753+0000 7f3097fff640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:35:27.756 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:35:27.753+0000 7f3097fff640 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2024-03-25T18:35:27.756 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:35:27.753+0000 7f3097fff640 10 monclient(hunting): dump: 2024-03-25T18:35:27.756 INFO:teuthology.misc.health.smithi001.stderr:epoch 1 2024-03-25T18:35:27.756 INFO:teuthology.misc.health.smithi001.stderr:fsid c8ea72cf-9e99-4383-9b52-6c23d365fdfc 2024-03-25T18:35:27.756 INFO:teuthology.misc.health.smithi001.stderr:last_changed 2024-03-25T18:25:50.510519+0000 2024-03-25T18:35:27.756 INFO:teuthology.misc.health.smithi001.stderr:created 2024-03-25T18:25:50.510519+0000 2024-03-25T18:35:27.756 INFO:teuthology.misc.health.smithi001.stderr:min_mon_release 19 (squid) 2024-03-25T18:35:27.756 INFO:teuthology.misc.health.smithi001.stderr:election_strategy: 3 2024-03-25T18:35:27.757 INFO:teuthology.misc.health.smithi001.stderr:0: v1:172.21.15.1:6789/0 mon.a 2024-03-25T18:35:27.757 INFO:teuthology.misc.health.smithi001.stderr:1: v1:172.21.15.115:6789/0 mon.b 2024-03-25T18:35:27.757 INFO:teuthology.misc.health.smithi001.stderr:2: v1:172.21.15.1:6790/0 mon.c 2024-03-25T18:35:27.757 INFO:teuthology.misc.health.smithi001.stderr: 2024-03-25T18:35:27.757 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:35:27.753+0000 7f3097fff640 10 monclient(hunting): _finish_auth 0 2024-03-25T18:35:27.782 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:35:27.780+0000 7f3097fff640 1 -- 172.21.15.1:0/132449110 <== mon.0 v1:172.21.15.1:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 4001032661 0 0) 0x7f30b013b920 con 0x7f30b00b1b00 2024-03-25T18:35:27.782 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:35:27.780+0000 7f3097fff640 1 -- 172.21.15.1:0/132449110 >> v1:172.21.15.1:6790/0 conn(0x7f30b00b44b0 legacy=0x7f30b0140160 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:35:27.782 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:35:27.780+0000 7f3097fff640 1 -- 172.21.15.1:0/132449110 >> v1:172.21.15.115:6789/0 conn(0x7f30b00b4e70 legacy=0x7f30b01e25f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:35:27.782 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:35:27.780+0000 7f3097fff640 10 monclient: _finish_hunting 0 2024-03-25T18:35:27.782 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:35:27.780+0000 7f3097fff640 1 monclient: found mon.a 2024-03-25T18:35:27.782 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:35:27.780+0000 7f3097fff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:35:27.782 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:35:27.780+0000 7f3097fff640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.1:6789/0 2024-03-25T18:35:27.783 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:35:27.780+0000 7f3097fff640 1 -- 172.21.15.1:0/132449110 --> v1:172.21.15.1:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f30b0140870 con 0x7f30b00b1b00 2024-03-25T18:35:27.783 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:35:27.780+0000 7f3097fff640 10 monclient: _finish_auth 0 2024-03-25T18:35:27.783 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:35:27.780+0000 7f3097fff640 10 monclient: _check_auth_tickets 2024-03-25T18:35:27.783 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:35:27.780+0000 7f3097fff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:35:27.783 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:35:27.780+0000 7f30b5fc7640 5 monclient: authenticate success, global_id 4599 2024-03-25T18:35:27.783 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:35:27.780+0000 7f30b5fc7640 10 monclient: _renew_subs 2024-03-25T18:35:27.783 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:35:27.780+0000 7f30b5fc7640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.1:6789/0 2024-03-25T18:35:27.783 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:35:27.780+0000 7f30b5fc7640 1 -- 172.21.15.1:0/132449110 --> v1:172.21.15.1:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f30b01337a0 con 0x7f30b00b1b00 2024-03-25T18:35:27.783 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:35:27.780+0000 7f30b5fc7640 10 client.4599.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T18:35:27.784 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:35:27.780+0000 7f30b5fc7640 10 monclient: _renew_subs 2024-03-25T18:35:27.784 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:35:27.780+0000 7f30b5fc7640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.1:6789/0 2024-03-25T18:35:27.784 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:35:27.780+0000 7f30b5fc7640 1 -- 172.21.15.1:0/132449110 --> v1:172.21.15.1:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f30b013ace0 con 0x7f30b00b1b00 2024-03-25T18:35:27.784 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:35:27.780+0000 7f30b5fc7640 1 librados: init done 2024-03-25T18:35:27.784 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:35:27.781+0000 7f30b5fc7640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T18:35:27.784 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:35:27.781+0000 7f30b5fc7640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:35:27.784 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:35:27.781+0000 7f30b5fc7640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.1:6789/0 2024-03-25T18:35:27.785 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:35:27.781+0000 7f30b5fc7640 1 -- 172.21.15.1:0/132449110 --> v1:172.21.15.1:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f30b004f730 con 0x7f30b00b1b00 2024-03-25T18:35:27.791 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:35:27.789+0000 7f30af7fe640 1 -- 172.21.15.1:0/132449110 >> v1:172.21.15.1:6789/0 conn(0x7f30b00b1b00 legacy=0x7f30b013afb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 16 2024-03-25T18:35:27.791 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:35:27.789+0000 7f30af7fe640 1 -- 172.21.15.1:0/132449110 >> v1:172.21.15.1:6789/0 conn(0x7f30b00b1b00 legacy=0x7f30b013afb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-03-25T18:35:27.791 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:35:27.789+0000 7f30af7fe640 1 --1- 172.21.15.1:0/132449110 >> v1:172.21.15.1:6789/0 conn(0x7f30b00b1b00 0x7f30b013afb0 :-1 s=OPENED pgs=64489 cs=1 l=1).handle_message read tag failed 2024-03-25T18:35:27.792 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:35:27.789+0000 7f30af7fe640 1 --1- 172.21.15.1:0/132449110 >> v1:172.21.15.1:6789/0 conn(0x7f30b00b1b00 0x7f30b013afb0 :-1 s=OPENED pgs=64489 cs=1 l=1).fault on lossy channel, failing 2024-03-25T18:35:27.792 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:35:27.789+0000 7f3097fff640 10 monclient: ms_handle_reset current mon v1:172.21.15.1:6789/0 2024-03-25T18:35:27.792 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:35:27.789+0000 7f3097fff640 10 monclient: _reopen_session rank -1 2024-03-25T18:35:27.792 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:35:27.789+0000 7f3097fff640 1 -- 172.21.15.1:0/132449110 >> v1:172.21.15.1:6789/0 conn(0x7f30b00b1b00 legacy=0x7f30b013afb0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-03-25T18:35:27.792 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:35:27.789+0000 7f3097fff640 10 monclient: _add_conns ranks=[1,0,2] 2024-03-25T18:35:27.792 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:35:27.789+0000 7f3097fff640 10 monclient(hunting): picked mon.b con 0x7f30b00b4e70 addr v1:172.21.15.115:6789/0 2024-03-25T18:35:27.792 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:35:27.789+0000 7f3097fff640 10 monclient(hunting): picked mon.a con 0x7f309c006af0 addr v1:172.21.15.1:6789/0 2024-03-25T18:35:27.792 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:35:27.789+0000 7f3097fff640 10 monclient(hunting): picked mon.c con 0x7f30b00b44b0 addr v1:172.21.15.1:6790/0 2024-03-25T18:35:27.792 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:35:27.789+0000 7f3097fff640 1 -- 172.21.15.1:0/132449110 --> v1:172.21.15.1:6789/0 -- auth(proto 0 30 bytes epoch 1) v1 -- 0x7f30b013b920 con 0x7f309c006af0 2024-03-25T18:35:27.792 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:35:27.789+0000 7f3097fff640 1 -- 172.21.15.1:0/132449110 --> v1:172.21.15.115:6789/0 -- auth(proto 0 30 bytes epoch 1) v1 -- 0x7f309c004eb0 con 0x7f30b00b4e70 2024-03-25T18:35:27.792 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:35:27.789+0000 7f3097fff640 1 -- 172.21.15.1:0/132449110 --> v1:172.21.15.1:6790/0 -- auth(proto 0 30 bytes epoch 1) v1 -- 0x7f309c013a10 con 0x7f30b00b44b0 2024-03-25T18:35:27.792 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:35:27.789+0000 7f3097fff640 10 monclient(hunting): _renew_subs 2024-03-25T18:35:27.793 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:35:27.790+0000 7f3097fff640 10 client.4599.objecter ms_handle_connect 0x7f30b00b44b0 2024-03-25T18:35:27.793 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:35:27.790+0000 7f3097fff640 10 client.4599.objecter resend_mon_ops 2024-03-25T18:35:27.793 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:35:27.790+0000 7f3097fff640 10 client.4599.objecter ms_handle_connect 0x7f30b00b4e70 2024-03-25T18:35:27.793 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:35:27.790+0000 7f3097fff640 10 client.4599.objecter resend_mon_ops 2024-03-25T18:35:28.066 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:35:28.064+0000 7f3097fff640 10 client.4599.objecter ms_handle_connect 0x7f309c006af0 2024-03-25T18:35:28.066 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:35:28.064+0000 7f3097fff640 10 client.4599.objecter resend_mon_ops 2024-03-25T18:35:28.119 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:35:28.117+0000 7f3097fff640 1 -- 172.21.15.1:0/132449110 <== mon.0 v1:172.21.15.1:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2953400984 0 0) 0x7f30b013b920 con 0x7f309c006af0 2024-03-25T18:35:28.119 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:35:28.117+0000 7f3097fff640 10 monclient(hunting): _negotiate 2024-03-25T18:35:28.119 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:35:28.117+0000 7f3097fff640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:35:28.119 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:35:28.117+0000 7f3097fff640 10 monclient(hunting): _init_auth already have auth, reseting 2024-03-25T18:35:28.119 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:35:28.117+0000 7f3097fff640 1 -- 172.21.15.1:0/132449110 --> v1:172.21.15.1:6789/0 -- auth(proto 2 132 bytes epoch 0) v1 -- 0x7f309c012940 con 0x7f309c006af0 2024-03-25T18:35:28.131 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:35:28.129+0000 7f3097fff640 1 -- 172.21.15.1:0/132449110 <== mon.0 v1:172.21.15.1:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 233+0+0 (unknown 4086189496 0 0) 0x7f309c012940 con 0x7f309c006af0 2024-03-25T18:35:28.131 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:35:28.129+0000 7f3097fff640 1 -- 172.21.15.1:0/132449110 >> v1:172.21.15.1:6790/0 conn(0x7f30b00b44b0 legacy=0x7f309c00fdb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:35:28.131 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:35:28.129+0000 7f3097fff640 1 -- 172.21.15.1:0/132449110 >> v1:172.21.15.115:6789/0 conn(0x7f30b00b4e70 legacy=0x7f309c006150 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:35:28.131 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:35:28.129+0000 7f3097fff640 10 monclient: _finish_hunting 0 2024-03-25T18:35:28.132 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:35:28.129+0000 7f3097fff640 1 monclient: found mon.a 2024-03-25T18:35:28.132 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:35:28.129+0000 7f3097fff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:35:28.132 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:35:28.129+0000 7f3097fff640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.1:6789/0 2024-03-25T18:35:28.132 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:35:28.129+0000 7f3097fff640 1 -- 172.21.15.1:0/132449110 --> v1:172.21.15.1:6789/0 -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+,osdmap=0}) v3 -- 0x7f309c014df0 con 0x7f309c006af0 2024-03-25T18:35:28.132 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:35:28.129+0000 7f3097fff640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:35:28.132 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:35:28.129+0000 7f3097fff640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.1:6789/0 2024-03-25T18:35:28.132 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:35:28.129+0000 7f3097fff640 1 -- 172.21.15.1:0/132449110 --> v1:172.21.15.1:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f30b013b920 con 0x7f309c006af0 2024-03-25T18:35:28.132 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:35:28.129+0000 7f3097fff640 10 monclient: _finish_auth 0 2024-03-25T18:35:28.132 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:35:28.129+0000 7f3097fff640 10 monclient: _check_auth_tickets 2024-03-25T18:35:28.132 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:35:28.129+0000 7f3097fff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:35:28.132 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:35:28.129+0000 7f3097fff640 1 -- 172.21.15.1:0/132449110 <== mon.0 v1:172.21.15.1:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1586965025 0 0) 0x7f30a40015d0 con 0x7f309c006af0 2024-03-25T18:35:28.132 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:35:28.129+0000 7f3097fff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:35:28.132 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:35:28.129+0000 7f3097fff640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-25T18:35:28.132 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:35:28.129+0000 7f3097fff640 10 monclient: dump: 2024-03-25T18:35:28.132 INFO:teuthology.misc.health.smithi001.stderr:epoch 1 2024-03-25T18:35:28.132 INFO:teuthology.misc.health.smithi001.stderr:fsid c8ea72cf-9e99-4383-9b52-6c23d365fdfc 2024-03-25T18:35:28.133 INFO:teuthology.misc.health.smithi001.stderr:last_changed 2024-03-25T18:25:50.510519+0000 2024-03-25T18:35:28.133 INFO:teuthology.misc.health.smithi001.stderr:created 2024-03-25T18:25:50.510519+0000 2024-03-25T18:35:28.133 INFO:teuthology.misc.health.smithi001.stderr:min_mon_release 19 (squid) 2024-03-25T18:35:28.133 INFO:teuthology.misc.health.smithi001.stderr:election_strategy: 3 2024-03-25T18:35:28.133 INFO:teuthology.misc.health.smithi001.stderr:0: v1:172.21.15.1:6789/0 mon.a 2024-03-25T18:35:28.133 INFO:teuthology.misc.health.smithi001.stderr:1: v1:172.21.15.115:6789/0 mon.b 2024-03-25T18:35:28.133 INFO:teuthology.misc.health.smithi001.stderr:2: v1:172.21.15.1:6790/0 mon.c 2024-03-25T18:35:28.133 INFO:teuthology.misc.health.smithi001.stderr: 2024-03-25T18:35:28.475 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:35:28.473+0000 7f3097fff640 1 -- 172.21.15.1:0/132449110 <== mon.0 v1:172.21.15.1:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f30a4001960 con 0x7f309c006af0 2024-03-25T18:35:28.475 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:35:28.473+0000 7f3097fff640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:35:28.475 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:35:28.473+0000 7f30adffb640 4 set_mon_vals no callback set 2024-03-25T18:35:28.476 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:35:28.474+0000 7f3097fff640 1 -- 172.21.15.1:0/132449110 <== mon.0 v1:172.21.15.1:6789/0 5 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 21787975 0 0) 0x7f30a401a650 con 0x7f309c006af0 2024-03-25T18:35:28.476 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:35:28.474+0000 7f30af7fe640 10 monclient: get_auth_request con 0x7f30b00b1b00 auth_method 0 2024-03-25T18:35:28.476 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:35:28.475+0000 7f3097fff640 10 client.4599.objecter ms_handle_connect 0x7f30b00b1b00 2024-03-25T18:35:28.484 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:35:28.482+0000 7f3097fff640 1 -- 172.21.15.1:0/132449110 <== mon.0 v1:172.21.15.1:6789/0 6 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 996254230 0 0) 0x7f30a4053fc0 con 0x7f309c006af0 2024-03-25T18:35:28.484 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:35:28.482+0000 7f3097fff640 10 client.4599.objecter ms_dispatch 0x7f30b0000b90 osd_map(10..10 src has 1..10) v4 2024-03-25T18:35:28.484 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:35:28.482+0000 7f3097fff640 3 client.4599.objecter handle_osd_map got epochs [10,10] > 0 2024-03-25T18:35:28.484 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:35:28.482+0000 7f3097fff640 3 client.4599.objecter handle_osd_map decoding full epoch 10 2024-03-25T18:35:28.484 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:35:28.482+0000 7f3097fff640 20 client.4599.objecter dump_active .. 0 homeless 2024-03-25T18:35:28.484 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:35:28.482+0000 7f3097fff640 1 -- 172.21.15.1:0/132449110 <== mon.0 v1:172.21.15.1:6789/0 7 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f30a404f900 con 0x7f309c006af0 2024-03-25T18:35:28.484 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:35:28.482+0000 7f3097fff640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:35:28.484 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:35:28.482+0000 7f3097fff640 10 monclient: _finish_command 1 = system:0 2024-03-25T18:35:28.744 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:35:28.742+0000 7f30b5fc7640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:35:28.744 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:35:28.742+0000 7f30b5fc7640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:35:28.744 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:35:28.742+0000 7f30b5fc7640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.1:6789/0 2024-03-25T18:35:28.744 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:35:28.742+0000 7f30b5fc7640 1 -- 172.21.15.1:0/132449110 --> v1:172.21.15.1:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f30b00aa090 con 0x7f309c006af0 2024-03-25T18:35:28.785 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:35:28.783+0000 7f3097fff640 1 -- 172.21.15.1:0/132449110 <== mon.0 v1:172.21.15.1:6789/0 8 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 4092213348) 0x7f30a4051a00 con 0x7f309c006af0 2024-03-25T18:35:28.785 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:35:28.783+0000 7f3097fff640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:35:28.785 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:35:28.783+0000 7f3097fff640 10 monclient: _finish_command 2 = system:0 2024-03-25T18:35:28.785 INFO:teuthology.misc.health.smithi001.stdout: 2024-03-25T18:35:28.785 INFO:teuthology.misc.health.smithi001.stdout:{"epoch":10,"fsid":"c8ea72cf-9e99-4383-9b52-6c23d365fdfc","created":"2024-03-25T18:26:38.856638+0000","modified":"2024-03-25T18:26:44.224208+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T18:26:44.224208+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"b9f34922-3a84-49be-8426-807c1eceaefc","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"42bf306f-55e3-4634-9874-8b4b73ac6644","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"292cb23a-18a9-4207-a7cc-4d5727feaec0","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"45e0e9ab-898d-4673-9590-b4117bc5e84a","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"831a5a09-73ab-4a61-9872-088e76bdbc66","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"88cd903d-3b55-4840-9396-6e3cfe005282","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"89db7355-bc9d-46f4-8ab0-e8d9e744e026","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"cac47bee-0914-4f08-896f-348838882866","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T18:35:28.788 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:35:28.785+0000 7f30b5fc7640 10 librados: watch_flush enter 2024-03-25T18:35:28.788 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:35:28.786+0000 7f30b5fc7640 10 librados: watch_flush exit 2024-03-25T18:35:28.788 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:35:28.786+0000 7f30b5fc7640 20 client.4599.objecter shutdown clearing up homeless session... 2024-03-25T18:35:28.788 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:35:28.786+0000 7f30b5fc7640 10 client.4599.objecter successfully canceled tick 2024-03-25T18:35:28.788 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:35:28.786+0000 7f30b5fc7640 1 -- 172.21.15.1:0/132449110 >> v1:172.21.15.115:6800/4237379176 conn(0x7f30b00b1b00 legacy=0x7f309c07bca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:35:28.788 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:35:28.786+0000 7f30b5fc7640 10 monclient: shutdown 2024-03-25T18:35:28.788 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:35:28.786+0000 7f30b5fc7640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:35:28.788 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:35:28.786+0000 7f30b5fc7640 1 -- 172.21.15.1:0/132449110 >> v1:172.21.15.1:6789/0 conn(0x7f309c006af0 legacy=0x7f309c00c310 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:35:28.788 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:35:28.786+0000 7f30b5fc7640 1 -- 172.21.15.1:0/132449110 shutdown_connections 2024-03-25T18:35:28.788 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:35:28.786+0000 7f30b5fc7640 1 -- 172.21.15.1:0/132449110 >> 172.21.15.1:0/132449110 conn(0x7f30b00617a0 msgr2=0x7f30b00b59e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:35:28.788 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:35:28.786+0000 7f30b5fc7640 1 -- 172.21.15.1:0/132449110 shutdown_connections 2024-03-25T18:35:28.789 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:35:28.786+0000 7f30b5fc7640 1 -- 172.21.15.1:0/132449110 wait complete. 2024-03-25T18:35:28.789 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:35:28.786+0000 7f30b5fc7640 1 librados: shutdown 2024-03-25T18:35:28.800 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T18:35:35.607 DEBUG:teuthology.orchestra.run.smithi001:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T18:35:35.731 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:35:35.729+0000 7f9defbaf640 10 monclient: get_monmap_and_config 2024-03-25T18:35:35.731 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:35:35.729+0000 7f9defbaf640 10 monclient: build_initial_monmap 2024-03-25T18:35:35.731 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:35:35.729+0000 7f9defbaf640 10 monclient: monmap: 2024-03-25T18:35:35.731 INFO:teuthology.misc.health.smithi001.stderr:epoch 0 2024-03-25T18:35:35.731 INFO:teuthology.misc.health.smithi001.stderr:fsid c8ea72cf-9e99-4383-9b52-6c23d365fdfc 2024-03-25T18:35:35.731 INFO:teuthology.misc.health.smithi001.stderr:last_changed 2024-03-25T18:35:35.731362+0000 2024-03-25T18:35:35.731 INFO:teuthology.misc.health.smithi001.stderr:created 2024-03-25T18:35:35.731362+0000 2024-03-25T18:35:35.731 INFO:teuthology.misc.health.smithi001.stderr:min_mon_release 0 (unknown) 2024-03-25T18:35:35.732 INFO:teuthology.misc.health.smithi001.stderr:election_strategy: 3 2024-03-25T18:35:35.732 INFO:teuthology.misc.health.smithi001.stderr:0: v1:172.21.15.1:6789/0 mon.noname-a 2024-03-25T18:35:35.732 INFO:teuthology.misc.health.smithi001.stderr:1: v1:172.21.15.115:6789/0 mon.noname-c 2024-03-25T18:35:35.732 INFO:teuthology.misc.health.smithi001.stderr:2: v1:172.21.15.1:6790/0 mon.noname-b 2024-03-25T18:35:35.732 INFO:teuthology.misc.health.smithi001.stderr: 2024-03-25T18:35:35.733 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:35:35.730+0000 7f9defbaf640 1 Processor -- start 2024-03-25T18:35:35.733 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:35:35.730+0000 7f9defbaf640 1 -- start start 2024-03-25T18:35:35.733 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:35:35.730+0000 7f9defbaf640 10 monclient: init 2024-03-25T18:35:35.733 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:35:35.731+0000 7f9defbaf640 10 monclient: _reopen_session rank -1 2024-03-25T18:35:35.733 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:35:35.731+0000 7f9defbaf640 10 monclient: _add_conns ranks=[1,0,2] 2024-03-25T18:35:35.733 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:35:35.731+0000 7f9defbaf640 10 monclient(hunting): picked mon.noname-c con 0x7f9de80b4e70 addr v1:172.21.15.115:6789/0 2024-03-25T18:35:35.733 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:35:35.731+0000 7f9defbaf640 10 monclient(hunting): picked mon.noname-a con 0x7f9de80b44b0 addr v1:172.21.15.1:6789/0 2024-03-25T18:35:35.733 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:35:35.731+0000 7f9defbaf640 10 monclient(hunting): picked mon.noname-b con 0x7f9de80b1b00 addr v1:172.21.15.1:6790/0 2024-03-25T18:35:35.733 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:35:35.731+0000 7f9defbaf640 1 -- --> v1:172.21.15.1:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9de80a96f0 con 0x7f9de80b44b0 2024-03-25T18:35:35.734 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:35:35.731+0000 7f9defbaf640 1 -- --> v1:172.21.15.115:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9de80ab0e0 con 0x7f9de80b4e70 2024-03-25T18:35:35.734 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:35:35.731+0000 7f9defbaf640 1 -- --> v1:172.21.15.1:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9de8145f70 con 0x7f9de80b1b00 2024-03-25T18:35:35.734 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:35:35.731+0000 7f9defbaf640 10 monclient(hunting): _renew_subs 2024-03-25T18:35:35.734 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:35:35.731+0000 7f9defbaf640 10 monclient(hunting): authenticate will time out at 8016.202165s 2024-03-25T18:35:35.752 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:35:35.751+0000 7f9ded14a640 1 --1- >> v1:172.21.15.1:6789/0 conn(0x7f9de80b44b0 0x7f9de8145860 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.1:6789/0 says I am v1:172.21.15.1:49970/0 (socket says 172.21.15.1:49970) 2024-03-25T18:35:35.752 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:35:35.751+0000 7f9ded14a640 1 -- 172.21.15.1:0/2218888055 learned_addr learned my addr 172.21.15.1:0/2218888055 (peer_addr_for_me v1:172.21.15.1:0/0) 2024-03-25T18:35:35.756 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:35:35.754+0000 7f9dec949640 1 -- 172.21.15.1:0/2218888055 <== mon.1 v1:172.21.15.115:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4003646629 0 0) 0x7f9de80ab0e0 con 0x7f9de80b4e70 2024-03-25T18:35:35.756 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:35:35.754+0000 7f9dec949640 10 monclient(hunting): _negotiate 2024-03-25T18:35:35.756 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:35:35.754+0000 7f9dec949640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:35:35.756 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:35:35.754+0000 7f9dec949640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:35:35.756 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:35:35.754+0000 7f9dec949640 10 monclient(hunting): my global_id is 4606 2024-03-25T18:35:35.756 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:35:35.754+0000 7f9dec949640 1 -- 172.21.15.1:0/2218888055 --> v1:172.21.15.115:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f9dd4003520 con 0x7f9de80b4e70 2024-03-25T18:35:35.761 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:35:35.759+0000 7f9dec949640 1 -- 172.21.15.1:0/2218888055 <== mon.1 v1:172.21.15.115:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1337710865 0 0) 0x7f9dd4003520 con 0x7f9de80b4e70 2024-03-25T18:35:35.761 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:35:35.759+0000 7f9dec949640 1 -- 172.21.15.1:0/2218888055 >> v1:172.21.15.1:6790/0 conn(0x7f9de80b1b00 legacy=0x7f9de814dc00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-25T18:35:35.761 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:35:35.759+0000 7f9dec949640 1 -- 172.21.15.1:0/2218888055 >> v1:172.21.15.1:6789/0 conn(0x7f9de80b44b0 legacy=0x7f9de8145860 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:35:35.761 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:35:35.759+0000 7f9dec949640 10 monclient: _finish_hunting 0 2024-03-25T18:35:35.762 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:35:35.759+0000 7f9dec949640 1 monclient: found mon.noname-c 2024-03-25T18:35:35.762 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:35:35.759+0000 7f9dec949640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:35:35.762 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:35:35.759+0000 7f9dec949640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.115:6789/0 2024-03-25T18:35:35.762 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:35:35.759+0000 7f9dec949640 1 -- 172.21.15.1:0/2218888055 --> v1:172.21.15.115:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9de80a9b80 con 0x7f9de80b4e70 2024-03-25T18:35:35.762 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:35:35.759+0000 7f9dec949640 10 monclient: _finish_auth 0 2024-03-25T18:35:35.762 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:35:35.759+0000 7f9dec949640 10 monclient: _check_auth_tickets 2024-03-25T18:35:35.762 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:35:35.759+0000 7f9dec949640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:35:35.762 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:35:35.759+0000 7f9dec949640 1 -- 172.21.15.1:0/2218888055 <== mon.1 v1:172.21.15.115:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1586965025 0 0) 0x7f9de4001690 con 0x7f9de80b4e70 2024-03-25T18:35:35.762 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:35:35.759+0000 7f9defbaf640 5 monclient: authenticate success, global_id 4606 2024-03-25T18:35:35.762 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:35:35.759+0000 7f9defbaf640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T18:35:35.762 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:35:35.759+0000 7f9dec949640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:35:35.762 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:35:35.759+0000 7f9dec949640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-03-25T18:35:35.762 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:35:35.759+0000 7f9dec949640 10 monclient: dump: 2024-03-25T18:35:35.762 INFO:teuthology.misc.health.smithi001.stderr:epoch 1 2024-03-25T18:35:35.762 INFO:teuthology.misc.health.smithi001.stderr:fsid c8ea72cf-9e99-4383-9b52-6c23d365fdfc 2024-03-25T18:35:35.762 INFO:teuthology.misc.health.smithi001.stderr:last_changed 2024-03-25T18:25:50.510519+0000 2024-03-25T18:35:35.763 INFO:teuthology.misc.health.smithi001.stderr:created 2024-03-25T18:25:50.510519+0000 2024-03-25T18:35:35.763 INFO:teuthology.misc.health.smithi001.stderr:min_mon_release 19 (squid) 2024-03-25T18:35:35.763 INFO:teuthology.misc.health.smithi001.stderr:election_strategy: 3 2024-03-25T18:35:35.763 INFO:teuthology.misc.health.smithi001.stderr:0: v1:172.21.15.1:6789/0 mon.a 2024-03-25T18:35:35.763 INFO:teuthology.misc.health.smithi001.stderr:1: v1:172.21.15.115:6789/0 mon.b 2024-03-25T18:35:35.763 INFO:teuthology.misc.health.smithi001.stderr:2: v1:172.21.15.1:6790/0 mon.c 2024-03-25T18:35:35.763 INFO:teuthology.misc.health.smithi001.stderr: 2024-03-25T18:35:35.763 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:35:35.759+0000 7f9defbaf640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T18:35:35.778 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:35:35.776+0000 7f9dec949640 1 -- 172.21.15.1:0/2218888055 <== mon.1 v1:172.21.15.115:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f9de4001860 con 0x7f9de80b4e70 2024-03-25T18:35:35.778 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:35:35.776+0000 7f9dec949640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:35:35.778 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:35:35.776+0000 7f9dec949640 1 -- 172.21.15.1:0/2218888055 <== mon.1 v1:172.21.15.115:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1586965025 0 0) 0x7f9de4001cb0 con 0x7f9de80b4e70 2024-03-25T18:35:35.778 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:35:35.776+0000 7f9dec949640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:35:35.778 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:35:35.776+0000 7f9dec949640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-25T18:35:35.778 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:35:35.776+0000 7f9dec949640 10 monclient: dump: 2024-03-25T18:35:35.778 INFO:teuthology.misc.health.smithi001.stderr:epoch 1 2024-03-25T18:35:35.778 INFO:teuthology.misc.health.smithi001.stderr:fsid c8ea72cf-9e99-4383-9b52-6c23d365fdfc 2024-03-25T18:35:35.779 INFO:teuthology.misc.health.smithi001.stderr:last_changed 2024-03-25T18:25:50.510519+0000 2024-03-25T18:35:35.779 INFO:teuthology.misc.health.smithi001.stderr:created 2024-03-25T18:25:50.510519+0000 2024-03-25T18:35:35.779 INFO:teuthology.misc.health.smithi001.stderr:min_mon_release 19 (squid) 2024-03-25T18:35:35.779 INFO:teuthology.misc.health.smithi001.stderr:election_strategy: 3 2024-03-25T18:35:35.779 INFO:teuthology.misc.health.smithi001.stderr:0: v1:172.21.15.1:6789/0 mon.a 2024-03-25T18:35:35.779 INFO:teuthology.misc.health.smithi001.stderr:1: v1:172.21.15.115:6789/0 mon.b 2024-03-25T18:35:35.779 INFO:teuthology.misc.health.smithi001.stderr:2: v1:172.21.15.1:6790/0 mon.c 2024-03-25T18:35:35.779 INFO:teuthology.misc.health.smithi001.stderr: 2024-03-25T18:35:35.779 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:35:35.776+0000 7f9defbaf640 10 monclient: get_monmap_and_config success 2024-03-25T18:35:35.779 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:35:35.776+0000 7f9defbaf640 4 set_mon_vals no callback set 2024-03-25T18:35:35.781 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:35:35.777+0000 7f9defbaf640 10 monclient: shutdown 2024-03-25T18:35:35.782 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:35:35.777+0000 7f9defbaf640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:35:35.782 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:35:35.777+0000 7f9defbaf640 1 -- 172.21.15.1:0/2218888055 >> v1:172.21.15.115:6789/0 conn(0x7f9de80b4e70 legacy=0x7f9de813f940 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:35:35.782 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:35:35.777+0000 7f9defbaf640 1 -- 172.21.15.1:0/2218888055 shutdown_connections 2024-03-25T18:35:35.782 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:35:35.777+0000 7f9defbaf640 1 -- 172.21.15.1:0/2218888055 >> 172.21.15.1:0/2218888055 conn(0x7f9de80617a0 msgr2=0x7f9de8061bb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:35:35.782 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:35:35.777+0000 7f9defbaf640 1 -- 172.21.15.1:0/2218888055 shutdown_connections 2024-03-25T18:35:35.782 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:35:35.777+0000 7f9defbaf640 1 -- 172.21.15.1:0/2218888055 wait complete. 2024-03-25T18:35:35.782 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:35:35.778+0000 7f9defbaf640 10 monclient: build_initial_monmap 2024-03-25T18:35:35.782 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:35:35.778+0000 7f9defbaf640 10 monclient: monmap: 2024-03-25T18:35:35.782 INFO:teuthology.misc.health.smithi001.stderr:epoch 0 2024-03-25T18:35:35.782 INFO:teuthology.misc.health.smithi001.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T18:35:35.782 INFO:teuthology.misc.health.smithi001.stderr:last_changed 0.000000 2024-03-25T18:35:35.782 INFO:teuthology.misc.health.smithi001.stderr:created 0.000000 2024-03-25T18:35:35.782 INFO:teuthology.misc.health.smithi001.stderr:min_mon_release 0 (unknown) 2024-03-25T18:35:35.782 INFO:teuthology.misc.health.smithi001.stderr:election_strategy: 1 2024-03-25T18:35:35.783 INFO:teuthology.misc.health.smithi001.stderr:0: v1:172.21.15.1:6789/0 mon.noname-a 2024-03-25T18:35:35.783 INFO:teuthology.misc.health.smithi001.stderr:1: v1:172.21.15.115:6789/0 mon.noname-b 2024-03-25T18:35:35.783 INFO:teuthology.misc.health.smithi001.stderr:2: v1:172.21.15.1:6790/0 mon.noname-c 2024-03-25T18:35:35.783 INFO:teuthology.misc.health.smithi001.stderr: 2024-03-25T18:35:35.783 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:35:35.778+0000 7f9defbaf640 1 librados: starting msgr at 2024-03-25T18:35:35.783 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:35:35.778+0000 7f9defbaf640 1 librados: starting objecter 2024-03-25T18:35:35.783 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:35:35.778+0000 7f9defbaf640 1 Processor -- start 2024-03-25T18:35:35.783 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:35:35.779+0000 7f9defbaf640 1 -- start start 2024-03-25T18:35:35.783 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:35:35.779+0000 7f9defbaf640 1 librados: setting wanted keys 2024-03-25T18:35:35.783 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:35:35.779+0000 7f9defbaf640 1 librados: calling monclient init 2024-03-25T18:35:35.783 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:35:35.779+0000 7f9defbaf640 10 monclient: init 2024-03-25T18:35:35.783 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:35:35.779+0000 7f9defbaf640 10 monclient: _reopen_session rank -1 2024-03-25T18:35:35.783 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:35:35.779+0000 7f9defbaf640 10 monclient: _add_conns ranks=[1,2,0] 2024-03-25T18:35:35.783 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:35:35.779+0000 7f9defbaf640 10 monclient(hunting): picked mon.noname-b con 0x7f9de80b1b00 addr v1:172.21.15.115:6789/0 2024-03-25T18:35:35.783 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:35:35.779+0000 7f9defbaf640 10 monclient(hunting): picked mon.noname-c con 0x7f9de80b44b0 addr v1:172.21.15.1:6790/0 2024-03-25T18:35:35.783 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:35:35.779+0000 7f9defbaf640 10 monclient(hunting): picked mon.noname-a con 0x7f9de80b4e70 addr v1:172.21.15.1:6789/0 2024-03-25T18:35:35.784 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:35:35.779+0000 7f9defbaf640 1 -- --> v1:172.21.15.1:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9de806fc40 con 0x7f9de80b4e70 2024-03-25T18:35:35.784 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:35:35.779+0000 7f9defbaf640 1 -- --> v1:172.21.15.115:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9de8072080 con 0x7f9de80b1b00 2024-03-25T18:35:35.784 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:35:35.779+0000 7f9defbaf640 1 -- --> v1:172.21.15.1:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9de8070160 con 0x7f9de80b44b0 2024-03-25T18:35:35.784 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:35:35.779+0000 7f9defbaf640 10 monclient(hunting): _renew_subs 2024-03-25T18:35:35.784 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:35:35.779+0000 7f9defbaf640 10 monclient(hunting): authenticate will time out at 8016.250568s 2024-03-25T18:35:35.785 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:35:35.783+0000 7f9dee14c640 1 --1- >> v1:172.21.15.1:6789/0 conn(0x7f9de80b4e70 0x7f9de8135250 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.1:6789/0 says I am v1:172.21.15.1:50000/0 (socket says 172.21.15.1:50000) 2024-03-25T18:35:35.785 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:35:35.783+0000 7f9dee14c640 1 -- 172.21.15.1:0/2961177610 learned_addr learned my addr 172.21.15.1:0/2961177610 (peer_addr_for_me v1:172.21.15.1:0/0) 2024-03-25T18:35:35.789 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:35:35.787+0000 7f9dde7fc640 10 client.?.objecter ms_handle_connect 0x7f9de80b4e70 2024-03-25T18:35:35.789 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:35:35.787+0000 7f9dde7fc640 10 client.?.objecter resend_mon_ops 2024-03-25T18:35:35.791 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:35:35.790+0000 7f9dde7fc640 10 client.?.objecter ms_handle_connect 0x7f9de80b44b0 2024-03-25T18:35:35.791 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:35:35.790+0000 7f9dde7fc640 10 client.?.objecter resend_mon_ops 2024-03-25T18:35:35.799 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:35:35.798+0000 7f9dde7fc640 1 -- 172.21.15.1:0/2961177610 <== mon.0 v1:172.21.15.1:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 993659646 0 0) 0x7f9de806fc40 con 0x7f9de80b4e70 2024-03-25T18:35:35.800 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:35:35.798+0000 7f9dde7fc640 10 monclient(hunting): _negotiate 2024-03-25T18:35:35.800 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:35:35.798+0000 7f9dde7fc640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:35:35.800 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:35:35.798+0000 7f9dde7fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:35:35.800 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:35:35.798+0000 7f9dde7fc640 10 monclient(hunting): my global_id is 4605 2024-03-25T18:35:35.800 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:35:35.798+0000 7f9dde7fc640 1 -- 172.21.15.1:0/2961177610 --> v1:172.21.15.1:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f9dcc004eb0 con 0x7f9de80b4e70 2024-03-25T18:35:35.803 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:35:35.801+0000 7f9dde7fc640 1 -- 172.21.15.1:0/2961177610 <== mon.2 v1:172.21.15.1:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3039699923 0 0) 0x7f9de8070160 con 0x7f9de80b44b0 2024-03-25T18:35:35.803 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:35:35.801+0000 7f9dde7fc640 10 monclient(hunting): _negotiate 2024-03-25T18:35:35.803 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:35:35.801+0000 7f9dde7fc640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:35:35.803 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:35:35.801+0000 7f9dde7fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:35:35.803 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:35:35.801+0000 7f9dde7fc640 10 monclient(hunting): my global_id is 4610 2024-03-25T18:35:35.803 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:35:35.801+0000 7f9dde7fc640 1 -- 172.21.15.1:0/2961177610 --> v1:172.21.15.1:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f9de806fc40 con 0x7f9de80b44b0 2024-03-25T18:35:35.808 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:35:35.806+0000 7f9dde7fc640 10 client.?.objecter ms_handle_connect 0x7f9de80b1b00 2024-03-25T18:35:35.808 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:35:35.806+0000 7f9dde7fc640 10 client.?.objecter resend_mon_ops 2024-03-25T18:35:35.811 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:35:35.809+0000 7f9dde7fc640 1 -- 172.21.15.1:0/2961177610 <== mon.0 v1:172.21.15.1:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1417655508 0 0) 0x7f9dcc004eb0 con 0x7f9de80b4e70 2024-03-25T18:35:35.811 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:35:35.809+0000 7f9dde7fc640 1 -- 172.21.15.1:0/2961177610 --> v1:172.21.15.1:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f9de8070160 con 0x7f9de80b4e70 2024-03-25T18:35:35.811 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:35:35.809+0000 7f9dde7fc640 1 -- 172.21.15.1:0/2961177610 <== mon.0 v1:172.21.15.1:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1586965025 0 0) 0x7f9de0001c60 con 0x7f9de80b4e70 2024-03-25T18:35:35.811 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:35:35.809+0000 7f9dde7fc640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:35:35.811 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:35:35.809+0000 7f9dde7fc640 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2024-03-25T18:35:35.811 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:35:35.809+0000 7f9dde7fc640 10 monclient(hunting): dump: 2024-03-25T18:35:35.812 INFO:teuthology.misc.health.smithi001.stderr:epoch 1 2024-03-25T18:35:35.812 INFO:teuthology.misc.health.smithi001.stderr:fsid c8ea72cf-9e99-4383-9b52-6c23d365fdfc 2024-03-25T18:35:35.812 INFO:teuthology.misc.health.smithi001.stderr:last_changed 2024-03-25T18:25:50.510519+0000 2024-03-25T18:35:35.812 INFO:teuthology.misc.health.smithi001.stderr:created 2024-03-25T18:25:50.510519+0000 2024-03-25T18:35:35.812 INFO:teuthology.misc.health.smithi001.stderr:min_mon_release 19 (squid) 2024-03-25T18:35:35.812 INFO:teuthology.misc.health.smithi001.stderr:election_strategy: 3 2024-03-25T18:35:35.812 INFO:teuthology.misc.health.smithi001.stderr:0: v1:172.21.15.1:6789/0 mon.a 2024-03-25T18:35:35.812 INFO:teuthology.misc.health.smithi001.stderr:1: v1:172.21.15.115:6789/0 mon.b 2024-03-25T18:35:35.812 INFO:teuthology.misc.health.smithi001.stderr:2: v1:172.21.15.1:6790/0 mon.c 2024-03-25T18:35:35.812 INFO:teuthology.misc.health.smithi001.stderr: 2024-03-25T18:35:35.812 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:35:35.809+0000 7f9dde7fc640 10 monclient(hunting): _finish_auth 0 2024-03-25T18:35:35.814 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:35:35.813+0000 7f9dde7fc640 1 -- 172.21.15.1:0/2961177610 <== mon.2 v1:172.21.15.1:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1781116266 0 0) 0x7f9de806fc40 con 0x7f9de80b44b0 2024-03-25T18:35:35.814 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:35:35.813+0000 7f9dde7fc640 1 -- 172.21.15.1:0/2961177610 --> v1:172.21.15.1:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f9dcc004eb0 con 0x7f9de80b44b0 2024-03-25T18:35:35.815 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:35:35.813+0000 7f9dde7fc640 1 -- 172.21.15.1:0/2961177610 <== mon.2 v1:172.21.15.1:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1586965025 0 0) 0x7f9dd80018a0 con 0x7f9de80b44b0 2024-03-25T18:35:35.815 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:35:35.813+0000 7f9dde7fc640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:35:35.815 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:35:35.813+0000 7f9dde7fc640 10 monclient(hunting): got monmap 1 from mon.c (according to old e1) 2024-03-25T18:35:35.815 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:35:35.813+0000 7f9dde7fc640 10 monclient(hunting): dump: 2024-03-25T18:35:35.815 INFO:teuthology.misc.health.smithi001.stderr:epoch 1 2024-03-25T18:35:35.815 INFO:teuthology.misc.health.smithi001.stderr:fsid c8ea72cf-9e99-4383-9b52-6c23d365fdfc 2024-03-25T18:35:35.815 INFO:teuthology.misc.health.smithi001.stderr:last_changed 2024-03-25T18:25:50.510519+0000 2024-03-25T18:35:35.815 INFO:teuthology.misc.health.smithi001.stderr:created 2024-03-25T18:25:50.510519+0000 2024-03-25T18:35:35.815 INFO:teuthology.misc.health.smithi001.stderr:min_mon_release 19 (squid) 2024-03-25T18:35:35.815 INFO:teuthology.misc.health.smithi001.stderr:election_strategy: 3 2024-03-25T18:35:35.815 INFO:teuthology.misc.health.smithi001.stderr:0: v1:172.21.15.1:6789/0 mon.a 2024-03-25T18:35:35.815 INFO:teuthology.misc.health.smithi001.stderr:1: v1:172.21.15.115:6789/0 mon.b 2024-03-25T18:35:35.815 INFO:teuthology.misc.health.smithi001.stderr:2: v1:172.21.15.1:6790/0 mon.c 2024-03-25T18:35:35.815 INFO:teuthology.misc.health.smithi001.stderr: 2024-03-25T18:35:35.820 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:35:35.818+0000 7f9dde7fc640 1 -- 172.21.15.1:0/2961177610 <== mon.2 v1:172.21.15.1:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 3160236605 0 0) 0x7f9dcc004eb0 con 0x7f9de80b44b0 2024-03-25T18:35:35.820 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:35:35.818+0000 7f9dde7fc640 1 -- 172.21.15.1:0/2961177610 >> v1:172.21.15.115:6789/0 conn(0x7f9de80b1b00 legacy=0x7f9de81db930 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:35:35.820 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:35:35.818+0000 7f9dde7fc640 1 -- 172.21.15.1:0/2961177610 >> v1:172.21.15.1:6789/0 conn(0x7f9de80b4e70 legacy=0x7f9de8135250 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:35:35.820 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:35:35.818+0000 7f9dde7fc640 10 monclient: _finish_hunting 0 2024-03-25T18:35:35.820 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:35:35.818+0000 7f9dde7fc640 1 monclient: found mon.c 2024-03-25T18:35:35.820 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:35:35.818+0000 7f9dde7fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:35:35.820 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:35:35.818+0000 7f9dde7fc640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.1:6790/0 2024-03-25T18:35:35.821 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:35:35.818+0000 7f9dde7fc640 1 -- 172.21.15.1:0/2961177610 --> v1:172.21.15.1:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9de80702f0 con 0x7f9de80b44b0 2024-03-25T18:35:35.821 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:35:35.818+0000 7f9dde7fc640 10 monclient: _finish_auth 0 2024-03-25T18:35:35.821 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:35:35.818+0000 7f9dde7fc640 10 monclient: _check_auth_tickets 2024-03-25T18:35:35.821 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:35:35.818+0000 7f9dde7fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:35:35.821 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:35:35.818+0000 7f9defbaf640 5 monclient: authenticate success, global_id 4610 2024-03-25T18:35:35.821 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:35:35.818+0000 7f9defbaf640 10 monclient: _renew_subs 2024-03-25T18:35:35.821 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:35:35.818+0000 7f9defbaf640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.1:6790/0 2024-03-25T18:35:35.821 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:35:35.818+0000 7f9defbaf640 1 -- 172.21.15.1:0/2961177610 --> v1:172.21.15.1:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9de806f760 con 0x7f9de80b44b0 2024-03-25T18:35:35.821 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:35:35.818+0000 7f9defbaf640 10 client.4610.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T18:35:35.821 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:35:35.818+0000 7f9defbaf640 10 monclient: _renew_subs 2024-03-25T18:35:35.821 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:35:35.818+0000 7f9defbaf640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.1:6790/0 2024-03-25T18:35:35.821 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:35:35.818+0000 7f9defbaf640 1 -- 172.21.15.1:0/2961177610 --> v1:172.21.15.1:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f9de81dc220 con 0x7f9de80b44b0 2024-03-25T18:35:35.821 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:35:35.818+0000 7f9defbaf640 1 librados: init done 2024-03-25T18:35:35.822 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:35:35.820+0000 7f9defbaf640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T18:35:35.822 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:35:35.820+0000 7f9defbaf640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:35:35.822 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:35:35.820+0000 7f9defbaf640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.1:6790/0 2024-03-25T18:35:35.822 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:35:35.820+0000 7f9defbaf640 1 -- 172.21.15.1:0/2961177610 --> v1:172.21.15.1:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f9de806fab0 con 0x7f9de80b44b0 2024-03-25T18:35:36.148 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:35:36.146+0000 7f9dde7fc640 1 -- 172.21.15.1:0/2961177610 <== mon.2 v1:172.21.15.1:6790/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f9dd8001cd0 con 0x7f9de80b44b0 2024-03-25T18:35:36.148 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:35:36.146+0000 7f9dde7fc640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:35:36.148 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:35:36.146+0000 7f9dde7fc640 1 -- 172.21.15.1:0/2961177610 <== mon.2 v1:172.21.15.1:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1586965025 0 0) 0x7f9dd8004980 con 0x7f9de80b44b0 2024-03-25T18:35:36.148 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:35:36.146+0000 7f9dde7fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:35:36.148 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:35:36.146+0000 7f9dde7fc640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-25T18:35:36.148 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:35:36.146+0000 7f9dde7fc640 10 monclient: dump: 2024-03-25T18:35:36.148 INFO:teuthology.misc.health.smithi001.stderr:epoch 1 2024-03-25T18:35:36.148 INFO:teuthology.misc.health.smithi001.stderr:fsid c8ea72cf-9e99-4383-9b52-6c23d365fdfc 2024-03-25T18:35:36.148 INFO:teuthology.misc.health.smithi001.stderr:last_changed 2024-03-25T18:25:50.510519+0000 2024-03-25T18:35:36.148 INFO:teuthology.misc.health.smithi001.stderr:created 2024-03-25T18:25:50.510519+0000 2024-03-25T18:35:36.149 INFO:teuthology.misc.health.smithi001.stderr:min_mon_release 19 (squid) 2024-03-25T18:35:36.149 INFO:teuthology.misc.health.smithi001.stderr:election_strategy: 3 2024-03-25T18:35:36.149 INFO:teuthology.misc.health.smithi001.stderr:0: v1:172.21.15.1:6789/0 mon.a 2024-03-25T18:35:36.149 INFO:teuthology.misc.health.smithi001.stderr:1: v1:172.21.15.115:6789/0 mon.b 2024-03-25T18:35:36.149 INFO:teuthology.misc.health.smithi001.stderr:2: v1:172.21.15.1:6790/0 mon.c 2024-03-25T18:35:36.149 INFO:teuthology.misc.health.smithi001.stderr: 2024-03-25T18:35:36.149 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:35:36.146+0000 7f9ddffff640 4 set_mon_vals no callback set 2024-03-25T18:35:36.150 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:35:36.147+0000 7f9dde7fc640 1 -- 172.21.15.1:0/2961177610 <== mon.2 v1:172.21.15.1:6790/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 21787975 0 0) 0x7f9dd8004ba0 con 0x7f9de80b44b0 2024-03-25T18:35:36.150 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:35:36.147+0000 7f9ded94b640 10 monclient: get_auth_request con 0x7f9dcc06de40 auth_method 0 2024-03-25T18:35:36.150 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:35:36.148+0000 7f9dde7fc640 10 client.4610.objecter ms_handle_connect 0x7f9dcc06de40 2024-03-25T18:35:36.155 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:35:36.153+0000 7f9dde7fc640 1 -- 172.21.15.1:0/2961177610 <== mon.2 v1:172.21.15.1:6790/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 996254230 0 0) 0x7f9dd8054770 con 0x7f9de80b44b0 2024-03-25T18:35:36.155 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:35:36.153+0000 7f9dde7fc640 10 client.4610.objecter ms_dispatch 0x7f9de8000b90 osd_map(10..10 src has 1..10) v4 2024-03-25T18:35:36.155 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:35:36.153+0000 7f9dde7fc640 3 client.4610.objecter handle_osd_map got epochs [10,10] > 0 2024-03-25T18:35:36.155 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:35:36.153+0000 7f9dde7fc640 3 client.4610.objecter handle_osd_map decoding full epoch 10 2024-03-25T18:35:36.156 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:35:36.154+0000 7f9dde7fc640 20 client.4610.objecter dump_active .. 0 homeless 2024-03-25T18:35:36.156 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:35:36.154+0000 7f9dde7fc640 1 -- 172.21.15.1:0/2961177610 <== mon.2 v1:172.21.15.1:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f9dd80500b0 con 0x7f9de80b44b0 2024-03-25T18:35:36.156 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:35:36.154+0000 7f9dde7fc640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:35:36.156 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:35:36.154+0000 7f9dde7fc640 10 monclient: _finish_command 1 = system:0 2024-03-25T18:35:36.412 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:35:36.410+0000 7f9defbaf640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:35:36.412 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:35:36.410+0000 7f9defbaf640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:35:36.412 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:35:36.410+0000 7f9defbaf640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.1:6790/0 2024-03-25T18:35:36.412 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:35:36.410+0000 7f9defbaf640 1 -- 172.21.15.1:0/2961177610 --> v1:172.21.15.1:6790/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f9de804f730 con 0x7f9de80b44b0 2024-03-25T18:35:36.447 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:35:36.445+0000 7f9dde7fc640 1 -- 172.21.15.1:0/2961177610 <== mon.2 v1:172.21.15.1:6790/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 4092213348) 0x7f9dd80521b0 con 0x7f9de80b44b0 2024-03-25T18:35:36.447 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:35:36.445+0000 7f9dde7fc640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:35:36.447 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:35:36.445+0000 7f9dde7fc640 10 monclient: _finish_command 2 = system:0 2024-03-25T18:35:36.447 INFO:teuthology.misc.health.smithi001.stdout: 2024-03-25T18:35:36.447 INFO:teuthology.misc.health.smithi001.stdout:{"epoch":10,"fsid":"c8ea72cf-9e99-4383-9b52-6c23d365fdfc","created":"2024-03-25T18:26:38.856638+0000","modified":"2024-03-25T18:26:44.224208+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T18:26:44.224208+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"b9f34922-3a84-49be-8426-807c1eceaefc","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"42bf306f-55e3-4634-9874-8b4b73ac6644","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"292cb23a-18a9-4207-a7cc-4d5727feaec0","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"45e0e9ab-898d-4673-9590-b4117bc5e84a","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"831a5a09-73ab-4a61-9872-088e76bdbc66","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"88cd903d-3b55-4840-9396-6e3cfe005282","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"89db7355-bc9d-46f4-8ab0-e8d9e744e026","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"cac47bee-0914-4f08-896f-348838882866","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T18:35:36.450 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:35:36.448+0000 7f9defbaf640 10 librados: watch_flush enter 2024-03-25T18:35:36.450 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:35:36.448+0000 7f9defbaf640 10 librados: watch_flush exit 2024-03-25T18:35:36.451 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:35:36.448+0000 7f9defbaf640 20 client.4610.objecter shutdown clearing up homeless session... 2024-03-25T18:35:36.451 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:35:36.448+0000 7f9defbaf640 10 client.4610.objecter successfully canceled tick 2024-03-25T18:35:36.451 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:35:36.448+0000 7f9defbaf640 1 -- 172.21.15.1:0/2961177610 >> v1:172.21.15.115:6800/4237379176 conn(0x7f9dcc06de40 legacy=0x7f9dcc070300 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:35:36.451 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:35:36.448+0000 7f9defbaf640 10 monclient: shutdown 2024-03-25T18:35:36.451 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:35:36.448+0000 7f9defbaf640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:35:36.451 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:35:36.448+0000 7f9defbaf640 1 -- 172.21.15.1:0/2961177610 >> v1:172.21.15.1:6790/0 conn(0x7f9de80b44b0 legacy=0x7f9de8134b40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:35:36.451 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:35:36.448+0000 7f9defbaf640 1 -- 172.21.15.1:0/2961177610 shutdown_connections 2024-03-25T18:35:36.451 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:35:36.448+0000 7f9defbaf640 1 -- 172.21.15.1:0/2961177610 >> 172.21.15.1:0/2961177610 conn(0x7f9de80617a0 msgr2=0x7f9de80b59e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:35:36.451 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:35:36.448+0000 7f9defbaf640 1 -- 172.21.15.1:0/2961177610 shutdown_connections 2024-03-25T18:35:36.451 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:35:36.448+0000 7f9defbaf640 1 -- 172.21.15.1:0/2961177610 wait complete. 2024-03-25T18:35:36.451 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:35:36.448+0000 7f9defbaf640 1 librados: shutdown 2024-03-25T18:35:36.463 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T18:35:42.965 INFO:tasks.ceph.mon.a.smithi001.stderr:==00:00:09:21.523 34876== Warning: unimplemented fcntl command: 1036 2024-03-25T18:35:42.971 INFO:tasks.ceph.mon.a.smithi001.stderr:==00:00:09:21.530 34876== Warning: unimplemented fcntl command: 1036 2024-03-25T18:35:43.000 INFO:tasks.ceph.mon.a.smithi001.stderr:==00:00:09:21.558 34876== Warning: unimplemented fcntl command: 1036 2024-03-25T18:35:43.020 INFO:tasks.ceph.mon.b.smithi115.stderr:==00:00:09:21.503 34836== Warning: unimplemented fcntl command: 1036 2024-03-25T18:35:43.037 INFO:tasks.ceph.mon.b.smithi115.stderr:==00:00:09:21.521 34836== Warning: unimplemented fcntl command: 1036 2024-03-25T18:35:43.039 INFO:tasks.ceph.mon.c.smithi001.stderr:==00:00:09:21.597 34877== Warning: unimplemented fcntl command: 1036 2024-03-25T18:35:43.044 INFO:tasks.ceph.mon.c.smithi001.stderr:==00:00:09:21.603 34877== Warning: unimplemented fcntl command: 1036 2024-03-25T18:35:43.062 INFO:tasks.ceph.mon.b.smithi115.stderr:==00:00:09:21.545 34836== Warning: unimplemented fcntl command: 1036 2024-03-25T18:35:43.079 INFO:tasks.ceph.mon.c.smithi001.stderr:==00:00:09:21.638 34877== Warning: unimplemented fcntl command: 1036 2024-03-25T18:35:43.268 DEBUG:teuthology.orchestra.run.smithi001:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T18:35:43.383 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:35:43.381+0000 7f06b9b39640 10 monclient: get_monmap_and_config 2024-03-25T18:35:43.383 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:35:43.381+0000 7f06b9b39640 10 monclient: build_initial_monmap 2024-03-25T18:35:43.383 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:35:43.381+0000 7f06b9b39640 10 monclient: monmap: 2024-03-25T18:35:43.383 INFO:teuthology.misc.health.smithi001.stderr:epoch 0 2024-03-25T18:35:43.383 INFO:teuthology.misc.health.smithi001.stderr:fsid c8ea72cf-9e99-4383-9b52-6c23d365fdfc 2024-03-25T18:35:43.383 INFO:teuthology.misc.health.smithi001.stderr:last_changed 2024-03-25T18:35:43.383346+0000 2024-03-25T18:35:43.383 INFO:teuthology.misc.health.smithi001.stderr:created 2024-03-25T18:35:43.383346+0000 2024-03-25T18:35:43.383 INFO:teuthology.misc.health.smithi001.stderr:min_mon_release 0 (unknown) 2024-03-25T18:35:43.383 INFO:teuthology.misc.health.smithi001.stderr:election_strategy: 3 2024-03-25T18:35:43.383 INFO:teuthology.misc.health.smithi001.stderr:0: v1:172.21.15.1:6789/0 mon.noname-a 2024-03-25T18:35:43.383 INFO:teuthology.misc.health.smithi001.stderr:1: v1:172.21.15.115:6789/0 mon.noname-c 2024-03-25T18:35:43.384 INFO:teuthology.misc.health.smithi001.stderr:2: v1:172.21.15.1:6790/0 mon.noname-b 2024-03-25T18:35:43.384 INFO:teuthology.misc.health.smithi001.stderr: 2024-03-25T18:35:43.385 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:35:43.382+0000 7f06b9b39640 1 Processor -- start 2024-03-25T18:35:43.385 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:35:43.382+0000 7f06b9b39640 1 -- start start 2024-03-25T18:35:43.385 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:35:43.382+0000 7f06b9b39640 10 monclient: init 2024-03-25T18:35:43.385 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:35:43.383+0000 7f06b9b39640 10 monclient: _reopen_session rank -1 2024-03-25T18:35:43.385 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:35:43.383+0000 7f06b9b39640 10 monclient: _add_conns ranks=[0,2,1] 2024-03-25T18:35:43.385 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:35:43.383+0000 7f06b9b39640 10 monclient(hunting): picked mon.noname-a con 0x7f06b40b4e70 addr v1:172.21.15.1:6789/0 2024-03-25T18:35:43.385 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:35:43.383+0000 7f06b9b39640 10 monclient(hunting): picked mon.noname-b con 0x7f06b40b44b0 addr v1:172.21.15.1:6790/0 2024-03-25T18:35:43.385 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:35:43.383+0000 7f06b9b39640 10 monclient(hunting): picked mon.noname-c con 0x7f06b40b1b00 addr v1:172.21.15.115:6789/0 2024-03-25T18:35:43.385 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:35:43.383+0000 7f06b9b39640 1 -- --> v1:172.21.15.1:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f06b40a96f0 con 0x7f06b40b4e70 2024-03-25T18:35:43.385 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:35:43.383+0000 7f06b9b39640 1 -- --> v1:172.21.15.115:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f06b40ab0e0 con 0x7f06b40b1b00 2024-03-25T18:35:43.386 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:35:43.383+0000 7f06b9b39640 1 -- --> v1:172.21.15.1:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f06b4145f70 con 0x7f06b40b44b0 2024-03-25T18:35:43.386 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:35:43.383+0000 7f06b9b39640 10 monclient(hunting): _renew_subs 2024-03-25T18:35:43.386 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:35:43.383+0000 7f06b9b39640 10 monclient(hunting): authenticate will time out at 8023.854082s 2024-03-25T18:35:43.386 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:35:43.384+0000 7f06b2ffd640 1 --1- >> v1:172.21.15.1:6790/0 conn(0x7f06b40b44b0 0x7f06b4145860 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.1:6790/0 says I am v1:172.21.15.1:36016/0 (socket says 172.21.15.1:36016) 2024-03-25T18:35:43.386 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:35:43.384+0000 7f06b2ffd640 1 -- 172.21.15.1:0/1190986055 learned_addr learned my addr 172.21.15.1:0/1190986055 (peer_addr_for_me v1:172.21.15.1:0/0) 2024-03-25T18:35:43.395 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:35:43.393+0000 7f06b27fc640 1 -- 172.21.15.1:0/1190986055 <== mon.1 v1:172.21.15.115:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 597608866 0 0) 0x7f06b40ab0e0 con 0x7f06b40b1b00 2024-03-25T18:35:43.395 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:35:43.393+0000 7f06b27fc640 10 monclient(hunting): _negotiate 2024-03-25T18:35:43.395 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:35:43.393+0000 7f06b27fc640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:35:43.395 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:35:43.393+0000 7f06b27fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:35:43.395 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:35:43.393+0000 7f06b27fc640 10 monclient(hunting): my global_id is 4612 2024-03-25T18:35:43.395 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:35:43.393+0000 7f06b27fc640 1 -- 172.21.15.1:0/1190986055 --> v1:172.21.15.115:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f069c003520 con 0x7f06b40b1b00 2024-03-25T18:35:43.396 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:35:43.395+0000 7f06b27fc640 1 -- 172.21.15.1:0/1190986055 <== mon.2 v1:172.21.15.1:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 363812648 0 0) 0x7f06b4145f70 con 0x7f06b40b44b0 2024-03-25T18:35:43.396 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:35:43.395+0000 7f06b27fc640 10 monclient(hunting): _negotiate 2024-03-25T18:35:43.396 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:35:43.395+0000 7f06b27fc640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:35:43.397 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:35:43.395+0000 7f06b27fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:35:43.397 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:35:43.395+0000 7f06b27fc640 10 monclient(hunting): my global_id is 4613 2024-03-25T18:35:43.397 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:35:43.395+0000 7f06b27fc640 1 -- 172.21.15.1:0/1190986055 --> v1:172.21.15.1:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f06b40ab0e0 con 0x7f06b40b44b0 2024-03-25T18:35:43.402 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:35:43.400+0000 7f06b27fc640 1 -- 172.21.15.1:0/1190986055 <== mon.1 v1:172.21.15.115:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3677992221 0 0) 0x7f069c003520 con 0x7f06b40b1b00 2024-03-25T18:35:43.402 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:35:43.400+0000 7f06b27fc640 1 -- 172.21.15.1:0/1190986055 >> v1:172.21.15.1:6790/0 conn(0x7f06b40b44b0 legacy=0x7f06b4145860 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:35:43.402 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:35:43.400+0000 7f06b27fc640 1 -- 172.21.15.1:0/1190986055 >> v1:172.21.15.1:6789/0 conn(0x7f06b40b4e70 legacy=0x7f06b413f940 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:35:43.402 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:35:43.400+0000 7f06b27fc640 10 monclient: _finish_hunting 0 2024-03-25T18:35:43.402 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:35:43.400+0000 7f06b27fc640 1 monclient: found mon.noname-c 2024-03-25T18:35:43.402 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:35:43.400+0000 7f06b27fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:35:43.402 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:35:43.400+0000 7f06b27fc640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.115:6789/0 2024-03-25T18:35:43.402 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:35:43.400+0000 7f06b27fc640 1 -- 172.21.15.1:0/1190986055 --> v1:172.21.15.115:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f06b40a9b80 con 0x7f06b40b1b00 2024-03-25T18:35:43.402 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:35:43.400+0000 7f06b27fc640 10 monclient: _finish_auth 0 2024-03-25T18:35:43.402 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:35:43.400+0000 7f06b27fc640 10 monclient: _check_auth_tickets 2024-03-25T18:35:43.402 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:35:43.400+0000 7f06b27fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:35:43.402 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:35:43.400+0000 7f06b27fc640 1 -- 172.21.15.1:0/1190986055 <== mon.1 v1:172.21.15.115:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1586965025 0 0) 0x7f06ac001a10 con 0x7f06b40b1b00 2024-03-25T18:35:43.402 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:35:43.400+0000 7f06b27fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:35:43.403 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:35:43.400+0000 7f06b27fc640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-03-25T18:35:43.403 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:35:43.400+0000 7f06b27fc640 10 monclient: dump: 2024-03-25T18:35:43.403 INFO:teuthology.misc.health.smithi001.stderr:epoch 1 2024-03-25T18:35:43.403 INFO:teuthology.misc.health.smithi001.stderr:fsid c8ea72cf-9e99-4383-9b52-6c23d365fdfc 2024-03-25T18:35:43.403 INFO:teuthology.misc.health.smithi001.stderr:last_changed 2024-03-25T18:25:50.510519+0000 2024-03-25T18:35:43.403 INFO:teuthology.misc.health.smithi001.stderr:created 2024-03-25T18:25:50.510519+0000 2024-03-25T18:35:43.403 INFO:teuthology.misc.health.smithi001.stderr:min_mon_release 19 (squid) 2024-03-25T18:35:43.403 INFO:teuthology.misc.health.smithi001.stderr:election_strategy: 3 2024-03-25T18:35:43.403 INFO:teuthology.misc.health.smithi001.stderr:0: v1:172.21.15.1:6789/0 mon.a 2024-03-25T18:35:43.403 INFO:teuthology.misc.health.smithi001.stderr:1: v1:172.21.15.115:6789/0 mon.b 2024-03-25T18:35:43.403 INFO:teuthology.misc.health.smithi001.stderr:2: v1:172.21.15.1:6790/0 mon.c 2024-03-25T18:35:43.403 INFO:teuthology.misc.health.smithi001.stderr: 2024-03-25T18:35:43.403 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:35:43.400+0000 7f06b9b39640 5 monclient: authenticate success, global_id 4612 2024-03-25T18:35:43.403 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:35:43.400+0000 7f06b9b39640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T18:35:43.412 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:35:43.411+0000 7f06b27fc640 1 -- 172.21.15.1:0/1190986055 <== mon.1 v1:172.21.15.115:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f06ac001b70 con 0x7f06b40b1b00 2024-03-25T18:35:43.413 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:35:43.411+0000 7f06b27fc640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:35:43.413 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:35:43.411+0000 7f06b27fc640 1 -- 172.21.15.1:0/1190986055 <== mon.1 v1:172.21.15.115:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1586965025 0 0) 0x7f06ac001f80 con 0x7f06b40b1b00 2024-03-25T18:35:43.413 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:35:43.411+0000 7f06b9b39640 10 monclient: get_monmap_and_config success 2024-03-25T18:35:43.413 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:35:43.411+0000 7f06b9b39640 4 set_mon_vals no callback set 2024-03-25T18:35:43.413 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:35:43.411+0000 7f06b27fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:35:43.413 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:35:43.411+0000 7f06b27fc640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-25T18:35:43.413 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:35:43.411+0000 7f06b27fc640 10 monclient: dump: 2024-03-25T18:35:43.413 INFO:teuthology.misc.health.smithi001.stderr:epoch 1 2024-03-25T18:35:43.413 INFO:teuthology.misc.health.smithi001.stderr:fsid c8ea72cf-9e99-4383-9b52-6c23d365fdfc 2024-03-25T18:35:43.413 INFO:teuthology.misc.health.smithi001.stderr:last_changed 2024-03-25T18:25:50.510519+0000 2024-03-25T18:35:43.413 INFO:teuthology.misc.health.smithi001.stderr:created 2024-03-25T18:25:50.510519+0000 2024-03-25T18:35:43.413 INFO:teuthology.misc.health.smithi001.stderr:min_mon_release 19 (squid) 2024-03-25T18:35:43.413 INFO:teuthology.misc.health.smithi001.stderr:election_strategy: 3 2024-03-25T18:35:43.413 INFO:teuthology.misc.health.smithi001.stderr:0: v1:172.21.15.1:6789/0 mon.a 2024-03-25T18:35:43.414 INFO:teuthology.misc.health.smithi001.stderr:1: v1:172.21.15.115:6789/0 mon.b 2024-03-25T18:35:43.414 INFO:teuthology.misc.health.smithi001.stderr:2: v1:172.21.15.1:6790/0 mon.c 2024-03-25T18:35:43.414 INFO:teuthology.misc.health.smithi001.stderr: 2024-03-25T18:35:43.415 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:35:43.412+0000 7f06b9b39640 10 monclient: shutdown 2024-03-25T18:35:43.416 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:35:43.412+0000 7f06b9b39640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:35:43.416 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:35:43.412+0000 7f06b9b39640 1 -- 172.21.15.1:0/1190986055 >> v1:172.21.15.115:6789/0 conn(0x7f06b40b1b00 legacy=0x7f06b414dc00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:35:43.416 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:35:43.412+0000 7f06b9b39640 1 -- 172.21.15.1:0/1190986055 shutdown_connections 2024-03-25T18:35:43.416 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:35:43.412+0000 7f06b9b39640 1 -- 172.21.15.1:0/1190986055 >> 172.21.15.1:0/1190986055 conn(0x7f06b40617a0 msgr2=0x7f06b4061bb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:35:43.416 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:35:43.412+0000 7f06b9b39640 1 -- 172.21.15.1:0/1190986055 shutdown_connections 2024-03-25T18:35:43.416 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:35:43.412+0000 7f06b9b39640 1 -- 172.21.15.1:0/1190986055 wait complete. 2024-03-25T18:35:43.416 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:35:43.412+0000 7f06b9b39640 10 monclient: build_initial_monmap 2024-03-25T18:35:43.416 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:35:43.412+0000 7f06b9b39640 10 monclient: monmap: 2024-03-25T18:35:43.416 INFO:teuthology.misc.health.smithi001.stderr:epoch 0 2024-03-25T18:35:43.416 INFO:teuthology.misc.health.smithi001.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T18:35:43.416 INFO:teuthology.misc.health.smithi001.stderr:last_changed 0.000000 2024-03-25T18:35:43.416 INFO:teuthology.misc.health.smithi001.stderr:created 0.000000 2024-03-25T18:35:43.416 INFO:teuthology.misc.health.smithi001.stderr:min_mon_release 0 (unknown) 2024-03-25T18:35:43.416 INFO:teuthology.misc.health.smithi001.stderr:election_strategy: 1 2024-03-25T18:35:43.416 INFO:teuthology.misc.health.smithi001.stderr:0: v1:172.21.15.1:6789/0 mon.noname-a 2024-03-25T18:35:43.416 INFO:teuthology.misc.health.smithi001.stderr:1: v1:172.21.15.115:6789/0 mon.noname-b 2024-03-25T18:35:43.417 INFO:teuthology.misc.health.smithi001.stderr:2: v1:172.21.15.1:6790/0 mon.noname-c 2024-03-25T18:35:43.417 INFO:teuthology.misc.health.smithi001.stderr: 2024-03-25T18:35:43.417 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:35:43.413+0000 7f06b9b39640 1 librados: starting msgr at 2024-03-25T18:35:43.417 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:35:43.413+0000 7f06b9b39640 1 librados: starting objecter 2024-03-25T18:35:43.417 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:35:43.413+0000 7f06b9b39640 1 Processor -- start 2024-03-25T18:35:43.417 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:35:43.413+0000 7f06b9b39640 1 -- start start 2024-03-25T18:35:43.417 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:35:43.413+0000 7f06b9b39640 1 librados: setting wanted keys 2024-03-25T18:35:43.417 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:35:43.413+0000 7f06b9b39640 1 librados: calling monclient init 2024-03-25T18:35:43.417 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:35:43.413+0000 7f06b9b39640 10 monclient: init 2024-03-25T18:35:43.417 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:35:43.414+0000 7f06b9b39640 10 monclient: _reopen_session rank -1 2024-03-25T18:35:43.417 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:35:43.414+0000 7f06b9b39640 10 monclient: _add_conns ranks=[2,1,0] 2024-03-25T18:35:43.417 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:35:43.414+0000 7f06b9b39640 10 monclient(hunting): picked mon.noname-c con 0x7f06b40b1b00 addr v1:172.21.15.1:6790/0 2024-03-25T18:35:43.417 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:35:43.414+0000 7f06b9b39640 10 monclient(hunting): picked mon.noname-b con 0x7f06b40b44b0 addr v1:172.21.15.115:6789/0 2024-03-25T18:35:43.417 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:35:43.414+0000 7f06b9b39640 10 monclient(hunting): picked mon.noname-a con 0x7f06b40b4e70 addr v1:172.21.15.1:6789/0 2024-03-25T18:35:43.417 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:35:43.414+0000 7f06b9b39640 1 -- --> v1:172.21.15.1:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f06b414e100 con 0x7f06b40b4e70 2024-03-25T18:35:43.417 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:35:43.414+0000 7f06b9b39640 1 -- --> v1:172.21.15.115:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f06b40ac0d0 con 0x7f06b40b44b0 2024-03-25T18:35:43.418 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:35:43.414+0000 7f06b9b39640 1 -- --> v1:172.21.15.1:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f06b40ac3b0 con 0x7f06b40b1b00 2024-03-25T18:35:43.418 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:35:43.414+0000 7f06b9b39640 10 monclient(hunting): _renew_subs 2024-03-25T18:35:43.418 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:35:43.414+0000 7f06b9b39640 10 monclient(hunting): authenticate will time out at 8023.884860s 2024-03-25T18:35:43.419 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:35:43.417+0000 7f06b3fff640 1 --1- >> v1:172.21.15.1:6789/0 conn(0x7f06b40b4e70 0x7f06b41e68b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.1:6789/0 says I am v1:172.21.15.1:50402/0 (socket says 172.21.15.1:50402) 2024-03-25T18:35:43.419 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:35:43.418+0000 7f06b3fff640 1 -- 172.21.15.1:0/1087502947 learned_addr learned my addr 172.21.15.1:0/1087502947 (peer_addr_for_me v1:172.21.15.1:0/0) 2024-03-25T18:35:43.423 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:35:43.422+0000 7f069bfff640 10 client.?.objecter ms_handle_connect 0x7f06b40b4e70 2024-03-25T18:35:43.424 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:35:43.422+0000 7f069bfff640 10 client.?.objecter resend_mon_ops 2024-03-25T18:35:43.427 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:35:43.425+0000 7f069bfff640 10 client.?.objecter ms_handle_connect 0x7f06b40b1b00 2024-03-25T18:35:43.427 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:35:43.425+0000 7f069bfff640 10 client.?.objecter resend_mon_ops 2024-03-25T18:35:43.429 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:35:43.428+0000 7f069bfff640 10 client.?.objecter ms_handle_connect 0x7f06b40b44b0 2024-03-25T18:35:43.429 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:35:43.428+0000 7f069bfff640 10 client.?.objecter resend_mon_ops 2024-03-25T18:35:43.431 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:35:43.430+0000 7f069bfff640 1 -- 172.21.15.1:0/1087502947 <== mon.0 v1:172.21.15.1:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3218799513 0 0) 0x7f06b414e100 con 0x7f06b40b4e70 2024-03-25T18:35:43.431 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:35:43.430+0000 7f069bfff640 10 monclient(hunting): _negotiate 2024-03-25T18:35:43.431 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:35:43.430+0000 7f069bfff640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:35:43.431 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:35:43.430+0000 7f069bfff640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:35:43.431 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:35:43.430+0000 7f069bfff640 10 monclient(hunting): my global_id is 4608 2024-03-25T18:35:43.432 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:35:43.430+0000 7f069bfff640 1 -- 172.21.15.1:0/1087502947 --> v1:172.21.15.1:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f06a0004eb0 con 0x7f06b40b4e70 2024-03-25T18:35:43.440 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:35:43.438+0000 7f069bfff640 1 -- 172.21.15.1:0/1087502947 <== mon.0 v1:172.21.15.1:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1452053099 0 0) 0x7f06a0004eb0 con 0x7f06b40b4e70 2024-03-25T18:35:43.440 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:35:43.438+0000 7f069bfff640 1 -- 172.21.15.1:0/1087502947 --> v1:172.21.15.1:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f06b414e100 con 0x7f06b40b4e70 2024-03-25T18:35:43.440 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:35:43.438+0000 7f069bfff640 1 -- 172.21.15.1:0/1087502947 <== mon.0 v1:172.21.15.1:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1586965025 0 0) 0x7f06ac001ec0 con 0x7f06b40b4e70 2024-03-25T18:35:43.440 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:35:43.438+0000 7f069bfff640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:35:43.440 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:35:43.438+0000 7f069bfff640 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2024-03-25T18:35:43.440 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:35:43.438+0000 7f069bfff640 10 monclient(hunting): dump: 2024-03-25T18:35:43.440 INFO:teuthology.misc.health.smithi001.stderr:epoch 1 2024-03-25T18:35:43.440 INFO:teuthology.misc.health.smithi001.stderr:fsid c8ea72cf-9e99-4383-9b52-6c23d365fdfc 2024-03-25T18:35:43.440 INFO:teuthology.misc.health.smithi001.stderr:last_changed 2024-03-25T18:25:50.510519+0000 2024-03-25T18:35:43.440 INFO:teuthology.misc.health.smithi001.stderr:created 2024-03-25T18:25:50.510519+0000 2024-03-25T18:35:43.440 INFO:teuthology.misc.health.smithi001.stderr:min_mon_release 19 (squid) 2024-03-25T18:35:43.440 INFO:teuthology.misc.health.smithi001.stderr:election_strategy: 3 2024-03-25T18:35:43.440 INFO:teuthology.misc.health.smithi001.stderr:0: v1:172.21.15.1:6789/0 mon.a 2024-03-25T18:35:43.440 INFO:teuthology.misc.health.smithi001.stderr:1: v1:172.21.15.115:6789/0 mon.b 2024-03-25T18:35:43.440 INFO:teuthology.misc.health.smithi001.stderr:2: v1:172.21.15.1:6790/0 mon.c 2024-03-25T18:35:43.441 INFO:teuthology.misc.health.smithi001.stderr: 2024-03-25T18:35:43.441 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:35:43.438+0000 7f069bfff640 10 monclient(hunting): _finish_auth 0 2024-03-25T18:35:43.442 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:35:43.441+0000 7f069bfff640 1 -- 172.21.15.1:0/1087502947 <== mon.2 v1:172.21.15.1:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4064593507 0 0) 0x7f06b40ac3b0 con 0x7f06b40b1b00 2024-03-25T18:35:43.442 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:35:43.441+0000 7f069bfff640 10 monclient(hunting): _negotiate 2024-03-25T18:35:43.442 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:35:43.441+0000 7f069bfff640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:35:43.443 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:35:43.441+0000 7f069bfff640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:35:43.443 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:35:43.441+0000 7f069bfff640 10 monclient(hunting): my global_id is 4616 2024-03-25T18:35:43.443 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:35:43.441+0000 7f069bfff640 1 -- 172.21.15.1:0/1087502947 --> v1:172.21.15.1:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f06a0004eb0 con 0x7f06b40b1b00 2024-03-25T18:35:43.445 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:35:43.443+0000 7f069bfff640 1 -- 172.21.15.1:0/1087502947 <== mon.1 v1:172.21.15.115:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1551511123 0 0) 0x7f06b40ac0d0 con 0x7f06b40b44b0 2024-03-25T18:35:43.446 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:35:43.443+0000 7f069bfff640 10 monclient(hunting): _negotiate 2024-03-25T18:35:43.446 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:35:43.443+0000 7f069bfff640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:35:43.446 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:35:43.443+0000 7f069bfff640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:35:43.446 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:35:43.443+0000 7f069bfff640 10 monclient(hunting): my global_id is 4615 2024-03-25T18:35:43.446 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:35:43.443+0000 7f069bfff640 1 -- 172.21.15.1:0/1087502947 --> v1:172.21.15.115:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f06b40ac3b0 con 0x7f06b40b44b0 2024-03-25T18:35:43.446 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:35:43.443+0000 7f069bfff640 1 -- 172.21.15.1:0/1087502947 <== mon.0 v1:172.21.15.1:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 3697970264 0 0) 0x7f06b414e100 con 0x7f06b40b4e70 2024-03-25T18:35:43.446 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:35:43.443+0000 7f069bfff640 1 -- 172.21.15.1:0/1087502947 >> v1:172.21.15.1:6790/0 conn(0x7f06b40b1b00 legacy=0x7f06b4070b30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:35:43.446 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:35:43.443+0000 7f069bfff640 1 -- 172.21.15.1:0/1087502947 >> v1:172.21.15.115:6789/0 conn(0x7f06b40b44b0 legacy=0x7f06b413c530 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:35:43.446 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:35:43.443+0000 7f069bfff640 10 monclient: _finish_hunting 0 2024-03-25T18:35:43.446 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:35:43.443+0000 7f069bfff640 1 monclient: found mon.a 2024-03-25T18:35:43.446 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:35:43.443+0000 7f069bfff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:35:43.446 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:35:43.443+0000 7f069bfff640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.1:6789/0 2024-03-25T18:35:43.446 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:35:43.443+0000 7f069bfff640 1 -- 172.21.15.1:0/1087502947 --> v1:172.21.15.1:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f06b40ac690 con 0x7f06b40b4e70 2024-03-25T18:35:43.446 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:35:43.443+0000 7f069bfff640 10 monclient: _finish_auth 0 2024-03-25T18:35:43.447 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:35:43.443+0000 7f069bfff640 10 monclient: _check_auth_tickets 2024-03-25T18:35:43.447 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:35:43.443+0000 7f069bfff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:35:43.447 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:35:43.443+0000 7f06b9b39640 5 monclient: authenticate success, global_id 4608 2024-03-25T18:35:43.447 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:35:43.443+0000 7f06b9b39640 10 monclient: _renew_subs 2024-03-25T18:35:43.447 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:35:43.443+0000 7f06b9b39640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.1:6789/0 2024-03-25T18:35:43.447 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:35:43.443+0000 7f06b9b39640 1 -- 172.21.15.1:0/1087502947 --> v1:172.21.15.1:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f06b4139860 con 0x7f06b40b4e70 2024-03-25T18:35:43.447 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:35:43.443+0000 7f06b9b39640 10 client.4608.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T18:35:43.447 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:35:43.444+0000 7f06b9b39640 10 monclient: _renew_subs 2024-03-25T18:35:43.447 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:35:43.444+0000 7f06b9b39640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.1:6789/0 2024-03-25T18:35:43.447 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:35:43.444+0000 7f06b9b39640 1 -- 172.21.15.1:0/1087502947 --> v1:172.21.15.1:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f06b40ab780 con 0x7f06b40b4e70 2024-03-25T18:35:43.447 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:35:43.444+0000 7f06b9b39640 1 librados: init done 2024-03-25T18:35:43.448 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:35:43.445+0000 7f06b9b39640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T18:35:43.448 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:35:43.445+0000 7f06b9b39640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:35:43.448 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:35:43.445+0000 7f06b9b39640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.1:6789/0 2024-03-25T18:35:43.448 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:35:43.445+0000 7f06b9b39640 1 -- 172.21.15.1:0/1087502947 --> v1:172.21.15.1:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f06b404f730 con 0x7f06b40b4e70 2024-03-25T18:35:43.751 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:35:43.749+0000 7f069bfff640 1 -- 172.21.15.1:0/1087502947 <== mon.0 v1:172.21.15.1:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f06ac003c20 con 0x7f06b40b4e70 2024-03-25T18:35:43.751 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:35:43.749+0000 7f069bfff640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:35:43.751 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:35:43.749+0000 7f069bfff640 1 -- 172.21.15.1:0/1087502947 <== mon.0 v1:172.21.15.1:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1586965025 0 0) 0x7f06ac0015d0 con 0x7f06b40b4e70 2024-03-25T18:35:43.751 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:35:43.749+0000 7f069bfff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:35:43.751 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:35:43.749+0000 7f069bfff640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-25T18:35:43.751 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:35:43.749+0000 7f069bfff640 10 monclient: dump: 2024-03-25T18:35:43.751 INFO:teuthology.misc.health.smithi001.stderr:epoch 1 2024-03-25T18:35:43.751 INFO:teuthology.misc.health.smithi001.stderr:fsid c8ea72cf-9e99-4383-9b52-6c23d365fdfc 2024-03-25T18:35:43.751 INFO:teuthology.misc.health.smithi001.stderr:last_changed 2024-03-25T18:25:50.510519+0000 2024-03-25T18:35:43.751 INFO:teuthology.misc.health.smithi001.stderr:created 2024-03-25T18:25:50.510519+0000 2024-03-25T18:35:43.751 INFO:teuthology.misc.health.smithi001.stderr:min_mon_release 19 (squid) 2024-03-25T18:35:43.751 INFO:teuthology.misc.health.smithi001.stderr:election_strategy: 3 2024-03-25T18:35:43.751 INFO:teuthology.misc.health.smithi001.stderr:0: v1:172.21.15.1:6789/0 mon.a 2024-03-25T18:35:43.752 INFO:teuthology.misc.health.smithi001.stderr:1: v1:172.21.15.115:6789/0 mon.b 2024-03-25T18:35:43.752 INFO:teuthology.misc.health.smithi001.stderr:2: v1:172.21.15.1:6790/0 mon.c 2024-03-25T18:35:43.752 INFO:teuthology.misc.health.smithi001.stderr: 2024-03-25T18:35:43.752 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:35:43.749+0000 7f06b1ffb640 4 set_mon_vals no callback set 2024-03-25T18:35:43.752 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:35:43.750+0000 7f069bfff640 1 -- 172.21.15.1:0/1087502947 <== mon.0 v1:172.21.15.1:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 21787975 0 0) 0x7f06ac004130 con 0x7f06b40b4e70 2024-03-25T18:35:43.752 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:35:43.750+0000 7f06b37fe640 10 monclient: get_auth_request con 0x7f06a006dea0 auth_method 0 2024-03-25T18:35:43.752 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:35:43.750+0000 7f069bfff640 10 client.4608.objecter ms_handle_connect 0x7f06a006dea0 2024-03-25T18:35:43.759 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:35:43.757+0000 7f069bfff640 1 -- 172.21.15.1:0/1087502947 <== mon.0 v1:172.21.15.1:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 996254230 0 0) 0x7f06ac054b50 con 0x7f06b40b4e70 2024-03-25T18:35:43.759 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:35:43.757+0000 7f069bfff640 10 client.4608.objecter ms_dispatch 0x7f06b4000b90 osd_map(10..10 src has 1..10) v4 2024-03-25T18:35:43.759 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:35:43.757+0000 7f069bfff640 3 client.4608.objecter handle_osd_map got epochs [10,10] > 0 2024-03-25T18:35:43.759 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:35:43.757+0000 7f069bfff640 3 client.4608.objecter handle_osd_map decoding full epoch 10 2024-03-25T18:35:43.759 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:35:43.757+0000 7f069bfff640 20 client.4608.objecter dump_active .. 0 homeless 2024-03-25T18:35:43.759 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:35:43.757+0000 7f069bfff640 1 -- 172.21.15.1:0/1087502947 <== mon.0 v1:172.21.15.1:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f06ac050660 con 0x7f06b40b4e70 2024-03-25T18:35:43.759 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:35:43.757+0000 7f069bfff640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:35:43.759 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:35:43.757+0000 7f069bfff640 10 monclient: _finish_command 1 = system:0 2024-03-25T18:35:44.009 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:35:44.008+0000 7f06b9b39640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:35:44.010 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:35:44.008+0000 7f06b9b39640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:35:44.010 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:35:44.008+0000 7f06b9b39640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.1:6789/0 2024-03-25T18:35:44.010 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:35:44.008+0000 7f06b9b39640 1 -- 172.21.15.1:0/1087502947 --> v1:172.21.15.1:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f06b40aa090 con 0x7f06b40b4e70 2024-03-25T18:35:44.055 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:35:44.053+0000 7f069bfff640 1 -- 172.21.15.1:0/1087502947 <== mon.0 v1:172.21.15.1:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 4092213348) 0x7f06ac052590 con 0x7f06b40b4e70 2024-03-25T18:35:44.055 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:35:44.053+0000 7f069bfff640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:35:44.055 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:35:44.053+0000 7f069bfff640 10 monclient: _finish_command 2 = system:0 2024-03-25T18:35:44.055 INFO:teuthology.misc.health.smithi001.stdout: 2024-03-25T18:35:44.056 INFO:teuthology.misc.health.smithi001.stdout:{"epoch":10,"fsid":"c8ea72cf-9e99-4383-9b52-6c23d365fdfc","created":"2024-03-25T18:26:38.856638+0000","modified":"2024-03-25T18:26:44.224208+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T18:26:44.224208+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"b9f34922-3a84-49be-8426-807c1eceaefc","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"42bf306f-55e3-4634-9874-8b4b73ac6644","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"292cb23a-18a9-4207-a7cc-4d5727feaec0","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"45e0e9ab-898d-4673-9590-b4117bc5e84a","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"831a5a09-73ab-4a61-9872-088e76bdbc66","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"88cd903d-3b55-4840-9396-6e3cfe005282","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"89db7355-bc9d-46f4-8ab0-e8d9e744e026","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"cac47bee-0914-4f08-896f-348838882866","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T18:35:44.058 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:35:44.056+0000 7f06b9b39640 10 librados: watch_flush enter 2024-03-25T18:35:44.058 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:35:44.056+0000 7f06b9b39640 10 librados: watch_flush exit 2024-03-25T18:35:44.058 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:35:44.056+0000 7f06b9b39640 20 client.4608.objecter shutdown clearing up homeless session... 2024-03-25T18:35:44.058 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:35:44.056+0000 7f06b9b39640 10 client.4608.objecter successfully canceled tick 2024-03-25T18:35:44.058 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:35:44.056+0000 7f06b9b39640 1 -- 172.21.15.1:0/1087502947 >> v1:172.21.15.115:6800/4237379176 conn(0x7f06a006dea0 legacy=0x7f06a0070360 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:35:44.058 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:35:44.056+0000 7f06b9b39640 10 monclient: shutdown 2024-03-25T18:35:44.059 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:35:44.056+0000 7f06b9b39640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:35:44.059 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:35:44.056+0000 7f06b9b39640 1 -- 172.21.15.1:0/1087502947 >> v1:172.21.15.1:6789/0 conn(0x7f06b40b4e70 legacy=0x7f06b41e68b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:35:44.059 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:35:44.056+0000 7f06b9b39640 1 -- 172.21.15.1:0/1087502947 shutdown_connections 2024-03-25T18:35:44.059 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:35:44.056+0000 7f06b9b39640 1 -- 172.21.15.1:0/1087502947 >> 172.21.15.1:0/1087502947 conn(0x7f06b40617a0 msgr2=0x7f06b40b6860 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:35:44.059 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:35:44.056+0000 7f06b9b39640 1 -- 172.21.15.1:0/1087502947 shutdown_connections 2024-03-25T18:35:44.059 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:35:44.056+0000 7f06b9b39640 1 -- 172.21.15.1:0/1087502947 wait complete. 2024-03-25T18:35:44.059 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:35:44.056+0000 7f06b9b39640 1 librados: shutdown 2024-03-25T18:35:44.068 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T18:35:50.875 DEBUG:teuthology.orchestra.run.smithi001:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T18:35:50.991 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:35:50.990+0000 7fc25a2c9640 10 monclient: get_monmap_and_config 2024-03-25T18:35:50.991 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:35:50.990+0000 7fc25a2c9640 10 monclient: build_initial_monmap 2024-03-25T18:35:50.992 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:35:50.990+0000 7fc25a2c9640 10 monclient: monmap: 2024-03-25T18:35:50.992 INFO:teuthology.misc.health.smithi001.stderr:epoch 0 2024-03-25T18:35:50.992 INFO:teuthology.misc.health.smithi001.stderr:fsid c8ea72cf-9e99-4383-9b52-6c23d365fdfc 2024-03-25T18:35:50.992 INFO:teuthology.misc.health.smithi001.stderr:last_changed 2024-03-25T18:35:50.991753+0000 2024-03-25T18:35:50.992 INFO:teuthology.misc.health.smithi001.stderr:created 2024-03-25T18:35:50.991753+0000 2024-03-25T18:35:50.992 INFO:teuthology.misc.health.smithi001.stderr:min_mon_release 0 (unknown) 2024-03-25T18:35:50.992 INFO:teuthology.misc.health.smithi001.stderr:election_strategy: 3 2024-03-25T18:35:50.992 INFO:teuthology.misc.health.smithi001.stderr:0: v1:172.21.15.1:6789/0 mon.noname-a 2024-03-25T18:35:50.992 INFO:teuthology.misc.health.smithi001.stderr:1: v1:172.21.15.115:6789/0 mon.noname-c 2024-03-25T18:35:50.992 INFO:teuthology.misc.health.smithi001.stderr:2: v1:172.21.15.1:6790/0 mon.noname-b 2024-03-25T18:35:50.992 INFO:teuthology.misc.health.smithi001.stderr: 2024-03-25T18:35:50.993 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:35:50.991+0000 7fc25a2c9640 1 Processor -- start 2024-03-25T18:35:50.993 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:35:50.991+0000 7fc25a2c9640 1 -- start start 2024-03-25T18:35:50.993 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:35:50.991+0000 7fc25a2c9640 10 monclient: init 2024-03-25T18:35:50.994 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:35:50.991+0000 7fc25a2c9640 10 monclient: _reopen_session rank -1 2024-03-25T18:35:50.994 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:35:50.991+0000 7fc25a2c9640 10 monclient: _add_conns ranks=[2,1,0] 2024-03-25T18:35:50.994 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:35:50.991+0000 7fc25a2c9640 10 monclient(hunting): picked mon.noname-b con 0x7fc2540b4e70 addr v1:172.21.15.1:6790/0 2024-03-25T18:35:50.994 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:35:50.991+0000 7fc25a2c9640 10 monclient(hunting): picked mon.noname-c con 0x7fc2540b44b0 addr v1:172.21.15.115:6789/0 2024-03-25T18:35:50.994 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:35:50.991+0000 7fc25a2c9640 10 monclient(hunting): picked mon.noname-a con 0x7fc2540b1b00 addr v1:172.21.15.1:6789/0 2024-03-25T18:35:50.994 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:35:50.991+0000 7fc25a2c9640 1 -- --> v1:172.21.15.1:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc2540a96f0 con 0x7fc2540b1b00 2024-03-25T18:35:50.994 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:35:50.991+0000 7fc25a2c9640 1 -- --> v1:172.21.15.115:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc2540ab0e0 con 0x7fc2540b44b0 2024-03-25T18:35:50.994 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:35:50.991+0000 7fc25a2c9640 1 -- --> v1:172.21.15.1:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc254145f70 con 0x7fc2540b4e70 2024-03-25T18:35:50.994 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:35:50.992+0000 7fc25a2c9640 10 monclient(hunting): _renew_subs 2024-03-25T18:35:50.994 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:35:50.992+0000 7fc25a2c9640 10 monclient(hunting): authenticate will time out at 8031.462664s 2024-03-25T18:35:51.008 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:35:51.006+0000 7fc253fff640 1 --1- >> v1:172.21.15.1:6790/0 conn(0x7fc2540b4e70 0x7fc25413f940 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.1:6790/0 says I am v1:172.21.15.1:40878/0 (socket says 172.21.15.1:40878) 2024-03-25T18:35:51.008 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:35:51.006+0000 7fc253fff640 1 -- 172.21.15.1:0/1343265478 learned_addr learned my addr 172.21.15.1:0/1343265478 (peer_addr_for_me v1:172.21.15.1:0/0) 2024-03-25T18:35:51.020 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:35:51.018+0000 7fc252ffd640 1 -- 172.21.15.1:0/1343265478 <== mon.2 v1:172.21.15.1:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2771633630 0 0) 0x7fc254145f70 con 0x7fc2540b4e70 2024-03-25T18:35:51.020 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:35:51.018+0000 7fc252ffd640 10 monclient(hunting): _negotiate 2024-03-25T18:35:51.020 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:35:51.018+0000 7fc252ffd640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:35:51.020 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:35:51.018+0000 7fc252ffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:35:51.020 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:35:51.018+0000 7fc252ffd640 10 monclient(hunting): my global_id is 4619 2024-03-25T18:35:51.020 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:35:51.018+0000 7fc252ffd640 1 -- 172.21.15.1:0/1343265478 --> v1:172.21.15.1:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fc23c003520 con 0x7fc2540b4e70 2024-03-25T18:35:51.025 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:35:51.024+0000 7fc252ffd640 1 -- 172.21.15.1:0/1343265478 <== mon.0 v1:172.21.15.1:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2993157979 0 0) 0x7fc2540a96f0 con 0x7fc2540b1b00 2024-03-25T18:35:51.025 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:35:51.024+0000 7fc252ffd640 10 monclient(hunting): _negotiate 2024-03-25T18:35:51.025 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:35:51.024+0000 7fc252ffd640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:35:51.025 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:35:51.024+0000 7fc252ffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:35:51.025 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:35:51.024+0000 7fc252ffd640 10 monclient(hunting): my global_id is 4611 2024-03-25T18:35:51.025 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:35:51.024+0000 7fc252ffd640 1 -- 172.21.15.1:0/1343265478 --> v1:172.21.15.1:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fc254145f70 con 0x7fc2540b1b00 2024-03-25T18:35:51.039 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:35:51.037+0000 7fc252ffd640 1 -- 172.21.15.1:0/1343265478 <== mon.1 v1:172.21.15.115:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 542496408 0 0) 0x7fc2540ab0e0 con 0x7fc2540b44b0 2024-03-25T18:35:51.039 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:35:51.037+0000 7fc252ffd640 10 monclient(hunting): _negotiate 2024-03-25T18:35:51.039 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:35:51.037+0000 7fc252ffd640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:35:51.039 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:35:51.037+0000 7fc252ffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:35:51.040 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:35:51.037+0000 7fc252ffd640 10 monclient(hunting): my global_id is 4618 2024-03-25T18:35:51.040 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:35:51.037+0000 7fc252ffd640 1 -- 172.21.15.1:0/1343265478 --> v1:172.21.15.115:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fc2540a96f0 con 0x7fc2540b44b0 2024-03-25T18:35:51.047 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:35:51.045+0000 7fc252ffd640 1 -- 172.21.15.1:0/1343265478 <== mon.2 v1:172.21.15.1:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2029670531 0 0) 0x7fc23c003520 con 0x7fc2540b4e70 2024-03-25T18:35:51.047 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:35:51.045+0000 7fc252ffd640 1 -- 172.21.15.1:0/1343265478 >> v1:172.21.15.115:6789/0 conn(0x7fc2540b44b0 legacy=0x7fc254145860 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:35:51.047 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:35:51.045+0000 7fc252ffd640 1 -- 172.21.15.1:0/1343265478 >> v1:172.21.15.1:6789/0 conn(0x7fc2540b1b00 legacy=0x7fc25414dc00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:35:51.047 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:35:51.045+0000 7fc252ffd640 10 monclient: _finish_hunting 0 2024-03-25T18:35:51.047 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:35:51.045+0000 7fc252ffd640 1 monclient: found mon.noname-b 2024-03-25T18:35:51.047 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:35:51.045+0000 7fc252ffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:35:51.047 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:35:51.045+0000 7fc252ffd640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.1:6790/0 2024-03-25T18:35:51.047 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:35:51.045+0000 7fc252ffd640 1 -- 172.21.15.1:0/1343265478 --> v1:172.21.15.1:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc2540a9b80 con 0x7fc2540b4e70 2024-03-25T18:35:51.047 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:35:51.045+0000 7fc252ffd640 10 monclient: _finish_auth 0 2024-03-25T18:35:51.047 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:35:51.045+0000 7fc252ffd640 10 monclient: _check_auth_tickets 2024-03-25T18:35:51.048 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:35:51.045+0000 7fc252ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:35:51.048 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:35:51.045+0000 7fc252ffd640 1 -- 172.21.15.1:0/1343265478 <== mon.2 v1:172.21.15.1:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1586965025 0 0) 0x7fc244001690 con 0x7fc2540b4e70 2024-03-25T18:35:51.048 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:35:51.045+0000 7fc252ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:35:51.048 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:35:51.045+0000 7fc252ffd640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-03-25T18:35:51.048 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:35:51.045+0000 7fc252ffd640 10 monclient: dump: 2024-03-25T18:35:51.048 INFO:teuthology.misc.health.smithi001.stderr:epoch 1 2024-03-25T18:35:51.048 INFO:teuthology.misc.health.smithi001.stderr:fsid c8ea72cf-9e99-4383-9b52-6c23d365fdfc 2024-03-25T18:35:51.048 INFO:teuthology.misc.health.smithi001.stderr:last_changed 2024-03-25T18:25:50.510519+0000 2024-03-25T18:35:51.048 INFO:teuthology.misc.health.smithi001.stderr:created 2024-03-25T18:25:50.510519+0000 2024-03-25T18:35:51.048 INFO:teuthology.misc.health.smithi001.stderr:min_mon_release 19 (squid) 2024-03-25T18:35:51.048 INFO:teuthology.misc.health.smithi001.stderr:election_strategy: 3 2024-03-25T18:35:51.048 INFO:teuthology.misc.health.smithi001.stderr:0: v1:172.21.15.1:6789/0 mon.a 2024-03-25T18:35:51.048 INFO:teuthology.misc.health.smithi001.stderr:1: v1:172.21.15.115:6789/0 mon.b 2024-03-25T18:35:51.048 INFO:teuthology.misc.health.smithi001.stderr:2: v1:172.21.15.1:6790/0 mon.c 2024-03-25T18:35:51.048 INFO:teuthology.misc.health.smithi001.stderr: 2024-03-25T18:35:51.048 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:35:51.045+0000 7fc25a2c9640 5 monclient: authenticate success, global_id 4619 2024-03-25T18:35:51.049 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:35:51.045+0000 7fc25a2c9640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T18:35:51.055 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:35:51.054+0000 7fc252ffd640 1 -- 172.21.15.1:0/1343265478 <== mon.2 v1:172.21.15.1:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fc2440017f0 con 0x7fc2540b4e70 2024-03-25T18:35:51.056 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:35:51.054+0000 7fc252ffd640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:35:51.056 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:35:51.054+0000 7fc252ffd640 1 -- 172.21.15.1:0/1343265478 <== mon.2 v1:172.21.15.1:6790/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1586965025 0 0) 0x7fc244001c00 con 0x7fc2540b4e70 2024-03-25T18:35:51.056 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:35:51.054+0000 7fc252ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:35:51.056 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:35:51.054+0000 7fc252ffd640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-25T18:35:51.056 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:35:51.054+0000 7fc252ffd640 10 monclient: dump: 2024-03-25T18:35:51.056 INFO:teuthology.misc.health.smithi001.stderr:epoch 1 2024-03-25T18:35:51.056 INFO:teuthology.misc.health.smithi001.stderr:fsid c8ea72cf-9e99-4383-9b52-6c23d365fdfc 2024-03-25T18:35:51.056 INFO:teuthology.misc.health.smithi001.stderr:last_changed 2024-03-25T18:25:50.510519+0000 2024-03-25T18:35:51.056 INFO:teuthology.misc.health.smithi001.stderr:created 2024-03-25T18:25:50.510519+0000 2024-03-25T18:35:51.056 INFO:teuthology.misc.health.smithi001.stderr:min_mon_release 19 (squid) 2024-03-25T18:35:51.056 INFO:teuthology.misc.health.smithi001.stderr:election_strategy: 3 2024-03-25T18:35:51.056 INFO:teuthology.misc.health.smithi001.stderr:0: v1:172.21.15.1:6789/0 mon.a 2024-03-25T18:35:51.056 INFO:teuthology.misc.health.smithi001.stderr:1: v1:172.21.15.115:6789/0 mon.b 2024-03-25T18:35:51.057 INFO:teuthology.misc.health.smithi001.stderr:2: v1:172.21.15.1:6790/0 mon.c 2024-03-25T18:35:51.057 INFO:teuthology.misc.health.smithi001.stderr: 2024-03-25T18:35:51.057 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:35:51.054+0000 7fc25a2c9640 10 monclient: get_monmap_and_config success 2024-03-25T18:35:51.057 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:35:51.054+0000 7fc25a2c9640 4 set_mon_vals no callback set 2024-03-25T18:35:51.059 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:35:51.055+0000 7fc25a2c9640 10 monclient: shutdown 2024-03-25T18:35:51.059 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:35:51.055+0000 7fc25a2c9640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:35:51.059 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:35:51.055+0000 7fc25a2c9640 1 -- 172.21.15.1:0/1343265478 >> v1:172.21.15.1:6790/0 conn(0x7fc2540b4e70 legacy=0x7fc25413f940 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:35:51.059 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:35:51.055+0000 7fc25a2c9640 1 -- 172.21.15.1:0/1343265478 shutdown_connections 2024-03-25T18:35:51.059 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:35:51.055+0000 7fc25a2c9640 1 -- 172.21.15.1:0/1343265478 >> 172.21.15.1:0/1343265478 conn(0x7fc2540617a0 msgr2=0x7fc254061bb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:35:51.059 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:35:51.055+0000 7fc25a2c9640 1 -- 172.21.15.1:0/1343265478 shutdown_connections 2024-03-25T18:35:51.059 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:35:51.055+0000 7fc25a2c9640 1 -- 172.21.15.1:0/1343265478 wait complete. 2024-03-25T18:35:51.059 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:35:51.056+0000 7fc25a2c9640 10 monclient: build_initial_monmap 2024-03-25T18:35:51.060 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:35:51.056+0000 7fc25a2c9640 10 monclient: monmap: 2024-03-25T18:35:51.060 INFO:teuthology.misc.health.smithi001.stderr:epoch 0 2024-03-25T18:35:51.060 INFO:teuthology.misc.health.smithi001.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T18:35:51.060 INFO:teuthology.misc.health.smithi001.stderr:last_changed 0.000000 2024-03-25T18:35:51.060 INFO:teuthology.misc.health.smithi001.stderr:created 0.000000 2024-03-25T18:35:51.060 INFO:teuthology.misc.health.smithi001.stderr:min_mon_release 0 (unknown) 2024-03-25T18:35:51.060 INFO:teuthology.misc.health.smithi001.stderr:election_strategy: 1 2024-03-25T18:35:51.060 INFO:teuthology.misc.health.smithi001.stderr:0: v1:172.21.15.1:6789/0 mon.noname-a 2024-03-25T18:35:51.060 INFO:teuthology.misc.health.smithi001.stderr:1: v1:172.21.15.115:6789/0 mon.noname-b 2024-03-25T18:35:51.060 INFO:teuthology.misc.health.smithi001.stderr:2: v1:172.21.15.1:6790/0 mon.noname-c 2024-03-25T18:35:51.060 INFO:teuthology.misc.health.smithi001.stderr: 2024-03-25T18:35:51.060 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:35:51.056+0000 7fc25a2c9640 1 librados: starting msgr at 2024-03-25T18:35:51.060 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:35:51.056+0000 7fc25a2c9640 1 librados: starting objecter 2024-03-25T18:35:51.060 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:35:51.056+0000 7fc25a2c9640 1 Processor -- start 2024-03-25T18:35:51.060 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:35:51.056+0000 7fc25a2c9640 1 -- start start 2024-03-25T18:35:51.060 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:35:51.056+0000 7fc25a2c9640 1 librados: setting wanted keys 2024-03-25T18:35:51.061 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:35:51.056+0000 7fc25a2c9640 1 librados: calling monclient init 2024-03-25T18:35:51.061 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:35:51.056+0000 7fc25a2c9640 10 monclient: init 2024-03-25T18:35:51.061 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:35:51.057+0000 7fc25a2c9640 10 monclient: _reopen_session rank -1 2024-03-25T18:35:51.061 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:35:51.057+0000 7fc25a2c9640 10 monclient: _add_conns ranks=[1,2,0] 2024-03-25T18:35:51.061 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:35:51.057+0000 7fc25a2c9640 10 monclient(hunting): picked mon.noname-b con 0x7fc2540b1b00 addr v1:172.21.15.115:6789/0 2024-03-25T18:35:51.061 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:35:51.057+0000 7fc25a2c9640 10 monclient(hunting): picked mon.noname-c con 0x7fc2540b44b0 addr v1:172.21.15.1:6790/0 2024-03-25T18:35:51.061 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:35:51.057+0000 7fc25a2c9640 10 monclient(hunting): picked mon.noname-a con 0x7fc254144d90 addr v1:172.21.15.1:6789/0 2024-03-25T18:35:51.061 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:35:51.057+0000 7fc25a2c9640 1 -- --> v1:172.21.15.1:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc254143140 con 0x7fc254144d90 2024-03-25T18:35:51.061 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:35:51.057+0000 7fc25a2c9640 1 -- --> v1:172.21.15.115:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc254140220 con 0x7fc2540b1b00 2024-03-25T18:35:51.061 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:35:51.057+0000 7fc25a2c9640 1 -- --> v1:172.21.15.1:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc254127f50 con 0x7fc2540b44b0 2024-03-25T18:35:51.061 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:35:51.057+0000 7fc25a2c9640 10 monclient(hunting): _renew_subs 2024-03-25T18:35:51.061 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:35:51.057+0000 7fc25a2c9640 10 monclient(hunting): authenticate will time out at 8031.528378s 2024-03-25T18:35:51.070 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:35:51.068+0000 7fc2537fe640 1 --1- >> v1:172.21.15.1:6790/0 conn(0x7fc2540b44b0 0x7fc254141730 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.1:6790/0 says I am v1:172.21.15.1:40964/0 (socket says 172.21.15.1:40964) 2024-03-25T18:35:51.070 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:35:51.068+0000 7fc2537fe640 1 -- 172.21.15.1:0/1640727737 learned_addr learned my addr 172.21.15.1:0/1640727737 (peer_addr_for_me v1:172.21.15.1:0/0) 2024-03-25T18:35:51.071 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:35:51.069+0000 7fc250ff9640 10 client.?.objecter ms_handle_connect 0x7fc2540b44b0 2024-03-25T18:35:51.071 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:35:51.069+0000 7fc250ff9640 10 client.?.objecter resend_mon_ops 2024-03-25T18:35:51.074 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:35:51.073+0000 7fc250ff9640 10 client.?.objecter ms_handle_connect 0x7fc2540b1b00 2024-03-25T18:35:51.075 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:35:51.073+0000 7fc250ff9640 10 client.?.objecter resend_mon_ops 2024-03-25T18:35:51.078 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:35:51.077+0000 7fc250ff9640 1 -- 172.21.15.1:0/1640727737 <== mon.2 v1:172.21.15.1:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3555037154 0 0) 0x7fc254127f50 con 0x7fc2540b44b0 2024-03-25T18:35:51.078 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:35:51.077+0000 7fc250ff9640 10 monclient(hunting): _negotiate 2024-03-25T18:35:51.078 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:35:51.077+0000 7fc250ff9640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:35:51.078 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:35:51.077+0000 7fc250ff9640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:35:51.079 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:35:51.077+0000 7fc250ff9640 10 monclient(hunting): my global_id is 4622 2024-03-25T18:35:51.079 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:35:51.077+0000 7fc250ff9640 1 -- 172.21.15.1:0/1640727737 --> v1:172.21.15.1:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fc240004eb0 con 0x7fc2540b44b0 2024-03-25T18:35:51.079 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:35:51.077+0000 7fc250ff9640 1 -- 172.21.15.1:0/1640727737 <== mon.1 v1:172.21.15.115:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2722661520 0 0) 0x7fc254140220 con 0x7fc2540b1b00 2024-03-25T18:35:51.080 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:35:51.078+0000 7fc250ff9640 10 monclient(hunting): _negotiate 2024-03-25T18:35:51.080 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:35:51.078+0000 7fc250ff9640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:35:51.080 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:35:51.078+0000 7fc250ff9640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:35:51.080 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:35:51.078+0000 7fc250ff9640 10 monclient(hunting): my global_id is 4621 2024-03-25T18:35:51.080 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:35:51.078+0000 7fc250ff9640 1 -- 172.21.15.1:0/1640727737 --> v1:172.21.15.115:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fc254127f50 con 0x7fc2540b1b00 2024-03-25T18:35:51.080 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:35:51.079+0000 7fc250ff9640 10 client.?.objecter ms_handle_connect 0x7fc254144d90 2024-03-25T18:35:51.080 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:35:51.079+0000 7fc250ff9640 10 client.?.objecter resend_mon_ops 2024-03-25T18:35:51.088 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:35:51.086+0000 7fc250ff9640 1 -- 172.21.15.1:0/1640727737 <== mon.1 v1:172.21.15.115:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1755424385 0 0) 0x7fc254127f50 con 0x7fc2540b1b00 2024-03-25T18:35:51.088 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:35:51.086+0000 7fc250ff9640 1 -- 172.21.15.1:0/1640727737 --> v1:172.21.15.115:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fc254140220 con 0x7fc2540b1b00 2024-03-25T18:35:51.088 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:35:51.086+0000 7fc250ff9640 1 -- 172.21.15.1:0/1640727737 <== mon.1 v1:172.21.15.115:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1586965025 0 0) 0x7fc244001b00 con 0x7fc2540b1b00 2024-03-25T18:35:51.088 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:35:51.086+0000 7fc250ff9640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:35:51.088 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:35:51.086+0000 7fc250ff9640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-03-25T18:35:51.088 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:35:51.086+0000 7fc250ff9640 10 monclient(hunting): dump: 2024-03-25T18:35:51.088 INFO:teuthology.misc.health.smithi001.stderr:epoch 1 2024-03-25T18:35:51.088 INFO:teuthology.misc.health.smithi001.stderr:fsid c8ea72cf-9e99-4383-9b52-6c23d365fdfc 2024-03-25T18:35:51.088 INFO:teuthology.misc.health.smithi001.stderr:last_changed 2024-03-25T18:25:50.510519+0000 2024-03-25T18:35:51.089 INFO:teuthology.misc.health.smithi001.stderr:created 2024-03-25T18:25:50.510519+0000 2024-03-25T18:35:51.089 INFO:teuthology.misc.health.smithi001.stderr:min_mon_release 19 (squid) 2024-03-25T18:35:51.089 INFO:teuthology.misc.health.smithi001.stderr:election_strategy: 3 2024-03-25T18:35:51.089 INFO:teuthology.misc.health.smithi001.stderr:0: v1:172.21.15.1:6789/0 mon.a 2024-03-25T18:35:51.089 INFO:teuthology.misc.health.smithi001.stderr:1: v1:172.21.15.115:6789/0 mon.b 2024-03-25T18:35:51.089 INFO:teuthology.misc.health.smithi001.stderr:2: v1:172.21.15.1:6790/0 mon.c 2024-03-25T18:35:51.089 INFO:teuthology.misc.health.smithi001.stderr: 2024-03-25T18:35:51.089 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:35:51.086+0000 7fc250ff9640 10 monclient(hunting): _finish_auth 0 2024-03-25T18:35:51.092 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:35:51.090+0000 7fc250ff9640 1 -- 172.21.15.1:0/1640727737 <== mon.0 v1:172.21.15.1:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3280926244 0 0) 0x7fc254143140 con 0x7fc254144d90 2024-03-25T18:35:51.092 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:35:51.090+0000 7fc250ff9640 10 monclient(hunting): _negotiate 2024-03-25T18:35:51.092 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:35:51.090+0000 7fc250ff9640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:35:51.092 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:35:51.090+0000 7fc250ff9640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:35:51.092 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:35:51.090+0000 7fc250ff9640 10 monclient(hunting): my global_id is 4614 2024-03-25T18:35:51.092 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:35:51.090+0000 7fc250ff9640 1 -- 172.21.15.1:0/1640727737 --> v1:172.21.15.1:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fc254127f50 con 0x7fc254144d90 2024-03-25T18:35:51.098 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:35:51.095+0000 7fc250ff9640 1 -- 172.21.15.1:0/1640727737 <== mon.1 v1:172.21.15.115:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 135610671 0 0) 0x7fc254140220 con 0x7fc2540b1b00 2024-03-25T18:35:51.098 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:35:51.095+0000 7fc250ff9640 1 -- 172.21.15.1:0/1640727737 >> v1:172.21.15.1:6790/0 conn(0x7fc2540b44b0 legacy=0x7fc254141730 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:35:51.098 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:35:51.095+0000 7fc250ff9640 1 -- 172.21.15.1:0/1640727737 >> v1:172.21.15.1:6789/0 conn(0x7fc254144d90 legacy=0x7fc2541e2800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:35:51.098 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:35:51.095+0000 7fc250ff9640 10 monclient: _finish_hunting 0 2024-03-25T18:35:51.098 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:35:51.095+0000 7fc250ff9640 1 monclient: found mon.b 2024-03-25T18:35:51.098 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:35:51.095+0000 7fc250ff9640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:35:51.098 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:35:51.095+0000 7fc250ff9640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.115:6789/0 2024-03-25T18:35:51.098 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:35:51.095+0000 7fc250ff9640 1 -- 172.21.15.1:0/1640727737 --> v1:172.21.15.115:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc254141e40 con 0x7fc2540b1b00 2024-03-25T18:35:51.098 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:35:51.095+0000 7fc250ff9640 10 monclient: _finish_auth 0 2024-03-25T18:35:51.098 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:35:51.095+0000 7fc250ff9640 10 monclient: _check_auth_tickets 2024-03-25T18:35:51.098 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:35:51.095+0000 7fc250ff9640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:35:51.098 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:35:51.095+0000 7fc25a2c9640 5 monclient: authenticate success, global_id 4621 2024-03-25T18:35:51.098 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:35:51.095+0000 7fc25a2c9640 10 monclient: _renew_subs 2024-03-25T18:35:51.099 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:35:51.095+0000 7fc25a2c9640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.115:6789/0 2024-03-25T18:35:51.099 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:35:51.095+0000 7fc25a2c9640 1 -- 172.21.15.1:0/1640727737 --> v1:172.21.15.115:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc254142090 con 0x7fc2540b1b00 2024-03-25T18:35:51.099 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:35:51.096+0000 7fc25a2c9640 10 client.4621.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T18:35:51.099 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:35:51.096+0000 7fc25a2c9640 10 monclient: _renew_subs 2024-03-25T18:35:51.099 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:35:51.096+0000 7fc25a2c9640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.115:6789/0 2024-03-25T18:35:51.099 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:35:51.096+0000 7fc25a2c9640 1 -- 172.21.15.1:0/1640727737 --> v1:172.21.15.115:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7fc25413a270 con 0x7fc2540b1b00 2024-03-25T18:35:51.099 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:35:51.096+0000 7fc25a2c9640 1 librados: init done 2024-03-25T18:35:51.099 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:35:51.097+0000 7fc25a2c9640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T18:35:51.100 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:35:51.097+0000 7fc25a2c9640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:35:51.100 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:35:51.097+0000 7fc25a2c9640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.115:6789/0 2024-03-25T18:35:51.100 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:35:51.097+0000 7fc25a2c9640 1 -- 172.21.15.1:0/1640727737 --> v1:172.21.15.115:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc2541403b0 con 0x7fc2540b1b00 2024-03-25T18:35:51.121 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:35:51.118+0000 7fc250ff9640 1 -- 172.21.15.1:0/1640727737 <== mon.1 v1:172.21.15.115:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fc244004a90 con 0x7fc2540b1b00 2024-03-25T18:35:51.121 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:35:51.118+0000 7fc250ff9640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:35:51.121 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:35:51.119+0000 7fc250ff9640 1 -- 172.21.15.1:0/1640727737 <== mon.1 v1:172.21.15.115:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1586965025 0 0) 0x7fc244004da0 con 0x7fc2540b1b00 2024-03-25T18:35:51.121 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:35:51.119+0000 7fc250ff9640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:35:51.121 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:35:51.119+0000 7fc250ff9640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-25T18:35:51.121 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:35:51.119+0000 7fc250ff9640 10 monclient: dump: 2024-03-25T18:35:51.121 INFO:teuthology.misc.health.smithi001.stderr:epoch 1 2024-03-25T18:35:51.121 INFO:teuthology.misc.health.smithi001.stderr:fsid c8ea72cf-9e99-4383-9b52-6c23d365fdfc 2024-03-25T18:35:51.121 INFO:teuthology.misc.health.smithi001.stderr:last_changed 2024-03-25T18:25:50.510519+0000 2024-03-25T18:35:51.121 INFO:teuthology.misc.health.smithi001.stderr:created 2024-03-25T18:25:50.510519+0000 2024-03-25T18:35:51.121 INFO:teuthology.misc.health.smithi001.stderr:min_mon_release 19 (squid) 2024-03-25T18:35:51.121 INFO:teuthology.misc.health.smithi001.stderr:election_strategy: 3 2024-03-25T18:35:51.121 INFO:teuthology.misc.health.smithi001.stderr:0: v1:172.21.15.1:6789/0 mon.a 2024-03-25T18:35:51.121 INFO:teuthology.misc.health.smithi001.stderr:1: v1:172.21.15.115:6789/0 mon.b 2024-03-25T18:35:51.121 INFO:teuthology.misc.health.smithi001.stderr:2: v1:172.21.15.1:6790/0 mon.c 2024-03-25T18:35:51.122 INFO:teuthology.misc.health.smithi001.stderr: 2024-03-25T18:35:51.122 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:35:51.119+0000 7fc2527fc640 4 set_mon_vals no callback set 2024-03-25T18:35:51.122 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:35:51.120+0000 7fc250ff9640 1 -- 172.21.15.1:0/1640727737 <== mon.1 v1:172.21.15.115:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 21787975 0 0) 0x7fc24401ad70 con 0x7fc2540b1b00 2024-03-25T18:35:51.122 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:35:51.120+0000 7fc2537fe640 10 monclient: get_auth_request con 0x7fc24006de10 auth_method 0 2024-03-25T18:35:51.122 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:35:51.120+0000 7fc250ff9640 10 client.4621.objecter ms_handle_connect 0x7fc24006de10 2024-03-25T18:35:51.124 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:35:51.122+0000 7fc250ff9640 1 -- 172.21.15.1:0/1640727737 <== mon.1 v1:172.21.15.115:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 996254230 0 0) 0x7fc244054760 con 0x7fc2540b1b00 2024-03-25T18:35:51.124 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:35:51.122+0000 7fc250ff9640 10 client.4621.objecter ms_dispatch 0x7fc254000b90 osd_map(10..10 src has 1..10) v4 2024-03-25T18:35:51.124 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:35:51.122+0000 7fc250ff9640 3 client.4621.objecter handle_osd_map got epochs [10,10] > 0 2024-03-25T18:35:51.124 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:35:51.122+0000 7fc250ff9640 3 client.4621.objecter handle_osd_map decoding full epoch 10 2024-03-25T18:35:51.124 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:35:51.122+0000 7fc250ff9640 20 client.4621.objecter dump_active .. 0 homeless 2024-03-25T18:35:51.380 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:35:51.378+0000 7fc250ff9640 1 -- 172.21.15.1:0/1640727737 <== mon.1 v1:172.21.15.115:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7fc244050270 con 0x7fc2540b1b00 2024-03-25T18:35:51.380 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:35:51.378+0000 7fc250ff9640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:35:51.380 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:35:51.378+0000 7fc250ff9640 10 monclient: _finish_command 1 = system:0 2024-03-25T18:35:51.632 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:35:51.630+0000 7fc25a2c9640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:35:51.632 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:35:51.630+0000 7fc25a2c9640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:35:51.632 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:35:51.630+0000 7fc25a2c9640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.115:6789/0 2024-03-25T18:35:51.632 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:35:51.630+0000 7fc25a2c9640 1 -- 172.21.15.1:0/1640727737 --> v1:172.21.15.115:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fc2540b2710 con 0x7fc2540b1b00 2024-03-25T18:35:51.653 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:35:51.651+0000 7fc250ff9640 1 -- 172.21.15.1:0/1640727737 <== mon.1 v1:172.21.15.115:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 4092213348) 0x7fc2440521a0 con 0x7fc2540b1b00 2024-03-25T18:35:51.653 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:35:51.651+0000 7fc250ff9640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:35:51.653 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:35:51.651+0000 7fc250ff9640 10 monclient: _finish_command 2 = system:0 2024-03-25T18:35:51.653 INFO:teuthology.misc.health.smithi001.stdout: 2024-03-25T18:35:51.653 INFO:teuthology.misc.health.smithi001.stdout:{"epoch":10,"fsid":"c8ea72cf-9e99-4383-9b52-6c23d365fdfc","created":"2024-03-25T18:26:38.856638+0000","modified":"2024-03-25T18:26:44.224208+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T18:26:44.224208+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"b9f34922-3a84-49be-8426-807c1eceaefc","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"42bf306f-55e3-4634-9874-8b4b73ac6644","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"292cb23a-18a9-4207-a7cc-4d5727feaec0","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"45e0e9ab-898d-4673-9590-b4117bc5e84a","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"831a5a09-73ab-4a61-9872-088e76bdbc66","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"88cd903d-3b55-4840-9396-6e3cfe005282","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"89db7355-bc9d-46f4-8ab0-e8d9e744e026","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"cac47bee-0914-4f08-896f-348838882866","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T18:35:51.656 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:35:51.653+0000 7fc25a2c9640 10 librados: watch_flush enter 2024-03-25T18:35:51.656 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:35:51.653+0000 7fc25a2c9640 10 librados: watch_flush exit 2024-03-25T18:35:51.656 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:35:51.653+0000 7fc25a2c9640 20 client.4621.objecter shutdown clearing up homeless session... 2024-03-25T18:35:51.656 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:35:51.653+0000 7fc25a2c9640 10 client.4621.objecter successfully canceled tick 2024-03-25T18:35:51.656 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:35:51.653+0000 7fc25a2c9640 1 -- 172.21.15.1:0/1640727737 >> v1:172.21.15.115:6800/4237379176 conn(0x7fc24006de10 legacy=0x7fc2400702d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:35:51.656 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:35:51.653+0000 7fc25a2c9640 10 monclient: shutdown 2024-03-25T18:35:51.656 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:35:51.653+0000 7fc25a2c9640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:35:51.656 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:35:51.653+0000 7fc25a2c9640 1 -- 172.21.15.1:0/1640727737 >> v1:172.21.15.115:6789/0 conn(0x7fc2540b1b00 legacy=0x7fc254140d50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:35:51.656 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:35:51.654+0000 7fc25a2c9640 1 -- 172.21.15.1:0/1640727737 shutdown_connections 2024-03-25T18:35:51.656 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:35:51.654+0000 7fc25a2c9640 1 -- 172.21.15.1:0/1640727737 >> 172.21.15.1:0/1640727737 conn(0x7fc2540617a0 msgr2=0x7fc2540b6520 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:35:51.656 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:35:51.654+0000 7fc25a2c9640 1 -- 172.21.15.1:0/1640727737 shutdown_connections 2024-03-25T18:35:51.657 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:35:51.654+0000 7fc25a2c9640 1 -- 172.21.15.1:0/1640727737 wait complete. 2024-03-25T18:35:51.657 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:35:51.654+0000 7fc25a2c9640 1 librados: shutdown 2024-03-25T18:35:51.668 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T18:35:58.475 DEBUG:teuthology.orchestra.run.smithi001:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T18:35:58.587 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:35:58.585+0000 7f423a152640 10 monclient: get_monmap_and_config 2024-03-25T18:35:58.587 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:35:58.585+0000 7f423a152640 10 monclient: build_initial_monmap 2024-03-25T18:35:58.587 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:35:58.585+0000 7f423a152640 10 monclient: monmap: 2024-03-25T18:35:58.587 INFO:teuthology.misc.health.smithi001.stderr:epoch 0 2024-03-25T18:35:58.588 INFO:teuthology.misc.health.smithi001.stderr:fsid c8ea72cf-9e99-4383-9b52-6c23d365fdfc 2024-03-25T18:35:58.588 INFO:teuthology.misc.health.smithi001.stderr:last_changed 2024-03-25T18:35:58.587414+0000 2024-03-25T18:35:58.588 INFO:teuthology.misc.health.smithi001.stderr:created 2024-03-25T18:35:58.587414+0000 2024-03-25T18:35:58.588 INFO:teuthology.misc.health.smithi001.stderr:min_mon_release 0 (unknown) 2024-03-25T18:35:58.588 INFO:teuthology.misc.health.smithi001.stderr:election_strategy: 3 2024-03-25T18:35:58.588 INFO:teuthology.misc.health.smithi001.stderr:0: v1:172.21.15.1:6789/0 mon.noname-a 2024-03-25T18:35:58.588 INFO:teuthology.misc.health.smithi001.stderr:1: v1:172.21.15.115:6789/0 mon.noname-c 2024-03-25T18:35:58.588 INFO:teuthology.misc.health.smithi001.stderr:2: v1:172.21.15.1:6790/0 mon.noname-b 2024-03-25T18:35:58.588 INFO:teuthology.misc.health.smithi001.stderr: 2024-03-25T18:35:58.589 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:35:58.586+0000 7f423a152640 1 Processor -- start 2024-03-25T18:35:58.589 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:35:58.586+0000 7f423a152640 1 -- start start 2024-03-25T18:35:58.589 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:35:58.586+0000 7f423a152640 10 monclient: init 2024-03-25T18:35:58.590 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:35:58.587+0000 7f423a152640 10 monclient: _reopen_session rank -1 2024-03-25T18:35:58.590 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:35:58.587+0000 7f423a152640 10 monclient: _add_conns ranks=[0,1,2] 2024-03-25T18:35:58.590 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:35:58.587+0000 7f423a152640 10 monclient(hunting): picked mon.noname-a con 0x7f42340b4e70 addr v1:172.21.15.1:6789/0 2024-03-25T18:35:58.590 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:35:58.587+0000 7f423a152640 10 monclient(hunting): picked mon.noname-c con 0x7f42340b44b0 addr v1:172.21.15.115:6789/0 2024-03-25T18:35:58.590 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:35:58.587+0000 7f423a152640 10 monclient(hunting): picked mon.noname-b con 0x7f42340b1b00 addr v1:172.21.15.1:6790/0 2024-03-25T18:35:58.590 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:35:58.587+0000 7f423a152640 1 -- --> v1:172.21.15.1:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f42340a96f0 con 0x7f42340b4e70 2024-03-25T18:35:58.590 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:35:58.587+0000 7f423a152640 1 -- --> v1:172.21.15.115:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f42340ab0e0 con 0x7f42340b44b0 2024-03-25T18:35:58.590 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:35:58.587+0000 7f423a152640 1 -- --> v1:172.21.15.1:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4234145f70 con 0x7f42340b1b00 2024-03-25T18:35:58.590 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:35:58.587+0000 7f423a152640 10 monclient(hunting): _renew_subs 2024-03-25T18:35:58.590 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:35:58.587+0000 7f423a152640 10 monclient(hunting): authenticate will time out at 8039.058271s 2024-03-25T18:35:58.593 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:35:58.591+0000 7f42337fe640 1 --1- >> v1:172.21.15.1:6789/0 conn(0x7f42340b4e70 0x7f423413f940 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.1:6789/0 says I am v1:172.21.15.1:36730/0 (socket says 172.21.15.1:36730) 2024-03-25T18:35:58.593 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:35:58.591+0000 7f42337fe640 1 -- 172.21.15.1:0/1755394588 learned_addr learned my addr 172.21.15.1:0/1755394588 (peer_addr_for_me v1:172.21.15.1:0/0) 2024-03-25T18:35:58.604 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:35:58.602+0000 7f42327fc640 1 -- 172.21.15.1:0/1755394588 <== mon.0 v1:172.21.15.1:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4107119385 0 0) 0x7f42340a96f0 con 0x7f42340b4e70 2024-03-25T18:35:58.604 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:35:58.602+0000 7f42327fc640 10 monclient(hunting): _negotiate 2024-03-25T18:35:58.604 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:35:58.602+0000 7f42327fc640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:35:58.604 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:35:58.602+0000 7f42327fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:35:58.604 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:35:58.602+0000 7f42327fc640 10 monclient(hunting): my global_id is 4617 2024-03-25T18:35:58.604 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:35:58.602+0000 7f42327fc640 1 -- 172.21.15.1:0/1755394588 --> v1:172.21.15.1:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f421c003520 con 0x7f42340b4e70 2024-03-25T18:35:58.609 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:35:58.608+0000 7f42327fc640 1 -- 172.21.15.1:0/1755394588 <== mon.2 v1:172.21.15.1:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4147403722 0 0) 0x7f4234145f70 con 0x7f42340b1b00 2024-03-25T18:35:58.609 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:35:58.608+0000 7f42327fc640 10 monclient(hunting): _negotiate 2024-03-25T18:35:58.610 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:35:58.608+0000 7f42327fc640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:35:58.610 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:35:58.608+0000 7f42327fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:35:58.610 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:35:58.608+0000 7f42327fc640 10 monclient(hunting): my global_id is 4625 2024-03-25T18:35:58.610 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:35:58.608+0000 7f42327fc640 1 -- 172.21.15.1:0/1755394588 --> v1:172.21.15.1:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f42340a96f0 con 0x7f42340b1b00 2024-03-25T18:35:58.610 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:35:58.608+0000 7f42327fc640 1 -- 172.21.15.1:0/1755394588 <== mon.1 v1:172.21.15.115:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2810953324 0 0) 0x7f42340ab0e0 con 0x7f42340b44b0 2024-03-25T18:35:58.610 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:35:58.608+0000 7f42327fc640 10 monclient(hunting): _negotiate 2024-03-25T18:35:58.610 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:35:58.608+0000 7f42327fc640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:35:58.610 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:35:58.608+0000 7f42327fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:35:58.610 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:35:58.608+0000 7f42327fc640 10 monclient(hunting): my global_id is 4624 2024-03-25T18:35:58.610 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:35:58.608+0000 7f42327fc640 1 -- 172.21.15.1:0/1755394588 --> v1:172.21.15.115:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f4234145f70 con 0x7f42340b44b0 2024-03-25T18:35:58.611 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:35:58.609+0000 7f42327fc640 1 -- 172.21.15.1:0/1755394588 <== mon.0 v1:172.21.15.1:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1157543526 0 0) 0x7f421c003520 con 0x7f42340b4e70 2024-03-25T18:35:58.611 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:35:58.609+0000 7f42327fc640 1 -- 172.21.15.1:0/1755394588 >> v1:172.21.15.1:6790/0 conn(0x7f42340b1b00 legacy=0x7f423414dc00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:35:58.611 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:35:58.609+0000 7f42327fc640 1 -- 172.21.15.1:0/1755394588 >> v1:172.21.15.115:6789/0 conn(0x7f42340b44b0 legacy=0x7f4234145860 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:35:58.612 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:35:58.609+0000 7f42327fc640 10 monclient: _finish_hunting 0 2024-03-25T18:35:58.612 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:35:58.609+0000 7f42327fc640 1 monclient: found mon.noname-a 2024-03-25T18:35:58.612 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:35:58.609+0000 7f42327fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:35:58.612 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:35:58.609+0000 7f42327fc640 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.1:6789/0 2024-03-25T18:35:58.612 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:35:58.609+0000 7f42327fc640 1 -- 172.21.15.1:0/1755394588 --> v1:172.21.15.1:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f42340a9b80 con 0x7f42340b4e70 2024-03-25T18:35:58.612 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:35:58.609+0000 7f42327fc640 10 monclient: _finish_auth 0 2024-03-25T18:35:58.612 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:35:58.609+0000 7f42327fc640 10 monclient: _check_auth_tickets 2024-03-25T18:35:58.612 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:35:58.609+0000 7f42327fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:35:58.612 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:35:58.609+0000 7f42327fc640 1 -- 172.21.15.1:0/1755394588 <== mon.0 v1:172.21.15.1:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1586965025 0 0) 0x7f4224001690 con 0x7f42340b4e70 2024-03-25T18:35:58.612 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:35:58.609+0000 7f42327fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:35:58.612 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:35:58.609+0000 7f42327fc640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-03-25T18:35:58.612 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:35:58.609+0000 7f42327fc640 10 monclient: dump: 2024-03-25T18:35:58.612 INFO:teuthology.misc.health.smithi001.stderr:epoch 1 2024-03-25T18:35:58.612 INFO:teuthology.misc.health.smithi001.stderr:fsid c8ea72cf-9e99-4383-9b52-6c23d365fdfc 2024-03-25T18:35:58.612 INFO:teuthology.misc.health.smithi001.stderr:last_changed 2024-03-25T18:25:50.510519+0000 2024-03-25T18:35:58.612 INFO:teuthology.misc.health.smithi001.stderr:created 2024-03-25T18:25:50.510519+0000 2024-03-25T18:35:58.613 INFO:teuthology.misc.health.smithi001.stderr:min_mon_release 19 (squid) 2024-03-25T18:35:58.613 INFO:teuthology.misc.health.smithi001.stderr:election_strategy: 3 2024-03-25T18:35:58.613 INFO:teuthology.misc.health.smithi001.stderr:0: v1:172.21.15.1:6789/0 mon.a 2024-03-25T18:35:58.613 INFO:teuthology.misc.health.smithi001.stderr:1: v1:172.21.15.115:6789/0 mon.b 2024-03-25T18:35:58.613 INFO:teuthology.misc.health.smithi001.stderr:2: v1:172.21.15.1:6790/0 mon.c 2024-03-25T18:35:58.613 INFO:teuthology.misc.health.smithi001.stderr: 2024-03-25T18:35:58.613 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:35:58.609+0000 7f423a152640 5 monclient: authenticate success, global_id 4617 2024-03-25T18:35:58.613 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:35:58.609+0000 7f423a152640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T18:35:58.622 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:35:58.621+0000 7f42327fc640 1 -- 172.21.15.1:0/1755394588 <== mon.0 v1:172.21.15.1:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f42240017f0 con 0x7f42340b4e70 2024-03-25T18:35:58.622 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:35:58.621+0000 7f42327fc640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:35:58.622 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:35:58.621+0000 7f42327fc640 1 -- 172.21.15.1:0/1755394588 <== mon.0 v1:172.21.15.1:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1586965025 0 0) 0x7f4224001c00 con 0x7f42340b4e70 2024-03-25T18:35:58.622 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:35:58.621+0000 7f423a152640 10 monclient: get_monmap_and_config success 2024-03-25T18:35:58.622 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:35:58.621+0000 7f423a152640 4 set_mon_vals no callback set 2024-03-25T18:35:58.623 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:35:58.621+0000 7f42327fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:35:58.623 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:35:58.621+0000 7f42327fc640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-25T18:35:58.623 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:35:58.621+0000 7f42327fc640 10 monclient: dump: 2024-03-25T18:35:58.623 INFO:teuthology.misc.health.smithi001.stderr:epoch 1 2024-03-25T18:35:58.623 INFO:teuthology.misc.health.smithi001.stderr:fsid c8ea72cf-9e99-4383-9b52-6c23d365fdfc 2024-03-25T18:35:58.623 INFO:teuthology.misc.health.smithi001.stderr:last_changed 2024-03-25T18:25:50.510519+0000 2024-03-25T18:35:58.623 INFO:teuthology.misc.health.smithi001.stderr:created 2024-03-25T18:25:50.510519+0000 2024-03-25T18:35:58.623 INFO:teuthology.misc.health.smithi001.stderr:min_mon_release 19 (squid) 2024-03-25T18:35:58.623 INFO:teuthology.misc.health.smithi001.stderr:election_strategy: 3 2024-03-25T18:35:58.623 INFO:teuthology.misc.health.smithi001.stderr:0: v1:172.21.15.1:6789/0 mon.a 2024-03-25T18:35:58.623 INFO:teuthology.misc.health.smithi001.stderr:1: v1:172.21.15.115:6789/0 mon.b 2024-03-25T18:35:58.623 INFO:teuthology.misc.health.smithi001.stderr:2: v1:172.21.15.1:6790/0 mon.c 2024-03-25T18:35:58.623 INFO:teuthology.misc.health.smithi001.stderr: 2024-03-25T18:35:58.625 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:35:58.621+0000 7f423a152640 10 monclient: shutdown 2024-03-25T18:35:58.625 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:35:58.621+0000 7f423a152640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:35:58.625 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:35:58.621+0000 7f423a152640 1 -- 172.21.15.1:0/1755394588 >> v1:172.21.15.1:6789/0 conn(0x7f42340b4e70 legacy=0x7f423413f940 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:35:58.625 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:35:58.621+0000 7f423a152640 1 -- 172.21.15.1:0/1755394588 shutdown_connections 2024-03-25T18:35:58.625 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:35:58.622+0000 7f423a152640 1 -- 172.21.15.1:0/1755394588 >> 172.21.15.1:0/1755394588 conn(0x7f42340617a0 msgr2=0x7f4234061bb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:35:58.625 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:35:58.622+0000 7f423a152640 1 -- 172.21.15.1:0/1755394588 shutdown_connections 2024-03-25T18:35:58.625 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:35:58.622+0000 7f423a152640 1 -- 172.21.15.1:0/1755394588 wait complete. 2024-03-25T18:35:58.625 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:35:58.622+0000 7f423a152640 10 monclient: build_initial_monmap 2024-03-25T18:35:58.625 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:35:58.622+0000 7f423a152640 10 monclient: monmap: 2024-03-25T18:35:58.625 INFO:teuthology.misc.health.smithi001.stderr:epoch 0 2024-03-25T18:35:58.625 INFO:teuthology.misc.health.smithi001.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T18:35:58.625 INFO:teuthology.misc.health.smithi001.stderr:last_changed 0.000000 2024-03-25T18:35:58.625 INFO:teuthology.misc.health.smithi001.stderr:created 0.000000 2024-03-25T18:35:58.625 INFO:teuthology.misc.health.smithi001.stderr:min_mon_release 0 (unknown) 2024-03-25T18:35:58.626 INFO:teuthology.misc.health.smithi001.stderr:election_strategy: 1 2024-03-25T18:35:58.626 INFO:teuthology.misc.health.smithi001.stderr:0: v1:172.21.15.1:6789/0 mon.noname-a 2024-03-25T18:35:58.626 INFO:teuthology.misc.health.smithi001.stderr:1: v1:172.21.15.115:6789/0 mon.noname-b 2024-03-25T18:35:58.626 INFO:teuthology.misc.health.smithi001.stderr:2: v1:172.21.15.1:6790/0 mon.noname-c 2024-03-25T18:35:58.626 INFO:teuthology.misc.health.smithi001.stderr: 2024-03-25T18:35:58.626 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:35:58.623+0000 7f423a152640 1 librados: starting msgr at 2024-03-25T18:35:58.626 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:35:58.623+0000 7f423a152640 1 librados: starting objecter 2024-03-25T18:35:58.626 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:35:58.623+0000 7f423a152640 1 Processor -- start 2024-03-25T18:35:58.626 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:35:58.623+0000 7f423a152640 1 -- start start 2024-03-25T18:35:58.626 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:35:58.623+0000 7f423a152640 1 librados: setting wanted keys 2024-03-25T18:35:58.626 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:35:58.623+0000 7f423a152640 1 librados: calling monclient init 2024-03-25T18:35:58.626 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:35:58.623+0000 7f423a152640 10 monclient: init 2024-03-25T18:35:58.627 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:35:58.624+0000 7f423a152640 10 monclient: _reopen_session rank -1 2024-03-25T18:35:58.627 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:35:58.624+0000 7f423a152640 10 monclient: _add_conns ranks=[0,1,2] 2024-03-25T18:35:58.627 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:35:58.624+0000 7f423a152640 10 monclient(hunting): picked mon.noname-a con 0x7f42340b1b00 addr v1:172.21.15.1:6789/0 2024-03-25T18:35:58.627 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:35:58.624+0000 7f423a152640 10 monclient(hunting): picked mon.noname-b con 0x7f42340b44b0 addr v1:172.21.15.115:6789/0 2024-03-25T18:35:58.627 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:35:58.624+0000 7f423a152640 10 monclient(hunting): picked mon.noname-c con 0x7f42340b4e70 addr v1:172.21.15.1:6790/0 2024-03-25T18:35:58.627 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:35:58.624+0000 7f423a152640 1 -- --> v1:172.21.15.1:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4234137b10 con 0x7f42340b1b00 2024-03-25T18:35:58.628 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:35:58.624+0000 7f423a152640 1 -- --> v1:172.21.15.115:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4234135ef0 con 0x7f42340b44b0 2024-03-25T18:35:58.628 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:35:58.624+0000 7f423a152640 1 -- --> v1:172.21.15.1:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f42341432b0 con 0x7f42340b4e70 2024-03-25T18:35:58.628 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:35:58.624+0000 7f423a152640 10 monclient(hunting): _renew_subs 2024-03-25T18:35:58.628 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:35:58.624+0000 7f423a152640 10 monclient(hunting): authenticate will time out at 8039.095409s 2024-03-25T18:35:58.629 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:35:58.628+0000 7f4233fff640 1 --1- >> v1:172.21.15.1:6790/0 conn(0x7f42340b4e70 0x7f42341e2e20 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.1:6790/0 says I am v1:172.21.15.1:40538/0 (socket says 172.21.15.1:40538) 2024-03-25T18:35:58.630 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:35:58.628+0000 7f4233fff640 1 -- 172.21.15.1:0/1051305529 learned_addr learned my addr 172.21.15.1:0/1051305529 (peer_addr_for_me v1:172.21.15.1:0/0) 2024-03-25T18:35:58.632 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:35:58.630+0000 7f421bfff640 10 client.?.objecter ms_handle_connect 0x7f42340b4e70 2024-03-25T18:35:58.632 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:35:58.630+0000 7f421bfff640 10 client.?.objecter resend_mon_ops 2024-03-25T18:35:58.635 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:35:58.633+0000 7f421bfff640 10 client.?.objecter ms_handle_connect 0x7f42340b44b0 2024-03-25T18:35:58.635 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:35:58.633+0000 7f421bfff640 10 client.?.objecter resend_mon_ops 2024-03-25T18:35:58.637 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:35:58.636+0000 7f421bfff640 1 -- 172.21.15.1:0/1051305529 <== mon.2 v1:172.21.15.1:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1435229406 0 0) 0x7f42341432b0 con 0x7f42340b4e70 2024-03-25T18:35:58.638 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:35:58.636+0000 7f421bfff640 10 monclient(hunting): _negotiate 2024-03-25T18:35:58.638 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:35:58.636+0000 7f421bfff640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:35:58.638 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:35:58.636+0000 7f421bfff640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:35:58.638 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:35:58.636+0000 7f421bfff640 10 monclient(hunting): my global_id is 4628 2024-03-25T18:35:58.638 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:35:58.636+0000 7f421bfff640 1 -- 172.21.15.1:0/1051305529 --> v1:172.21.15.1:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f4220004eb0 con 0x7f42340b4e70 2024-03-25T18:35:58.654 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:35:58.653+0000 7f421bfff640 1 -- 172.21.15.1:0/1051305529 <== mon.1 v1:172.21.15.115:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1206001542 0 0) 0x7f4234135ef0 con 0x7f42340b44b0 2024-03-25T18:35:58.654 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:35:58.653+0000 7f421bfff640 10 monclient(hunting): _negotiate 2024-03-25T18:35:58.655 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:35:58.653+0000 7f421bfff640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:35:58.655 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:35:58.653+0000 7f421bfff640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:35:58.655 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:35:58.653+0000 7f421bfff640 10 monclient(hunting): my global_id is 4627 2024-03-25T18:35:58.655 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:35:58.653+0000 7f421bfff640 1 -- 172.21.15.1:0/1051305529 --> v1:172.21.15.115:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f42341432b0 con 0x7f42340b44b0 2024-03-25T18:35:58.656 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:35:58.655+0000 7f421bfff640 10 client.?.objecter ms_handle_connect 0x7f42340b1b00 2024-03-25T18:35:58.657 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:35:58.655+0000 7f421bfff640 10 client.?.objecter resend_mon_ops 2024-03-25T18:35:58.663 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:35:58.661+0000 7f421bfff640 1 -- 172.21.15.1:0/1051305529 <== mon.0 v1:172.21.15.1:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2361387113 0 0) 0x7f4234137b10 con 0x7f42340b1b00 2024-03-25T18:35:58.663 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:35:58.661+0000 7f421bfff640 10 monclient(hunting): _negotiate 2024-03-25T18:35:58.663 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:35:58.661+0000 7f421bfff640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:35:58.663 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:35:58.661+0000 7f421bfff640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:35:58.663 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:35:58.661+0000 7f421bfff640 10 monclient(hunting): my global_id is 4620 2024-03-25T18:35:58.663 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:35:58.661+0000 7f421bfff640 1 -- 172.21.15.1:0/1051305529 --> v1:172.21.15.1:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f4234135ef0 con 0x7f42340b1b00 2024-03-25T18:35:58.670 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:35:58.666+0000 7f421bfff640 1 -- 172.21.15.1:0/1051305529 <== mon.1 v1:172.21.15.115:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 382509578 0 0) 0x7f42341432b0 con 0x7f42340b44b0 2024-03-25T18:35:58.670 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:35:58.666+0000 7f421bfff640 1 -- 172.21.15.1:0/1051305529 --> v1:172.21.15.115:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f4234137b10 con 0x7f42340b44b0 2024-03-25T18:35:58.670 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:35:58.666+0000 7f421bfff640 1 -- 172.21.15.1:0/1051305529 <== mon.1 v1:172.21.15.115:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1586965025 0 0) 0x7f42280018b0 con 0x7f42340b44b0 2024-03-25T18:35:58.670 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:35:58.666+0000 7f421bfff640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:35:58.670 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:35:58.666+0000 7f421bfff640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-03-25T18:35:58.670 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:35:58.666+0000 7f421bfff640 10 monclient(hunting): dump: 2024-03-25T18:35:58.671 INFO:teuthology.misc.health.smithi001.stderr:epoch 1 2024-03-25T18:35:58.671 INFO:teuthology.misc.health.smithi001.stderr:fsid c8ea72cf-9e99-4383-9b52-6c23d365fdfc 2024-03-25T18:35:58.671 INFO:teuthology.misc.health.smithi001.stderr:last_changed 2024-03-25T18:25:50.510519+0000 2024-03-25T18:35:58.671 INFO:teuthology.misc.health.smithi001.stderr:created 2024-03-25T18:25:50.510519+0000 2024-03-25T18:35:58.671 INFO:teuthology.misc.health.smithi001.stderr:min_mon_release 19 (squid) 2024-03-25T18:35:58.671 INFO:teuthology.misc.health.smithi001.stderr:election_strategy: 3 2024-03-25T18:35:58.671 INFO:teuthology.misc.health.smithi001.stderr:0: v1:172.21.15.1:6789/0 mon.a 2024-03-25T18:35:58.671 INFO:teuthology.misc.health.smithi001.stderr:1: v1:172.21.15.115:6789/0 mon.b 2024-03-25T18:35:58.671 INFO:teuthology.misc.health.smithi001.stderr:2: v1:172.21.15.1:6790/0 mon.c 2024-03-25T18:35:58.671 INFO:teuthology.misc.health.smithi001.stderr: 2024-03-25T18:35:58.671 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:35:58.666+0000 7f421bfff640 10 monclient(hunting): _finish_auth 0 2024-03-25T18:35:58.672 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:35:58.669+0000 7f421bfff640 1 -- 172.21.15.1:0/1051305529 <== mon.2 v1:172.21.15.1:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2483828622 0 0) 0x7f4220004eb0 con 0x7f42340b4e70 2024-03-25T18:35:58.672 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:35:58.669+0000 7f421bfff640 1 -- 172.21.15.1:0/1051305529 --> v1:172.21.15.1:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f42341432b0 con 0x7f42340b4e70 2024-03-25T18:35:58.672 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:35:58.669+0000 7f421bfff640 1 -- 172.21.15.1:0/1051305529 <== mon.2 v1:172.21.15.1:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1586965025 0 0) 0x7f422c001d30 con 0x7f42340b4e70 2024-03-25T18:35:58.672 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:35:58.669+0000 7f421bfff640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:35:58.672 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:35:58.669+0000 7f421bfff640 10 monclient(hunting): got monmap 1 from mon.c (according to old e1) 2024-03-25T18:35:58.672 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:35:58.669+0000 7f421bfff640 10 monclient(hunting): dump: 2024-03-25T18:35:58.672 INFO:teuthology.misc.health.smithi001.stderr:epoch 1 2024-03-25T18:35:58.672 INFO:teuthology.misc.health.smithi001.stderr:fsid c8ea72cf-9e99-4383-9b52-6c23d365fdfc 2024-03-25T18:35:58.672 INFO:teuthology.misc.health.smithi001.stderr:last_changed 2024-03-25T18:25:50.510519+0000 2024-03-25T18:35:58.672 INFO:teuthology.misc.health.smithi001.stderr:created 2024-03-25T18:25:50.510519+0000 2024-03-25T18:35:58.672 INFO:teuthology.misc.health.smithi001.stderr:min_mon_release 19 (squid) 2024-03-25T18:35:58.672 INFO:teuthology.misc.health.smithi001.stderr:election_strategy: 3 2024-03-25T18:35:58.672 INFO:teuthology.misc.health.smithi001.stderr:0: v1:172.21.15.1:6789/0 mon.a 2024-03-25T18:35:58.672 INFO:teuthology.misc.health.smithi001.stderr:1: v1:172.21.15.115:6789/0 mon.b 2024-03-25T18:35:58.673 INFO:teuthology.misc.health.smithi001.stderr:2: v1:172.21.15.1:6790/0 mon.c 2024-03-25T18:35:58.673 INFO:teuthology.misc.health.smithi001.stderr: 2024-03-25T18:35:58.675 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:35:58.673+0000 7f421bfff640 1 -- 172.21.15.1:0/1051305529 <== mon.1 v1:172.21.15.115:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 1731333743 0 0) 0x7f4234137b10 con 0x7f42340b44b0 2024-03-25T18:35:58.675 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:35:58.673+0000 7f421bfff640 1 -- 172.21.15.1:0/1051305529 >> v1:172.21.15.1:6790/0 conn(0x7f42340b4e70 legacy=0x7f42341e2e20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:35:58.675 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:35:58.673+0000 7f421bfff640 1 -- 172.21.15.1:0/1051305529 >> v1:172.21.15.1:6789/0 conn(0x7f42340b1b00 legacy=0x7f4234133ff0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:35:58.675 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:35:58.673+0000 7f421bfff640 10 monclient: _finish_hunting 0 2024-03-25T18:35:58.675 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:35:58.673+0000 7f421bfff640 1 monclient: found mon.b 2024-03-25T18:35:58.676 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:35:58.673+0000 7f421bfff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:35:58.676 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:35:58.673+0000 7f421bfff640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.115:6789/0 2024-03-25T18:35:58.676 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:35:58.673+0000 7f421bfff640 1 -- 172.21.15.1:0/1051305529 --> v1:172.21.15.115:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4234143440 con 0x7f42340b44b0 2024-03-25T18:35:58.676 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:35:58.673+0000 7f421bfff640 10 monclient: _finish_auth 0 2024-03-25T18:35:58.676 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:35:58.673+0000 7f421bfff640 10 monclient: _check_auth_tickets 2024-03-25T18:35:58.676 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:35:58.673+0000 7f421bfff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:35:58.676 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:35:58.673+0000 7f423a152640 5 monclient: authenticate success, global_id 4627 2024-03-25T18:35:58.676 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:35:58.673+0000 7f423a152640 10 monclient: _renew_subs 2024-03-25T18:35:58.676 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:35:58.673+0000 7f423a152640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.115:6789/0 2024-03-25T18:35:58.676 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:35:58.673+0000 7f423a152640 1 -- 172.21.15.1:0/1051305529 --> v1:172.21.15.115:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f42341db770 con 0x7f42340b44b0 2024-03-25T18:35:58.676 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:35:58.673+0000 7f423a152640 10 client.4627.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T18:35:58.676 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:35:58.673+0000 7f423a152640 10 monclient: _renew_subs 2024-03-25T18:35:58.676 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:35:58.673+0000 7f423a152640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.115:6789/0 2024-03-25T18:35:58.676 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:35:58.673+0000 7f423a152640 1 -- 172.21.15.1:0/1051305529 --> v1:172.21.15.115:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f42341348e0 con 0x7f42340b44b0 2024-03-25T18:35:58.676 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:35:58.673+0000 7f423a152640 1 librados: init done 2024-03-25T18:35:58.677 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:35:58.674+0000 7f423a152640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T18:35:58.677 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:35:58.674+0000 7f423a152640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:35:58.677 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:35:58.674+0000 7f423a152640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.115:6789/0 2024-03-25T18:35:58.677 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:35:58.674+0000 7f423a152640 1 -- 172.21.15.1:0/1051305529 --> v1:172.21.15.115:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f423404f730 con 0x7f42340b44b0 2024-03-25T18:35:58.945 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:35:58.943+0000 7f421bfff640 1 -- 172.21.15.1:0/1051305529 <== mon.1 v1:172.21.15.115:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f4228003610 con 0x7f42340b44b0 2024-03-25T18:35:58.945 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:35:58.943+0000 7f421bfff640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:35:58.945 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:35:58.943+0000 7f421bfff640 1 -- 172.21.15.1:0/1051305529 <== mon.1 v1:172.21.15.115:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1586965025 0 0) 0x7f42280039a0 con 0x7f42340b44b0 2024-03-25T18:35:58.945 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:35:58.943+0000 7f421bfff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:35:58.945 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:35:58.943+0000 7f421bfff640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-25T18:35:58.945 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:35:58.943+0000 7f421bfff640 10 monclient: dump: 2024-03-25T18:35:58.945 INFO:teuthology.misc.health.smithi001.stderr:epoch 1 2024-03-25T18:35:58.945 INFO:teuthology.misc.health.smithi001.stderr:fsid c8ea72cf-9e99-4383-9b52-6c23d365fdfc 2024-03-25T18:35:58.945 INFO:teuthology.misc.health.smithi001.stderr:last_changed 2024-03-25T18:25:50.510519+0000 2024-03-25T18:35:58.945 INFO:teuthology.misc.health.smithi001.stderr:created 2024-03-25T18:25:50.510519+0000 2024-03-25T18:35:58.946 INFO:teuthology.misc.health.smithi001.stderr:min_mon_release 19 (squid) 2024-03-25T18:35:58.946 INFO:teuthology.misc.health.smithi001.stderr:election_strategy: 3 2024-03-25T18:35:58.946 INFO:teuthology.misc.health.smithi001.stderr:0: v1:172.21.15.1:6789/0 mon.a 2024-03-25T18:35:58.946 INFO:teuthology.misc.health.smithi001.stderr:1: v1:172.21.15.115:6789/0 mon.b 2024-03-25T18:35:58.946 INFO:teuthology.misc.health.smithi001.stderr:2: v1:172.21.15.1:6790/0 mon.c 2024-03-25T18:35:58.946 INFO:teuthology.misc.health.smithi001.stderr: 2024-03-25T18:35:58.946 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:35:58.943+0000 7f4231ffb640 4 set_mon_vals no callback set 2024-03-25T18:35:58.946 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:35:58.944+0000 7f421bfff640 1 -- 172.21.15.1:0/1051305529 <== mon.1 v1:172.21.15.115:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 21787975 0 0) 0x7f4228003bc0 con 0x7f42340b44b0 2024-03-25T18:35:58.947 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:35:58.945+0000 7f42337fe640 10 monclient: get_auth_request con 0x7f422006df70 auth_method 0 2024-03-25T18:35:58.947 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:35:58.945+0000 7f421bfff640 10 client.4627.objecter ms_handle_connect 0x7f422006df70 2024-03-25T18:35:58.950 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:35:58.948+0000 7f421bfff640 1 -- 172.21.15.1:0/1051305529 <== mon.1 v1:172.21.15.115:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 996254230 0 0) 0x7f4228053530 con 0x7f42340b44b0 2024-03-25T18:35:58.950 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:35:58.948+0000 7f421bfff640 10 client.4627.objecter ms_dispatch 0x7f4234000b90 osd_map(10..10 src has 1..10) v4 2024-03-25T18:35:58.950 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:35:58.948+0000 7f421bfff640 3 client.4627.objecter handle_osd_map got epochs [10,10] > 0 2024-03-25T18:35:58.950 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:35:58.948+0000 7f421bfff640 3 client.4627.objecter handle_osd_map decoding full epoch 10 2024-03-25T18:35:58.950 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:35:58.948+0000 7f421bfff640 20 client.4627.objecter dump_active .. 0 homeless 2024-03-25T18:35:58.950 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:35:58.949+0000 7f421bfff640 1 -- 172.21.15.1:0/1051305529 <== mon.1 v1:172.21.15.115:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f422804f040 con 0x7f42340b44b0 2024-03-25T18:35:58.951 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:35:58.949+0000 7f421bfff640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:35:58.951 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:35:58.949+0000 7f421bfff640 10 monclient: _finish_command 1 = system:0 2024-03-25T18:35:59.171 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:35:59.170+0000 7f423a152640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:35:59.171 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:35:59.170+0000 7f423a152640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:35:59.171 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:35:59.170+0000 7f423a152640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.115:6789/0 2024-03-25T18:35:59.171 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:35:59.170+0000 7f423a152640 1 -- 172.21.15.1:0/1051305529 --> v1:172.21.15.115:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f42341360a0 con 0x7f42340b44b0 2024-03-25T18:35:59.221 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:35:59.220+0000 7f421bfff640 1 -- 172.21.15.1:0/1051305529 <== mon.1 v1:172.21.15.115:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 4092213348) 0x7f42341360a0 con 0x7f42340b44b0 2024-03-25T18:35:59.222 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:35:59.220+0000 7f421bfff640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:35:59.222 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:35:59.220+0000 7f421bfff640 10 monclient: _finish_command 2 = system:0 2024-03-25T18:35:59.222 INFO:teuthology.misc.health.smithi001.stdout: 2024-03-25T18:35:59.222 INFO:teuthology.misc.health.smithi001.stdout:{"epoch":10,"fsid":"c8ea72cf-9e99-4383-9b52-6c23d365fdfc","created":"2024-03-25T18:26:38.856638+0000","modified":"2024-03-25T18:26:44.224208+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T18:26:44.224208+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"b9f34922-3a84-49be-8426-807c1eceaefc","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"42bf306f-55e3-4634-9874-8b4b73ac6644","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"292cb23a-18a9-4207-a7cc-4d5727feaec0","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"45e0e9ab-898d-4673-9590-b4117bc5e84a","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"831a5a09-73ab-4a61-9872-088e76bdbc66","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"88cd903d-3b55-4840-9396-6e3cfe005282","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"89db7355-bc9d-46f4-8ab0-e8d9e744e026","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"cac47bee-0914-4f08-896f-348838882866","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T18:35:59.224 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:35:59.222+0000 7f423a152640 10 librados: watch_flush enter 2024-03-25T18:35:59.225 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:35:59.222+0000 7f423a152640 10 librados: watch_flush exit 2024-03-25T18:35:59.225 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:35:59.222+0000 7f423a152640 20 client.4627.objecter shutdown clearing up homeless session... 2024-03-25T18:35:59.225 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:35:59.222+0000 7f423a152640 10 client.4627.objecter successfully canceled tick 2024-03-25T18:35:59.225 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:35:59.222+0000 7f423a152640 1 -- 172.21.15.1:0/1051305529 >> v1:172.21.15.115:6800/4237379176 conn(0x7f422006df70 legacy=0x7f4220070430 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:35:59.225 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:35:59.222+0000 7f423a152640 10 monclient: shutdown 2024-03-25T18:35:59.225 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:35:59.222+0000 7f423a152640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:35:59.225 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:35:59.222+0000 7f423a152640 1 -- 172.21.15.1:0/1051305529 >> v1:172.21.15.115:6789/0 conn(0x7f42340b44b0 legacy=0x7f4234136380 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:35:59.225 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:35:59.222+0000 7f423a152640 1 -- 172.21.15.1:0/1051305529 shutdown_connections 2024-03-25T18:35:59.225 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:35:59.222+0000 7f423a152640 1 -- 172.21.15.1:0/1051305529 >> 172.21.15.1:0/1051305529 conn(0x7f42340617a0 msgr2=0x7f42340b2710 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:35:59.225 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:35:59.223+0000 7f423a152640 1 -- 172.21.15.1:0/1051305529 shutdown_connections 2024-03-25T18:35:59.225 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:35:59.223+0000 7f423a152640 1 -- 172.21.15.1:0/1051305529 wait complete. 2024-03-25T18:35:59.225 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:35:59.223+0000 7f423a152640 1 librados: shutdown 2024-03-25T18:35:59.236 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T18:36:02.995 INFO:tasks.ceph.mon.a.smithi001.stderr:==00:00:09:41.554 34876== Warning: unimplemented fcntl command: 1036 2024-03-25T18:36:03.007 INFO:tasks.ceph.mon.a.smithi001.stderr:==00:00:09:41.566 34876== Warning: unimplemented fcntl command: 1036 2024-03-25T18:36:03.035 INFO:tasks.ceph.mon.a.smithi001.stderr:==00:00:09:41.593 34876== Warning: unimplemented fcntl command: 1036 2024-03-25T18:36:03.058 INFO:tasks.ceph.mon.b.smithi115.stderr:==00:00:09:41.542 34836== Warning: unimplemented fcntl command: 1036 2024-03-25T18:36:03.059 INFO:tasks.ceph.mon.c.smithi001.stderr:==00:00:09:41.617 34877== Warning: unimplemented fcntl command: 1036 2024-03-25T18:36:03.061 INFO:tasks.ceph.mon.b.smithi115.stderr:==00:00:09:41.545 34836== Warning: unimplemented fcntl command: 1036 2024-03-25T18:36:03.070 INFO:tasks.ceph.mon.c.smithi001.stderr:==00:00:09:41.629 34877== Warning: unimplemented fcntl command: 1036 2024-03-25T18:36:03.087 INFO:tasks.ceph.mon.b.smithi115.stderr:==00:00:09:41.571 34836== Warning: unimplemented fcntl command: 1036 2024-03-25T18:36:03.099 INFO:tasks.ceph.mon.c.smithi001.stderr:==00:00:09:41.657 34877== Warning: unimplemented fcntl command: 1036 2024-03-25T18:36:06.042 DEBUG:teuthology.orchestra.run.smithi001:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T18:36:06.160 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:36:06.160+0000 7f59d020f640 10 monclient: get_monmap_and_config 2024-03-25T18:36:06.160 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:36:06.160+0000 7f59d020f640 10 monclient: build_initial_monmap 2024-03-25T18:36:06.160 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:36:06.160+0000 7f59d020f640 10 monclient: monmap: 2024-03-25T18:36:06.161 INFO:teuthology.misc.health.smithi001.stderr:epoch 0 2024-03-25T18:36:06.161 INFO:teuthology.misc.health.smithi001.stderr:fsid c8ea72cf-9e99-4383-9b52-6c23d365fdfc 2024-03-25T18:36:06.161 INFO:teuthology.misc.health.smithi001.stderr:last_changed 2024-03-25T18:36:06.160662+0000 2024-03-25T18:36:06.161 INFO:teuthology.misc.health.smithi001.stderr:created 2024-03-25T18:36:06.160662+0000 2024-03-25T18:36:06.161 INFO:teuthology.misc.health.smithi001.stderr:min_mon_release 0 (unknown) 2024-03-25T18:36:06.161 INFO:teuthology.misc.health.smithi001.stderr:election_strategy: 3 2024-03-25T18:36:06.161 INFO:teuthology.misc.health.smithi001.stderr:0: v1:172.21.15.1:6789/0 mon.noname-a 2024-03-25T18:36:06.161 INFO:teuthology.misc.health.smithi001.stderr:1: v1:172.21.15.115:6789/0 mon.noname-c 2024-03-25T18:36:06.161 INFO:teuthology.misc.health.smithi001.stderr:2: v1:172.21.15.1:6790/0 mon.noname-b 2024-03-25T18:36:06.161 INFO:teuthology.misc.health.smithi001.stderr: 2024-03-25T18:36:06.162 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:36:06.160+0000 7f59d020f640 1 Processor -- start 2024-03-25T18:36:06.162 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:36:06.161+0000 7f59d020f640 1 -- start start 2024-03-25T18:36:06.162 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:36:06.161+0000 7f59d020f640 10 monclient: init 2024-03-25T18:36:06.162 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:36:06.161+0000 7f59d020f640 10 monclient: _reopen_session rank -1 2024-03-25T18:36:06.163 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:36:06.161+0000 7f59d020f640 10 monclient: _add_conns ranks=[2,0,1] 2024-03-25T18:36:06.163 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:36:06.161+0000 7f59d020f640 10 monclient(hunting): picked mon.noname-b con 0x7f59c80b4ff0 addr v1:172.21.15.1:6790/0 2024-03-25T18:36:06.163 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:36:06.161+0000 7f59d020f640 10 monclient(hunting): picked mon.noname-a con 0x7f59c80b4670 addr v1:172.21.15.1:6789/0 2024-03-25T18:36:06.163 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:36:06.161+0000 7f59d020f640 10 monclient(hunting): picked mon.noname-c con 0x7f59c813c480 addr v1:172.21.15.115:6789/0 2024-03-25T18:36:06.163 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:36:06.161+0000 7f59d020f640 1 -- --> v1:172.21.15.1:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f59c80ab0e0 con 0x7f59c80b4670 2024-03-25T18:36:06.163 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:36:06.161+0000 7f59d020f640 1 -- --> v1:172.21.15.115:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f59c80ab3c0 con 0x7f59c813c480 2024-03-25T18:36:06.163 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:36:06.162+0000 7f59d020f640 1 -- --> v1:172.21.15.1:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f59c80a9f30 con 0x7f59c80b4ff0 2024-03-25T18:36:06.163 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:36:06.162+0000 7f59d020f640 10 monclient(hunting): _renew_subs 2024-03-25T18:36:06.163 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:36:06.162+0000 7f59d020f640 10 monclient(hunting): authenticate will time out at 8046.631668s 2024-03-25T18:36:06.173 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:36:06.172+0000 7f59cdfab640 1 --1- >> v1:172.21.15.1:6790/0 conn(0x7f59c80b4ff0 0x7f59c813bd70 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.1:6790/0 says I am v1:172.21.15.1:41136/0 (socket says 172.21.15.1:41136) 2024-03-25T18:36:06.173 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:36:06.172+0000 7f59cdfab640 1 -- 172.21.15.1:0/3003344413 learned_addr learned my addr 172.21.15.1:0/3003344413 (peer_addr_for_me v1:172.21.15.1:0/0) 2024-03-25T18:36:06.187 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:36:06.187+0000 7f59ccfa9640 1 -- 172.21.15.1:0/3003344413 <== mon.2 v1:172.21.15.1:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 80587920 0 0) 0x7f59c80a9f30 con 0x7f59c80b4ff0 2024-03-25T18:36:06.187 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:36:06.187+0000 7f59ccfa9640 10 monclient(hunting): _negotiate 2024-03-25T18:36:06.187 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:36:06.187+0000 7f59ccfa9640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:36:06.187 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:36:06.187+0000 7f59ccfa9640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:36:06.188 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:36:06.187+0000 7f59ccfa9640 10 monclient(hunting): my global_id is 4631 2024-03-25T18:36:06.188 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:36:06.187+0000 7f59ccfa9640 1 -- 172.21.15.1:0/3003344413 --> v1:172.21.15.1:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f59b4003520 con 0x7f59c80b4ff0 2024-03-25T18:36:06.193 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:36:06.193+0000 7f59ccfa9640 1 -- 172.21.15.1:0/3003344413 <== mon.2 v1:172.21.15.1:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 4002640658 0 0) 0x7f59b4003520 con 0x7f59c80b4ff0 2024-03-25T18:36:06.193 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:36:06.193+0000 7f59ccfa9640 1 -- 172.21.15.1:0/3003344413 >> v1:172.21.15.115:6789/0 conn(0x7f59c813c480 legacy=0x7f59c814dc00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:36:06.194 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:36:06.193+0000 7f59ccfa9640 1 -- 172.21.15.1:0/3003344413 >> v1:172.21.15.1:6789/0 conn(0x7f59c80b4670 legacy=0x7f59c81425c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:36:06.194 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:36:06.193+0000 7f59ccfa9640 10 monclient: _finish_hunting 0 2024-03-25T18:36:06.194 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:36:06.193+0000 7f59ccfa9640 1 monclient: found mon.noname-b 2024-03-25T18:36:06.194 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:36:06.193+0000 7f59ccfa9640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:36:06.194 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:36:06.193+0000 7f59ccfa9640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.1:6790/0 2024-03-25T18:36:06.194 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:36:06.193+0000 7f59ccfa9640 1 -- 172.21.15.1:0/3003344413 --> v1:172.21.15.1:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f59c80a9b70 con 0x7f59c80b4ff0 2024-03-25T18:36:06.194 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:36:06.193+0000 7f59ccfa9640 10 monclient: _finish_auth 0 2024-03-25T18:36:06.194 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:36:06.193+0000 7f59ccfa9640 10 monclient: _check_auth_tickets 2024-03-25T18:36:06.194 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:36:06.193+0000 7f59ccfa9640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:36:06.194 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:36:06.193+0000 7f59ccfa9640 1 -- 172.21.15.1:0/3003344413 <== mon.2 v1:172.21.15.1:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1586965025 0 0) 0x7f59c4001690 con 0x7f59c80b4ff0 2024-03-25T18:36:06.194 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:36:06.193+0000 7f59ccfa9640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:36:06.194 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:36:06.193+0000 7f59ccfa9640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-03-25T18:36:06.194 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:36:06.193+0000 7f59ccfa9640 10 monclient: dump: 2024-03-25T18:36:06.194 INFO:teuthology.misc.health.smithi001.stderr:epoch 1 2024-03-25T18:36:06.195 INFO:teuthology.misc.health.smithi001.stderr:fsid c8ea72cf-9e99-4383-9b52-6c23d365fdfc 2024-03-25T18:36:06.195 INFO:teuthology.misc.health.smithi001.stderr:last_changed 2024-03-25T18:25:50.510519+0000 2024-03-25T18:36:06.195 INFO:teuthology.misc.health.smithi001.stderr:created 2024-03-25T18:25:50.510519+0000 2024-03-25T18:36:06.195 INFO:teuthology.misc.health.smithi001.stderr:min_mon_release 19 (squid) 2024-03-25T18:36:06.195 INFO:teuthology.misc.health.smithi001.stderr:election_strategy: 3 2024-03-25T18:36:06.195 INFO:teuthology.misc.health.smithi001.stderr:0: v1:172.21.15.1:6789/0 mon.a 2024-03-25T18:36:06.195 INFO:teuthology.misc.health.smithi001.stderr:1: v1:172.21.15.115:6789/0 mon.b 2024-03-25T18:36:06.195 INFO:teuthology.misc.health.smithi001.stderr:2: v1:172.21.15.1:6790/0 mon.c 2024-03-25T18:36:06.195 INFO:teuthology.misc.health.smithi001.stderr: 2024-03-25T18:36:06.195 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:36:06.193+0000 7f59d020f640 5 monclient: authenticate success, global_id 4631 2024-03-25T18:36:06.195 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:36:06.193+0000 7f59d020f640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T18:36:06.205 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:36:06.204+0000 7f59ccfa9640 1 -- 172.21.15.1:0/3003344413 <== mon.2 v1:172.21.15.1:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f59c40017f0 con 0x7f59c80b4ff0 2024-03-25T18:36:06.205 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:36:06.204+0000 7f59ccfa9640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:36:06.205 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:36:06.204+0000 7f59ccfa9640 1 -- 172.21.15.1:0/3003344413 <== mon.2 v1:172.21.15.1:6790/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1586965025 0 0) 0x7f59c4001c00 con 0x7f59c80b4ff0 2024-03-25T18:36:06.205 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:36:06.204+0000 7f59ccfa9640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:36:06.205 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:36:06.204+0000 7f59ccfa9640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-25T18:36:06.205 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:36:06.204+0000 7f59ccfa9640 10 monclient: dump: 2024-03-25T18:36:06.205 INFO:teuthology.misc.health.smithi001.stderr:epoch 1 2024-03-25T18:36:06.205 INFO:teuthology.misc.health.smithi001.stderr:fsid c8ea72cf-9e99-4383-9b52-6c23d365fdfc 2024-03-25T18:36:06.205 INFO:teuthology.misc.health.smithi001.stderr:last_changed 2024-03-25T18:25:50.510519+0000 2024-03-25T18:36:06.206 INFO:teuthology.misc.health.smithi001.stderr:created 2024-03-25T18:25:50.510519+0000 2024-03-25T18:36:06.206 INFO:teuthology.misc.health.smithi001.stderr:min_mon_release 19 (squid) 2024-03-25T18:36:06.206 INFO:teuthology.misc.health.smithi001.stderr:election_strategy: 3 2024-03-25T18:36:06.206 INFO:teuthology.misc.health.smithi001.stderr:0: v1:172.21.15.1:6789/0 mon.a 2024-03-25T18:36:06.206 INFO:teuthology.misc.health.smithi001.stderr:1: v1:172.21.15.115:6789/0 mon.b 2024-03-25T18:36:06.206 INFO:teuthology.misc.health.smithi001.stderr:2: v1:172.21.15.1:6790/0 mon.c 2024-03-25T18:36:06.206 INFO:teuthology.misc.health.smithi001.stderr: 2024-03-25T18:36:06.206 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:36:06.204+0000 7f59d020f640 10 monclient: get_monmap_and_config success 2024-03-25T18:36:06.206 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:36:06.204+0000 7f59d020f640 4 set_mon_vals no callback set 2024-03-25T18:36:06.208 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:36:06.205+0000 7f59d020f640 10 monclient: shutdown 2024-03-25T18:36:06.208 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:36:06.205+0000 7f59d020f640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:36:06.208 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:36:06.205+0000 7f59d020f640 1 -- 172.21.15.1:0/3003344413 >> v1:172.21.15.1:6790/0 conn(0x7f59c80b4ff0 legacy=0x7f59c813bd70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:36:06.208 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:36:06.205+0000 7f59d020f640 1 -- 172.21.15.1:0/3003344413 shutdown_connections 2024-03-25T18:36:06.208 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:36:06.205+0000 7f59d020f640 1 -- 172.21.15.1:0/3003344413 >> 172.21.15.1:0/3003344413 conn(0x7f59c80617a0 msgr2=0x7f59c8061bb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:36:06.208 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:36:06.205+0000 7f59d020f640 1 -- 172.21.15.1:0/3003344413 shutdown_connections 2024-03-25T18:36:06.208 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:36:06.205+0000 7f59d020f640 1 -- 172.21.15.1:0/3003344413 wait complete. 2024-03-25T18:36:06.208 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:36:06.206+0000 7f59d020f640 10 monclient: build_initial_monmap 2024-03-25T18:36:06.208 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:36:06.206+0000 7f59d020f640 10 monclient: monmap: 2024-03-25T18:36:06.209 INFO:teuthology.misc.health.smithi001.stderr:epoch 0 2024-03-25T18:36:06.209 INFO:teuthology.misc.health.smithi001.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T18:36:06.209 INFO:teuthology.misc.health.smithi001.stderr:last_changed 0.000000 2024-03-25T18:36:06.209 INFO:teuthology.misc.health.smithi001.stderr:created 0.000000 2024-03-25T18:36:06.209 INFO:teuthology.misc.health.smithi001.stderr:min_mon_release 0 (unknown) 2024-03-25T18:36:06.209 INFO:teuthology.misc.health.smithi001.stderr:election_strategy: 1 2024-03-25T18:36:06.209 INFO:teuthology.misc.health.smithi001.stderr:0: v1:172.21.15.1:6789/0 mon.noname-a 2024-03-25T18:36:06.209 INFO:teuthology.misc.health.smithi001.stderr:1: v1:172.21.15.115:6789/0 mon.noname-b 2024-03-25T18:36:06.209 INFO:teuthology.misc.health.smithi001.stderr:2: v1:172.21.15.1:6790/0 mon.noname-c 2024-03-25T18:36:06.209 INFO:teuthology.misc.health.smithi001.stderr: 2024-03-25T18:36:06.209 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:36:06.206+0000 7f59d020f640 1 librados: starting msgr at 2024-03-25T18:36:06.209 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:36:06.206+0000 7f59d020f640 1 librados: starting objecter 2024-03-25T18:36:06.209 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:36:06.206+0000 7f59d020f640 1 Processor -- start 2024-03-25T18:36:06.209 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:36:06.206+0000 7f59d020f640 1 -- start start 2024-03-25T18:36:06.209 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:36:06.206+0000 7f59d020f640 1 librados: setting wanted keys 2024-03-25T18:36:06.209 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:36:06.206+0000 7f59d020f640 1 librados: calling monclient init 2024-03-25T18:36:06.210 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:36:06.206+0000 7f59d020f640 10 monclient: init 2024-03-25T18:36:06.210 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:36:06.207+0000 7f59d020f640 10 monclient: _reopen_session rank -1 2024-03-25T18:36:06.210 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:36:06.207+0000 7f59d020f640 10 monclient: _add_conns ranks=[0,1,2] 2024-03-25T18:36:06.210 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:36:06.207+0000 7f59d020f640 10 monclient(hunting): picked mon.noname-a con 0x7f59c80b4670 addr v1:172.21.15.1:6789/0 2024-03-25T18:36:06.210 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:36:06.207+0000 7f59d020f640 10 monclient(hunting): picked mon.noname-b con 0x7f59c80b4ff0 addr v1:172.21.15.115:6789/0 2024-03-25T18:36:06.210 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:36:06.207+0000 7f59d020f640 10 monclient(hunting): picked mon.noname-c con 0x7f59c813c480 addr v1:172.21.15.1:6790/0 2024-03-25T18:36:06.210 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:36:06.207+0000 7f59d020f640 1 -- --> v1:172.21.15.1:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f59c81312a0 con 0x7f59c80b4670 2024-03-25T18:36:06.210 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:36:06.207+0000 7f59d020f640 1 -- --> v1:172.21.15.115:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f59c8132730 con 0x7f59c80b4ff0 2024-03-25T18:36:06.210 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:36:06.207+0000 7f59d020f640 1 -- --> v1:172.21.15.1:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f59c81314f0 con 0x7f59c813c480 2024-03-25T18:36:06.210 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:36:06.207+0000 7f59d020f640 10 monclient(hunting): _renew_subs 2024-03-25T18:36:06.210 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:36:06.207+0000 7f59d020f640 10 monclient(hunting): authenticate will time out at 8046.677153s 2024-03-25T18:36:06.221 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:36:06.221+0000 7f59cd7aa640 1 --1- >> v1:172.21.15.115:6789/0 conn(0x7f59c80b4ff0 0x7f59c812fcb0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.115:6789/0 says I am v1:172.21.15.1:37272/0 (socket says 172.21.15.1:37272) 2024-03-25T18:36:06.221 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:36:06.221+0000 7f59cd7aa640 1 -- 172.21.15.1:0/1102599944 learned_addr learned my addr 172.21.15.1:0/1102599944 (peer_addr_for_me v1:172.21.15.1:0/0) 2024-03-25T18:36:06.222 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:36:06.222+0000 7f59be7fc640 10 client.?.objecter ms_handle_connect 0x7f59c80b4ff0 2024-03-25T18:36:06.222 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:36:06.222+0000 7f59be7fc640 10 client.?.objecter resend_mon_ops 2024-03-25T18:36:06.227 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:36:06.227+0000 7f59be7fc640 1 -- 172.21.15.1:0/1102599944 <== mon.1 v1:172.21.15.115:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3242463708 0 0) 0x7f59c8132730 con 0x7f59c80b4ff0 2024-03-25T18:36:06.227 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:36:06.227+0000 7f59be7fc640 10 monclient(hunting): _negotiate 2024-03-25T18:36:06.228 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:36:06.227+0000 7f59be7fc640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:36:06.228 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:36:06.227+0000 7f59be7fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:36:06.228 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:36:06.227+0000 7f59be7fc640 10 monclient(hunting): my global_id is 4633 2024-03-25T18:36:06.228 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:36:06.227+0000 7f59be7fc640 1 -- 172.21.15.1:0/1102599944 --> v1:172.21.15.115:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f59ac004eb0 con 0x7f59c80b4ff0 2024-03-25T18:36:06.229 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:36:06.229+0000 7f59be7fc640 10 client.?.objecter ms_handle_connect 0x7f59c813c480 2024-03-25T18:36:06.230 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:36:06.229+0000 7f59be7fc640 10 client.?.objecter resend_mon_ops 2024-03-25T18:36:06.239 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:36:06.239+0000 7f59be7fc640 1 -- 172.21.15.1:0/1102599944 <== mon.1 v1:172.21.15.115:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1035425396 0 0) 0x7f59ac004eb0 con 0x7f59c80b4ff0 2024-03-25T18:36:06.239 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:36:06.239+0000 7f59be7fc640 1 -- 172.21.15.1:0/1102599944 --> v1:172.21.15.115:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f59c8132730 con 0x7f59c80b4ff0 2024-03-25T18:36:06.239 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:36:06.239+0000 7f59be7fc640 1 -- 172.21.15.1:0/1102599944 <== mon.1 v1:172.21.15.115:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1586965025 0 0) 0x7f59b8001a30 con 0x7f59c80b4ff0 2024-03-25T18:36:06.239 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:36:06.239+0000 7f59be7fc640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:36:06.240 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:36:06.239+0000 7f59be7fc640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-03-25T18:36:06.240 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:36:06.239+0000 7f59be7fc640 10 monclient(hunting): dump: 2024-03-25T18:36:06.240 INFO:teuthology.misc.health.smithi001.stderr:epoch 1 2024-03-25T18:36:06.240 INFO:teuthology.misc.health.smithi001.stderr:fsid c8ea72cf-9e99-4383-9b52-6c23d365fdfc 2024-03-25T18:36:06.240 INFO:teuthology.misc.health.smithi001.stderr:last_changed 2024-03-25T18:25:50.510519+0000 2024-03-25T18:36:06.240 INFO:teuthology.misc.health.smithi001.stderr:created 2024-03-25T18:25:50.510519+0000 2024-03-25T18:36:06.240 INFO:teuthology.misc.health.smithi001.stderr:min_mon_release 19 (squid) 2024-03-25T18:36:06.240 INFO:teuthology.misc.health.smithi001.stderr:election_strategy: 3 2024-03-25T18:36:06.240 INFO:teuthology.misc.health.smithi001.stderr:0: v1:172.21.15.1:6789/0 mon.a 2024-03-25T18:36:06.240 INFO:teuthology.misc.health.smithi001.stderr:1: v1:172.21.15.115:6789/0 mon.b 2024-03-25T18:36:06.240 INFO:teuthology.misc.health.smithi001.stderr:2: v1:172.21.15.1:6790/0 mon.c 2024-03-25T18:36:06.240 INFO:teuthology.misc.health.smithi001.stderr: 2024-03-25T18:36:06.240 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:36:06.239+0000 7f59be7fc640 10 monclient(hunting): _finish_auth 0 2024-03-25T18:36:06.241 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:36:06.239+0000 7f59be7fc640 10 client.?.objecter ms_handle_connect 0x7f59c80b4670 2024-03-25T18:36:06.241 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:36:06.239+0000 7f59be7fc640 10 client.?.objecter resend_mon_ops 2024-03-25T18:36:06.257 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:36:06.256+0000 7f59be7fc640 1 -- 172.21.15.1:0/1102599944 <== mon.2 v1:172.21.15.1:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4240588300 0 0) 0x7f59c81314f0 con 0x7f59c813c480 2024-03-25T18:36:06.257 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:36:06.256+0000 7f59be7fc640 10 monclient(hunting): _negotiate 2024-03-25T18:36:06.257 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:36:06.256+0000 7f59be7fc640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:36:06.257 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:36:06.256+0000 7f59be7fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:36:06.257 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:36:06.256+0000 7f59be7fc640 10 monclient(hunting): my global_id is 4634 2024-03-25T18:36:06.257 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:36:06.256+0000 7f59be7fc640 1 -- 172.21.15.1:0/1102599944 --> v1:172.21.15.1:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f59ac004eb0 con 0x7f59c813c480 2024-03-25T18:36:06.259 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:36:06.258+0000 7f59be7fc640 1 -- 172.21.15.1:0/1102599944 <== mon.1 v1:172.21.15.115:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 288129665 0 0) 0x7f59c8132730 con 0x7f59c80b4ff0 2024-03-25T18:36:06.259 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:36:06.258+0000 7f59be7fc640 1 -- 172.21.15.1:0/1102599944 >> v1:172.21.15.1:6790/0 conn(0x7f59c813c480 legacy=0x7f59c81303c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:36:06.259 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:36:06.258+0000 7f59be7fc640 1 -- 172.21.15.1:0/1102599944 >> v1:172.21.15.1:6789/0 conn(0x7f59c80b4670 legacy=0x7f59c8132b50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:36:06.259 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:36:06.258+0000 7f59be7fc640 10 monclient: _finish_hunting 0 2024-03-25T18:36:06.260 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:36:06.258+0000 7f59be7fc640 1 monclient: found mon.b 2024-03-25T18:36:06.260 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:36:06.258+0000 7f59be7fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:36:06.260 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:36:06.258+0000 7f59be7fc640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.115:6789/0 2024-03-25T18:36:06.260 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:36:06.258+0000 7f59be7fc640 1 -- 172.21.15.1:0/1102599944 --> v1:172.21.15.115:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f59c81341c0 con 0x7f59c80b4ff0 2024-03-25T18:36:06.260 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:36:06.258+0000 7f59be7fc640 10 monclient: _finish_auth 0 2024-03-25T18:36:06.260 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:36:06.258+0000 7f59be7fc640 10 monclient: _check_auth_tickets 2024-03-25T18:36:06.260 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:36:06.258+0000 7f59be7fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:36:06.260 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:36:06.258+0000 7f59d020f640 5 monclient: authenticate success, global_id 4633 2024-03-25T18:36:06.260 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:36:06.258+0000 7f59d020f640 10 monclient: _renew_subs 2024-03-25T18:36:06.260 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:36:06.258+0000 7f59d020f640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.115:6789/0 2024-03-25T18:36:06.260 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:36:06.258+0000 7f59d020f640 1 -- 172.21.15.1:0/1102599944 --> v1:172.21.15.115:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f59c8130ad0 con 0x7f59c80b4ff0 2024-03-25T18:36:06.260 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:36:06.258+0000 7f59d020f640 10 client.4633.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T18:36:06.260 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:36:06.258+0000 7f59d020f640 10 monclient: _renew_subs 2024-03-25T18:36:06.260 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:36:06.258+0000 7f59d020f640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.115:6789/0 2024-03-25T18:36:06.260 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:36:06.258+0000 7f59d020f640 1 -- 172.21.15.1:0/1102599944 --> v1:172.21.15.115:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f59c81458a0 con 0x7f59c80b4ff0 2024-03-25T18:36:06.260 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:36:06.258+0000 7f59d020f640 1 librados: init done 2024-03-25T18:36:06.261 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:36:06.259+0000 7f59d020f640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T18:36:06.261 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:36:06.259+0000 7f59d020f640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:36:06.261 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:36:06.259+0000 7f59d020f640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.115:6789/0 2024-03-25T18:36:06.261 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:36:06.259+0000 7f59d020f640 1 -- 172.21.15.1:0/1102599944 --> v1:172.21.15.115:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f59c806dbd0 con 0x7f59c80b4ff0 2024-03-25T18:36:06.531 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:36:06.530+0000 7f59be7fc640 1 -- 172.21.15.1:0/1102599944 <== mon.1 v1:172.21.15.115:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f59b8001cf0 con 0x7f59c80b4ff0 2024-03-25T18:36:06.531 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:36:06.530+0000 7f59be7fc640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:36:06.531 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:36:06.530+0000 7f59be7fc640 1 -- 172.21.15.1:0/1102599944 <== mon.1 v1:172.21.15.115:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1586965025 0 0) 0x7f59b8003a60 con 0x7f59c80b4ff0 2024-03-25T18:36:06.531 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:36:06.530+0000 7f59be7fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:36:06.531 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:36:06.530+0000 7f59be7fc640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-25T18:36:06.531 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:36:06.530+0000 7f59be7fc640 10 monclient: dump: 2024-03-25T18:36:06.531 INFO:teuthology.misc.health.smithi001.stderr:epoch 1 2024-03-25T18:36:06.532 INFO:teuthology.misc.health.smithi001.stderr:fsid c8ea72cf-9e99-4383-9b52-6c23d365fdfc 2024-03-25T18:36:06.532 INFO:teuthology.misc.health.smithi001.stderr:last_changed 2024-03-25T18:25:50.510519+0000 2024-03-25T18:36:06.532 INFO:teuthology.misc.health.smithi001.stderr:created 2024-03-25T18:25:50.510519+0000 2024-03-25T18:36:06.532 INFO:teuthology.misc.health.smithi001.stderr:min_mon_release 19 (squid) 2024-03-25T18:36:06.532 INFO:teuthology.misc.health.smithi001.stderr:election_strategy: 3 2024-03-25T18:36:06.532 INFO:teuthology.misc.health.smithi001.stderr:0: v1:172.21.15.1:6789/0 mon.a 2024-03-25T18:36:06.532 INFO:teuthology.misc.health.smithi001.stderr:1: v1:172.21.15.115:6789/0 mon.b 2024-03-25T18:36:06.532 INFO:teuthology.misc.health.smithi001.stderr:2: v1:172.21.15.1:6790/0 mon.c 2024-03-25T18:36:06.532 INFO:teuthology.misc.health.smithi001.stderr: 2024-03-25T18:36:06.532 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:36:06.530+0000 7f59bffff640 4 set_mon_vals no callback set 2024-03-25T18:36:06.533 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:36:06.531+0000 7f59be7fc640 1 -- 172.21.15.1:0/1102599944 <== mon.1 v1:172.21.15.115:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 21787975 0 0) 0x7f59b8003c80 con 0x7f59c80b4ff0 2024-03-25T18:36:06.533 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:36:06.532+0000 7f59cdfab640 10 monclient: get_auth_request con 0x7f59ac06de20 auth_method 0 2024-03-25T18:36:06.533 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:36:06.532+0000 7f59be7fc640 10 client.4633.objecter ms_handle_connect 0x7f59ac06de20 2024-03-25T18:36:06.537 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:36:06.536+0000 7f59be7fc640 1 -- 172.21.15.1:0/1102599944 <== mon.1 v1:172.21.15.115:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 996254230 0 0) 0x7f59b8054810 con 0x7f59c80b4ff0 2024-03-25T18:36:06.537 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:36:06.536+0000 7f59be7fc640 10 client.4633.objecter ms_dispatch 0x7f59c8000b90 osd_map(10..10 src has 1..10) v4 2024-03-25T18:36:06.537 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:36:06.536+0000 7f59be7fc640 3 client.4633.objecter handle_osd_map got epochs [10,10] > 0 2024-03-25T18:36:06.537 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:36:06.536+0000 7f59be7fc640 3 client.4633.objecter handle_osd_map decoding full epoch 10 2024-03-25T18:36:06.537 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:36:06.536+0000 7f59be7fc640 20 client.4633.objecter dump_active .. 0 homeless 2024-03-25T18:36:06.537 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:36:06.536+0000 7f59be7fc640 1 -- 172.21.15.1:0/1102599944 <== mon.1 v1:172.21.15.115:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f59b8050320 con 0x7f59c80b4ff0 2024-03-25T18:36:06.537 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:36:06.536+0000 7f59be7fc640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:36:06.537 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:36:06.536+0000 7f59be7fc640 10 monclient: _finish_command 1 = system:0 2024-03-25T18:36:06.793 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:36:06.792+0000 7f59d020f640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:36:06.793 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:36:06.792+0000 7f59d020f640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:36:06.793 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:36:06.792+0000 7f59d020f640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.115:6789/0 2024-03-25T18:36:06.793 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:36:06.792+0000 7f59d020f640 1 -- 172.21.15.1:0/1102599944 --> v1:172.21.15.115:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f59c8130c60 con 0x7f59c80b4ff0 2024-03-25T18:36:06.819 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:36:06.818+0000 7f59be7fc640 1 -- 172.21.15.1:0/1102599944 <== mon.1 v1:172.21.15.115:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 4092213348) 0x7f59b8052250 con 0x7f59c80b4ff0 2024-03-25T18:36:06.819 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:36:06.818+0000 7f59be7fc640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:36:06.819 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:36:06.818+0000 7f59be7fc640 10 monclient: _finish_command 2 = system:0 2024-03-25T18:36:06.819 INFO:teuthology.misc.health.smithi001.stdout: 2024-03-25T18:36:06.819 INFO:teuthology.misc.health.smithi001.stdout:{"epoch":10,"fsid":"c8ea72cf-9e99-4383-9b52-6c23d365fdfc","created":"2024-03-25T18:26:38.856638+0000","modified":"2024-03-25T18:26:44.224208+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T18:26:44.224208+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"b9f34922-3a84-49be-8426-807c1eceaefc","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"42bf306f-55e3-4634-9874-8b4b73ac6644","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"292cb23a-18a9-4207-a7cc-4d5727feaec0","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"45e0e9ab-898d-4673-9590-b4117bc5e84a","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"831a5a09-73ab-4a61-9872-088e76bdbc66","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"88cd903d-3b55-4840-9396-6e3cfe005282","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"89db7355-bc9d-46f4-8ab0-e8d9e744e026","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"cac47bee-0914-4f08-896f-348838882866","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T18:36:06.822 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:36:06.821+0000 7f59d020f640 10 librados: watch_flush enter 2024-03-25T18:36:06.822 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:36:06.821+0000 7f59d020f640 10 librados: watch_flush exit 2024-03-25T18:36:06.822 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:36:06.821+0000 7f59d020f640 20 client.4633.objecter shutdown clearing up homeless session... 2024-03-25T18:36:06.822 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:36:06.821+0000 7f59d020f640 10 client.4633.objecter successfully canceled tick 2024-03-25T18:36:06.822 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:36:06.821+0000 7f59d020f640 1 -- 172.21.15.1:0/1102599944 >> v1:172.21.15.115:6800/4237379176 conn(0x7f59ac06de20 legacy=0x7f59ac0702e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:36:06.823 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:36:06.821+0000 7f59d020f640 10 monclient: shutdown 2024-03-25T18:36:06.823 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:36:06.821+0000 7f59d020f640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:36:06.823 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:36:06.821+0000 7f59d020f640 1 -- 172.21.15.1:0/1102599944 >> v1:172.21.15.115:6789/0 conn(0x7f59c80b4ff0 legacy=0x7f59c812fcb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:36:06.823 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:36:06.821+0000 7f59d020f640 1 -- 172.21.15.1:0/1102599944 shutdown_connections 2024-03-25T18:36:06.823 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:36:06.821+0000 7f59d020f640 1 -- 172.21.15.1:0/1102599944 >> 172.21.15.1:0/1102599944 conn(0x7f59c80617a0 msgr2=0x7f59c81407f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:36:06.823 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:36:06.821+0000 7f59d020f640 1 -- 172.21.15.1:0/1102599944 shutdown_connections 2024-03-25T18:36:06.823 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:36:06.821+0000 7f59d020f640 1 -- 172.21.15.1:0/1102599944 wait complete. 2024-03-25T18:36:06.823 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:36:06.821+0000 7f59d020f640 1 librados: shutdown 2024-03-25T18:36:06.832 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T18:36:13.639 DEBUG:teuthology.orchestra.run.smithi001:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T18:36:13.755 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:36:13.755+0000 7f191ae63640 10 monclient: get_monmap_and_config 2024-03-25T18:36:13.756 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:36:13.755+0000 7f191ae63640 10 monclient: build_initial_monmap 2024-03-25T18:36:13.756 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:36:13.755+0000 7f191ae63640 10 monclient: monmap: 2024-03-25T18:36:13.756 INFO:teuthology.misc.health.smithi001.stderr:epoch 0 2024-03-25T18:36:13.756 INFO:teuthology.misc.health.smithi001.stderr:fsid c8ea72cf-9e99-4383-9b52-6c23d365fdfc 2024-03-25T18:36:13.756 INFO:teuthology.misc.health.smithi001.stderr:last_changed 2024-03-25T18:36:13.755641+0000 2024-03-25T18:36:13.756 INFO:teuthology.misc.health.smithi001.stderr:created 2024-03-25T18:36:13.755641+0000 2024-03-25T18:36:13.756 INFO:teuthology.misc.health.smithi001.stderr:min_mon_release 0 (unknown) 2024-03-25T18:36:13.756 INFO:teuthology.misc.health.smithi001.stderr:election_strategy: 3 2024-03-25T18:36:13.756 INFO:teuthology.misc.health.smithi001.stderr:0: v1:172.21.15.1:6789/0 mon.noname-a 2024-03-25T18:36:13.756 INFO:teuthology.misc.health.smithi001.stderr:1: v1:172.21.15.115:6789/0 mon.noname-c 2024-03-25T18:36:13.756 INFO:teuthology.misc.health.smithi001.stderr:2: v1:172.21.15.1:6790/0 mon.noname-b 2024-03-25T18:36:13.756 INFO:teuthology.misc.health.smithi001.stderr: 2024-03-25T18:36:13.757 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:36:13.756+0000 7f191ae63640 1 Processor -- start 2024-03-25T18:36:13.757 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:36:13.756+0000 7f191ae63640 1 -- start start 2024-03-25T18:36:13.757 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:36:13.756+0000 7f191ae63640 10 monclient: init 2024-03-25T18:36:13.758 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:36:13.757+0000 7f191ae63640 10 monclient: _reopen_session rank -1 2024-03-25T18:36:13.758 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:36:13.757+0000 7f191ae63640 10 monclient: _add_conns ranks=[0,2,1] 2024-03-25T18:36:13.758 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:36:13.757+0000 7f191ae63640 10 monclient(hunting): picked mon.noname-a con 0x7f19140b4ff0 addr v1:172.21.15.1:6789/0 2024-03-25T18:36:13.758 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:36:13.757+0000 7f191ae63640 10 monclient(hunting): picked mon.noname-b con 0x7f19140b4670 addr v1:172.21.15.1:6790/0 2024-03-25T18:36:13.758 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:36:13.757+0000 7f191ae63640 10 monclient(hunting): picked mon.noname-c con 0x7f191413c480 addr v1:172.21.15.115:6789/0 2024-03-25T18:36:13.759 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:36:13.757+0000 7f191ae63640 1 -- --> v1:172.21.15.1:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f19140ab0e0 con 0x7f19140b4ff0 2024-03-25T18:36:13.759 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:36:13.757+0000 7f191ae63640 1 -- --> v1:172.21.15.115:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f19140ab3c0 con 0x7f191413c480 2024-03-25T18:36:13.759 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:36:13.757+0000 7f191ae63640 1 -- --> v1:172.21.15.1:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f19140a9f30 con 0x7f19140b4670 2024-03-25T18:36:13.759 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:36:13.757+0000 7f191ae63640 10 monclient(hunting): _renew_subs 2024-03-25T18:36:13.759 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:36:13.757+0000 7f191ae63640 10 monclient(hunting): authenticate will time out at 8054.227439s 2024-03-25T18:36:13.770 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:36:13.770+0000 7f1919400640 1 --1- >> v1:172.21.15.115:6789/0 conn(0x7f191413c480 0x7f191414dc00 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.115:6789/0 says I am v1:172.21.15.1:54510/0 (socket says 172.21.15.1:54510) 2024-03-25T18:36:13.770 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:36:13.770+0000 7f1919400640 1 -- 172.21.15.1:0/3795826442 learned_addr learned my addr 172.21.15.1:0/3795826442 (peer_addr_for_me v1:172.21.15.1:0/0) 2024-03-25T18:36:13.778 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:36:13.777+0000 7f19137fe640 1 -- 172.21.15.1:0/3795826442 <== mon.1 v1:172.21.15.115:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1302359405 0 0) 0x7f19140ab3c0 con 0x7f191413c480 2024-03-25T18:36:13.778 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:36:13.777+0000 7f19137fe640 10 monclient(hunting): _negotiate 2024-03-25T18:36:13.778 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:36:13.777+0000 7f19137fe640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:36:13.778 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:36:13.777+0000 7f19137fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:36:13.778 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:36:13.777+0000 7f19137fe640 10 monclient(hunting): my global_id is 4636 2024-03-25T18:36:13.778 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:36:13.777+0000 7f19137fe640 1 -- 172.21.15.1:0/3795826442 --> v1:172.21.15.115:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f1904003520 con 0x7f191413c480 2024-03-25T18:36:13.783 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:36:13.782+0000 7f19137fe640 1 -- 172.21.15.1:0/3795826442 <== mon.1 v1:172.21.15.115:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1430108093 0 0) 0x7f1904003520 con 0x7f191413c480 2024-03-25T18:36:13.783 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:36:13.782+0000 7f19137fe640 1 -- 172.21.15.1:0/3795826442 >> v1:172.21.15.1:6790/0 conn(0x7f19140b4670 legacy=0x7f19141425c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-25T18:36:13.783 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:36:13.782+0000 7f19137fe640 1 -- 172.21.15.1:0/3795826442 >> v1:172.21.15.1:6789/0 conn(0x7f19140b4ff0 legacy=0x7f191413bd70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-25T18:36:13.783 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:36:13.782+0000 7f19137fe640 10 monclient: _finish_hunting 0 2024-03-25T18:36:13.783 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:36:13.782+0000 7f19137fe640 1 monclient: found mon.noname-c 2024-03-25T18:36:13.783 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:36:13.782+0000 7f19137fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:36:13.783 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:36:13.782+0000 7f19137fe640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.115:6789/0 2024-03-25T18:36:13.783 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:36:13.782+0000 7f19137fe640 1 -- 172.21.15.1:0/3795826442 --> v1:172.21.15.115:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f19140a9b70 con 0x7f191413c480 2024-03-25T18:36:13.783 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:36:13.782+0000 7f19137fe640 10 monclient: _finish_auth 0 2024-03-25T18:36:13.784 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:36:13.782+0000 7f19137fe640 10 monclient: _check_auth_tickets 2024-03-25T18:36:13.784 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:36:13.782+0000 7f19137fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:36:13.784 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:36:13.782+0000 7f191ae63640 5 monclient: authenticate success, global_id 4636 2024-03-25T18:36:13.784 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:36:13.782+0000 7f191ae63640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T18:36:13.784 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:36:13.784+0000 7f19137fe640 1 -- 172.21.15.1:0/3795826442 <== mon.1 v1:172.21.15.115:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1586965025 0 0) 0x7f190c001a10 con 0x7f191413c480 2024-03-25T18:36:13.784 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:36:13.784+0000 7f19137fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:36:13.784 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:36:13.784+0000 7f19137fe640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-03-25T18:36:13.784 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:36:13.784+0000 7f19137fe640 10 monclient: dump: 2024-03-25T18:36:13.784 INFO:teuthology.misc.health.smithi001.stderr:epoch 1 2024-03-25T18:36:13.784 INFO:teuthology.misc.health.smithi001.stderr:fsid c8ea72cf-9e99-4383-9b52-6c23d365fdfc 2024-03-25T18:36:13.785 INFO:teuthology.misc.health.smithi001.stderr:last_changed 2024-03-25T18:25:50.510519+0000 2024-03-25T18:36:13.785 INFO:teuthology.misc.health.smithi001.stderr:created 2024-03-25T18:25:50.510519+0000 2024-03-25T18:36:13.785 INFO:teuthology.misc.health.smithi001.stderr:min_mon_release 19 (squid) 2024-03-25T18:36:13.785 INFO:teuthology.misc.health.smithi001.stderr:election_strategy: 3 2024-03-25T18:36:13.785 INFO:teuthology.misc.health.smithi001.stderr:0: v1:172.21.15.1:6789/0 mon.a 2024-03-25T18:36:13.785 INFO:teuthology.misc.health.smithi001.stderr:1: v1:172.21.15.115:6789/0 mon.b 2024-03-25T18:36:13.785 INFO:teuthology.misc.health.smithi001.stderr:2: v1:172.21.15.1:6790/0 mon.c 2024-03-25T18:36:13.785 INFO:teuthology.misc.health.smithi001.stderr: 2024-03-25T18:36:13.785 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:36:13.784+0000 7f191ae63640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T18:36:13.794 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:36:13.794+0000 7f19137fe640 1 -- 172.21.15.1:0/3795826442 <== mon.1 v1:172.21.15.115:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f190c001ce0 con 0x7f191413c480 2024-03-25T18:36:13.794 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:36:13.794+0000 7f19137fe640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:36:13.795 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:36:13.794+0000 7f19137fe640 1 -- 172.21.15.1:0/3795826442 <== mon.1 v1:172.21.15.115:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1586965025 0 0) 0x7f190c002070 con 0x7f191413c480 2024-03-25T18:36:13.795 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:36:13.794+0000 7f19137fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:36:13.795 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:36:13.794+0000 7f19137fe640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-25T18:36:13.795 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:36:13.794+0000 7f19137fe640 10 monclient: dump: 2024-03-25T18:36:13.795 INFO:teuthology.misc.health.smithi001.stderr:epoch 1 2024-03-25T18:36:13.795 INFO:teuthology.misc.health.smithi001.stderr:fsid c8ea72cf-9e99-4383-9b52-6c23d365fdfc 2024-03-25T18:36:13.795 INFO:teuthology.misc.health.smithi001.stderr:last_changed 2024-03-25T18:25:50.510519+0000 2024-03-25T18:36:13.795 INFO:teuthology.misc.health.smithi001.stderr:created 2024-03-25T18:25:50.510519+0000 2024-03-25T18:36:13.795 INFO:teuthology.misc.health.smithi001.stderr:min_mon_release 19 (squid) 2024-03-25T18:36:13.795 INFO:teuthology.misc.health.smithi001.stderr:election_strategy: 3 2024-03-25T18:36:13.795 INFO:teuthology.misc.health.smithi001.stderr:0: v1:172.21.15.1:6789/0 mon.a 2024-03-25T18:36:13.795 INFO:teuthology.misc.health.smithi001.stderr:1: v1:172.21.15.115:6789/0 mon.b 2024-03-25T18:36:13.795 INFO:teuthology.misc.health.smithi001.stderr:2: v1:172.21.15.1:6790/0 mon.c 2024-03-25T18:36:13.795 INFO:teuthology.misc.health.smithi001.stderr: 2024-03-25T18:36:13.795 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:36:13.794+0000 7f191ae63640 10 monclient: get_monmap_and_config success 2024-03-25T18:36:13.796 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:36:13.794+0000 7f191ae63640 4 set_mon_vals no callback set 2024-03-25T18:36:13.797 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:36:13.795+0000 7f191ae63640 10 monclient: shutdown 2024-03-25T18:36:13.797 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:36:13.795+0000 7f191ae63640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:36:13.797 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:36:13.795+0000 7f191ae63640 1 -- 172.21.15.1:0/3795826442 >> v1:172.21.15.115:6789/0 conn(0x7f191413c480 legacy=0x7f191414dc00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:36:13.797 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:36:13.795+0000 7f191ae63640 1 -- 172.21.15.1:0/3795826442 shutdown_connections 2024-03-25T18:36:13.797 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:36:13.795+0000 7f191ae63640 1 -- 172.21.15.1:0/3795826442 >> 172.21.15.1:0/3795826442 conn(0x7f19140617a0 msgr2=0x7f1914061bb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:36:13.797 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:36:13.795+0000 7f191ae63640 1 -- 172.21.15.1:0/3795826442 shutdown_connections 2024-03-25T18:36:13.797 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:36:13.795+0000 7f191ae63640 1 -- 172.21.15.1:0/3795826442 wait complete. 2024-03-25T18:36:13.797 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:36:13.796+0000 7f191ae63640 10 monclient: build_initial_monmap 2024-03-25T18:36:13.797 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:36:13.796+0000 7f191ae63640 10 monclient: monmap: 2024-03-25T18:36:13.797 INFO:teuthology.misc.health.smithi001.stderr:epoch 0 2024-03-25T18:36:13.797 INFO:teuthology.misc.health.smithi001.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T18:36:13.798 INFO:teuthology.misc.health.smithi001.stderr:last_changed 0.000000 2024-03-25T18:36:13.798 INFO:teuthology.misc.health.smithi001.stderr:created 0.000000 2024-03-25T18:36:13.798 INFO:teuthology.misc.health.smithi001.stderr:min_mon_release 0 (unknown) 2024-03-25T18:36:13.798 INFO:teuthology.misc.health.smithi001.stderr:election_strategy: 1 2024-03-25T18:36:13.798 INFO:teuthology.misc.health.smithi001.stderr:0: v1:172.21.15.1:6789/0 mon.noname-a 2024-03-25T18:36:13.798 INFO:teuthology.misc.health.smithi001.stderr:1: v1:172.21.15.115:6789/0 mon.noname-b 2024-03-25T18:36:13.798 INFO:teuthology.misc.health.smithi001.stderr:2: v1:172.21.15.1:6790/0 mon.noname-c 2024-03-25T18:36:13.798 INFO:teuthology.misc.health.smithi001.stderr: 2024-03-25T18:36:13.798 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:36:13.796+0000 7f191ae63640 1 librados: starting msgr at 2024-03-25T18:36:13.798 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:36:13.796+0000 7f191ae63640 1 librados: starting objecter 2024-03-25T18:36:13.798 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:36:13.796+0000 7f191ae63640 1 Processor -- start 2024-03-25T18:36:13.798 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:36:13.796+0000 7f191ae63640 1 -- start start 2024-03-25T18:36:13.798 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:36:13.796+0000 7f191ae63640 1 librados: setting wanted keys 2024-03-25T18:36:13.798 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:36:13.796+0000 7f191ae63640 1 librados: calling monclient init 2024-03-25T18:36:13.799 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:36:13.796+0000 7f191ae63640 10 monclient: init 2024-03-25T18:36:13.800 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:36:13.797+0000 7f191ae63640 10 monclient: _reopen_session rank -1 2024-03-25T18:36:13.800 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:36:13.797+0000 7f191ae63640 10 monclient: _add_conns ranks=[1,2,0] 2024-03-25T18:36:13.800 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:36:13.797+0000 7f191ae63640 10 monclient(hunting): picked mon.noname-b con 0x7f19140b4670 addr v1:172.21.15.115:6789/0 2024-03-25T18:36:13.800 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:36:13.797+0000 7f191ae63640 10 monclient(hunting): picked mon.noname-c con 0x7f19140b4ff0 addr v1:172.21.15.1:6790/0 2024-03-25T18:36:13.800 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:36:13.797+0000 7f191ae63640 10 monclient(hunting): picked mon.noname-a con 0x7f191413c480 addr v1:172.21.15.1:6789/0 2024-03-25T18:36:13.800 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:36:13.797+0000 7f191ae63640 1 -- --> v1:172.21.15.1:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f19141446b0 con 0x7f191413c480 2024-03-25T18:36:13.800 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:36:13.797+0000 7f191ae63640 1 -- --> v1:172.21.15.115:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1914144990 con 0x7f19140b4670 2024-03-25T18:36:13.800 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:36:13.797+0000 7f191ae63640 1 -- --> v1:172.21.15.1:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1914144c70 con 0x7f19140b4ff0 2024-03-25T18:36:13.800 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:36:13.797+0000 7f191ae63640 10 monclient(hunting): _renew_subs 2024-03-25T18:36:13.800 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:36:13.797+0000 7f191ae63640 10 monclient(hunting): authenticate will time out at 8054.267426s 2024-03-25T18:36:13.807 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:36:13.807+0000 7f1913fff640 1 --1- >> v1:172.21.15.1:6790/0 conn(0x7f19140b4ff0 0x7f1914145500 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.1:6790/0 says I am v1:172.21.15.1:51186/0 (socket says 172.21.15.1:51186) 2024-03-25T18:36:13.807 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:36:13.807+0000 7f1913fff640 1 -- 172.21.15.1:0/919068291 learned_addr learned my addr 172.21.15.1:0/919068291 (peer_addr_for_me v1:172.21.15.1:0/0) 2024-03-25T18:36:13.809 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:36:13.809+0000 7f19117fa640 10 client.?.objecter ms_handle_connect 0x7f19140b4ff0 2024-03-25T18:36:13.809 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:36:13.809+0000 7f19117fa640 10 client.?.objecter resend_mon_ops 2024-03-25T18:36:13.810 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:36:13.810+0000 7f19117fa640 10 client.?.objecter ms_handle_connect 0x7f19140b4670 2024-03-25T18:36:13.810 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:36:13.810+0000 7f19117fa640 10 client.?.objecter resend_mon_ops 2024-03-25T18:36:13.826 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:36:13.826+0000 7f19117fa640 10 client.?.objecter ms_handle_connect 0x7f191413c480 2024-03-25T18:36:13.827 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:36:13.826+0000 7f19117fa640 10 client.?.objecter resend_mon_ops 2024-03-25T18:36:13.880 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:36:13.879+0000 7f19117fa640 1 -- 172.21.15.1:0/919068291 <== mon.0 v1:172.21.15.1:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3904035590 0 0) 0x7f19141446b0 con 0x7f191413c480 2024-03-25T18:36:13.880 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:36:13.879+0000 7f19117fa640 10 monclient(hunting): _negotiate 2024-03-25T18:36:13.880 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:36:13.879+0000 7f19117fa640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:36:13.880 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:36:13.879+0000 7f19117fa640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:36:13.880 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:36:13.879+0000 7f19117fa640 10 monclient(hunting): my global_id is 4626 2024-03-25T18:36:13.880 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:36:13.879+0000 7f19117fa640 1 -- 172.21.15.1:0/919068291 --> v1:172.21.15.1:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f18f8004eb0 con 0x7f191413c480 2024-03-25T18:36:13.894 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:36:13.893+0000 7f19117fa640 1 -- 172.21.15.1:0/919068291 <== mon.0 v1:172.21.15.1:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1486914976 0 0) 0x7f18f8004eb0 con 0x7f191413c480 2024-03-25T18:36:13.894 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:36:13.893+0000 7f19117fa640 1 -- 172.21.15.1:0/919068291 --> v1:172.21.15.1:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f19141446b0 con 0x7f191413c480 2024-03-25T18:36:13.894 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:36:13.893+0000 7f19117fa640 1 -- 172.21.15.1:0/919068291 <== mon.0 v1:172.21.15.1:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1586965025 0 0) 0x7f190c0019d0 con 0x7f191413c480 2024-03-25T18:36:13.894 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:36:13.893+0000 7f19117fa640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:36:13.894 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:36:13.893+0000 7f19117fa640 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2024-03-25T18:36:13.894 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:36:13.893+0000 7f19117fa640 10 monclient(hunting): dump: 2024-03-25T18:36:13.894 INFO:teuthology.misc.health.smithi001.stderr:epoch 1 2024-03-25T18:36:13.894 INFO:teuthology.misc.health.smithi001.stderr:fsid c8ea72cf-9e99-4383-9b52-6c23d365fdfc 2024-03-25T18:36:13.894 INFO:teuthology.misc.health.smithi001.stderr:last_changed 2024-03-25T18:25:50.510519+0000 2024-03-25T18:36:13.895 INFO:teuthology.misc.health.smithi001.stderr:created 2024-03-25T18:25:50.510519+0000 2024-03-25T18:36:13.895 INFO:teuthology.misc.health.smithi001.stderr:min_mon_release 19 (squid) 2024-03-25T18:36:13.895 INFO:teuthology.misc.health.smithi001.stderr:election_strategy: 3 2024-03-25T18:36:13.895 INFO:teuthology.misc.health.smithi001.stderr:0: v1:172.21.15.1:6789/0 mon.a 2024-03-25T18:36:13.895 INFO:teuthology.misc.health.smithi001.stderr:1: v1:172.21.15.115:6789/0 mon.b 2024-03-25T18:36:13.895 INFO:teuthology.misc.health.smithi001.stderr:2: v1:172.21.15.1:6790/0 mon.c 2024-03-25T18:36:13.895 INFO:teuthology.misc.health.smithi001.stderr: 2024-03-25T18:36:13.895 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:36:13.893+0000 7f19117fa640 10 monclient(hunting): _finish_auth 0 2024-03-25T18:36:13.901 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:36:13.900+0000 7f19117fa640 1 -- 172.21.15.1:0/919068291 <== mon.2 v1:172.21.15.1:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1797719439 0 0) 0x7f1914144c70 con 0x7f19140b4ff0 2024-03-25T18:36:13.901 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:36:13.900+0000 7f19117fa640 10 monclient(hunting): _negotiate 2024-03-25T18:36:13.901 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:36:13.900+0000 7f19117fa640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:36:13.901 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:36:13.900+0000 7f19117fa640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:36:13.901 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:36:13.900+0000 7f19117fa640 10 monclient(hunting): my global_id is 4637 2024-03-25T18:36:13.901 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:36:13.900+0000 7f19117fa640 1 -- 172.21.15.1:0/919068291 --> v1:172.21.15.1:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f18f8004eb0 con 0x7f19140b4ff0 2024-03-25T18:36:13.902 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:36:13.901+0000 7f19117fa640 1 -- 172.21.15.1:0/919068291 <== mon.1 v1:172.21.15.115:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4140603466 0 0) 0x7f1914144990 con 0x7f19140b4670 2024-03-25T18:36:13.902 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:36:13.901+0000 7f19117fa640 10 monclient(hunting): _negotiate 2024-03-25T18:36:13.902 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:36:13.901+0000 7f19117fa640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:36:13.902 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:36:13.901+0000 7f19117fa640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:36:13.902 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:36:13.901+0000 7f19117fa640 10 monclient(hunting): my global_id is 4639 2024-03-25T18:36:13.902 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:36:13.901+0000 7f19117fa640 1 -- 172.21.15.1:0/919068291 --> v1:172.21.15.115:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f1914144c70 con 0x7f19140b4670 2024-03-25T18:36:13.916 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:36:13.915+0000 7f19117fa640 1 -- 172.21.15.1:0/919068291 <== mon.0 v1:172.21.15.1:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 4799596 0 0) 0x7f19141446b0 con 0x7f191413c480 2024-03-25T18:36:13.916 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:36:13.915+0000 7f19117fa640 1 -- 172.21.15.1:0/919068291 >> v1:172.21.15.1:6790/0 conn(0x7f19140b4ff0 legacy=0x7f1914145500 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:36:13.916 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:36:13.915+0000 7f19117fa640 1 -- 172.21.15.1:0/919068291 >> v1:172.21.15.115:6789/0 conn(0x7f19140b4670 legacy=0x7f1914143680 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:36:13.916 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:36:13.915+0000 7f19117fa640 10 monclient: _finish_hunting 0 2024-03-25T18:36:13.916 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:36:13.915+0000 7f19117fa640 1 monclient: found mon.a 2024-03-25T18:36:13.916 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:36:13.915+0000 7f19117fa640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:36:13.916 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:36:13.915+0000 7f19117fa640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.1:6789/0 2024-03-25T18:36:13.916 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:36:13.915+0000 7f19117fa640 1 -- 172.21.15.1:0/919068291 --> v1:172.21.15.1:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f19141329e0 con 0x7f191413c480 2024-03-25T18:36:13.916 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:36:13.915+0000 7f19117fa640 10 monclient: _finish_auth 0 2024-03-25T18:36:13.916 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:36:13.915+0000 7f19117fa640 10 monclient: _check_auth_tickets 2024-03-25T18:36:13.916 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:36:13.915+0000 7f19117fa640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:36:13.917 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:36:13.915+0000 7f191ae63640 5 monclient: authenticate success, global_id 4626 2024-03-25T18:36:13.917 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:36:13.915+0000 7f191ae63640 10 monclient: _renew_subs 2024-03-25T18:36:13.917 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:36:13.915+0000 7f191ae63640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.1:6789/0 2024-03-25T18:36:13.917 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:36:13.915+0000 7f191ae63640 1 -- 172.21.15.1:0/919068291 --> v1:172.21.15.1:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f191413ef90 con 0x7f191413c480 2024-03-25T18:36:13.917 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:36:13.915+0000 7f191ae63640 10 client.4626.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T18:36:13.917 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:36:13.915+0000 7f191ae63640 10 monclient: _renew_subs 2024-03-25T18:36:13.917 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:36:13.915+0000 7f191ae63640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.1:6789/0 2024-03-25T18:36:13.917 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:36:13.915+0000 7f191ae63640 1 -- 172.21.15.1:0/919068291 --> v1:172.21.15.1:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f19141e36c0 con 0x7f191413c480 2024-03-25T18:36:13.917 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:36:13.915+0000 7f191ae63640 1 librados: init done 2024-03-25T18:36:13.918 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:36:13.916+0000 7f191ae63640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T18:36:13.918 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:36:13.916+0000 7f191ae63640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:36:13.918 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:36:13.916+0000 7f191ae63640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.1:6789/0 2024-03-25T18:36:13.918 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:36:13.916+0000 7f191ae63640 1 -- 172.21.15.1:0/919068291 --> v1:172.21.15.1:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f19140b2690 con 0x7f191413c480 2024-03-25T18:36:14.245 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:36:14.244+0000 7f19117fa640 1 -- 172.21.15.1:0/919068291 <== mon.0 v1:172.21.15.1:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f190c004e10 con 0x7f191413c480 2024-03-25T18:36:14.245 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:36:14.244+0000 7f19117fa640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:36:14.245 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:36:14.244+0000 7f19117fa640 1 -- 172.21.15.1:0/919068291 <== mon.0 v1:172.21.15.1:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1586965025 0 0) 0x7f190c005220 con 0x7f191413c480 2024-03-25T18:36:14.245 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:36:14.244+0000 7f19117fa640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:36:14.245 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:36:14.244+0000 7f19117fa640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-25T18:36:14.246 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:36:14.244+0000 7f19117fa640 10 monclient: dump: 2024-03-25T18:36:14.246 INFO:teuthology.misc.health.smithi001.stderr:epoch 1 2024-03-25T18:36:14.246 INFO:teuthology.misc.health.smithi001.stderr:fsid c8ea72cf-9e99-4383-9b52-6c23d365fdfc 2024-03-25T18:36:14.246 INFO:teuthology.misc.health.smithi001.stderr:last_changed 2024-03-25T18:25:50.510519+0000 2024-03-25T18:36:14.246 INFO:teuthology.misc.health.smithi001.stderr:created 2024-03-25T18:25:50.510519+0000 2024-03-25T18:36:14.246 INFO:teuthology.misc.health.smithi001.stderr:min_mon_release 19 (squid) 2024-03-25T18:36:14.246 INFO:teuthology.misc.health.smithi001.stderr:election_strategy: 3 2024-03-25T18:36:14.246 INFO:teuthology.misc.health.smithi001.stderr:0: v1:172.21.15.1:6789/0 mon.a 2024-03-25T18:36:14.246 INFO:teuthology.misc.health.smithi001.stderr:1: v1:172.21.15.115:6789/0 mon.b 2024-03-25T18:36:14.246 INFO:teuthology.misc.health.smithi001.stderr:2: v1:172.21.15.1:6790/0 mon.c 2024-03-25T18:36:14.246 INFO:teuthology.misc.health.smithi001.stderr: 2024-03-25T18:36:14.246 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:36:14.244+0000 7f1912ffd640 4 set_mon_vals no callback set 2024-03-25T18:36:14.247 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:36:14.245+0000 7f19117fa640 1 -- 172.21.15.1:0/919068291 <== mon.0 v1:172.21.15.1:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 21787975 0 0) 0x7f190c005440 con 0x7f191413c480 2024-03-25T18:36:14.247 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:36:14.246+0000 7f1918bff640 10 monclient: get_auth_request con 0x7f18f806de80 auth_method 0 2024-03-25T18:36:14.247 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:36:14.246+0000 7f19117fa640 10 client.4626.objecter ms_handle_connect 0x7f18f806de80 2024-03-25T18:36:14.253 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:36:14.253+0000 7f19117fa640 1 -- 172.21.15.1:0/919068291 <== mon.0 v1:172.21.15.1:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 996254230 0 0) 0x7f190c054c60 con 0x7f191413c480 2024-03-25T18:36:14.253 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:36:14.253+0000 7f19117fa640 10 client.4626.objecter ms_dispatch 0x7f1914000b90 osd_map(10..10 src has 1..10) v4 2024-03-25T18:36:14.254 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:36:14.253+0000 7f19117fa640 3 client.4626.objecter handle_osd_map got epochs [10,10] > 0 2024-03-25T18:36:14.254 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:36:14.253+0000 7f19117fa640 3 client.4626.objecter handle_osd_map decoding full epoch 10 2024-03-25T18:36:14.254 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:36:14.253+0000 7f19117fa640 20 client.4626.objecter dump_active .. 0 homeless 2024-03-25T18:36:14.254 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:36:14.253+0000 7f19117fa640 1 -- 172.21.15.1:0/919068291 <== mon.0 v1:172.21.15.1:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f190c050770 con 0x7f191413c480 2024-03-25T18:36:14.254 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:36:14.253+0000 7f19117fa640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:36:14.254 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:36:14.253+0000 7f19117fa640 10 monclient: _finish_command 1 = system:0 2024-03-25T18:36:14.507 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:36:14.506+0000 7f191ae63640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:36:14.507 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:36:14.506+0000 7f191ae63640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:36:14.507 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:36:14.506+0000 7f191ae63640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.1:6789/0 2024-03-25T18:36:14.507 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:36:14.506+0000 7f191ae63640 1 -- 172.21.15.1:0/919068291 --> v1:172.21.15.1:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f19140b3fd0 con 0x7f191413c480 2024-03-25T18:36:14.527 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:36:14.526+0000 7f19117fa640 1 -- 172.21.15.1:0/919068291 <== mon.0 v1:172.21.15.1:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 4092213348) 0x7f190c0526a0 con 0x7f191413c480 2024-03-25T18:36:14.527 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:36:14.526+0000 7f19117fa640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:36:14.527 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:36:14.526+0000 7f19117fa640 10 monclient: _finish_command 2 = system:0 2024-03-25T18:36:14.527 INFO:teuthology.misc.health.smithi001.stdout: 2024-03-25T18:36:14.527 INFO:teuthology.misc.health.smithi001.stdout:{"epoch":10,"fsid":"c8ea72cf-9e99-4383-9b52-6c23d365fdfc","created":"2024-03-25T18:26:38.856638+0000","modified":"2024-03-25T18:26:44.224208+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T18:26:44.224208+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"b9f34922-3a84-49be-8426-807c1eceaefc","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"42bf306f-55e3-4634-9874-8b4b73ac6644","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"292cb23a-18a9-4207-a7cc-4d5727feaec0","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"45e0e9ab-898d-4673-9590-b4117bc5e84a","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"831a5a09-73ab-4a61-9872-088e76bdbc66","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"88cd903d-3b55-4840-9396-6e3cfe005282","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"89db7355-bc9d-46f4-8ab0-e8d9e744e026","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"cac47bee-0914-4f08-896f-348838882866","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T18:36:14.529 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:36:14.528+0000 7f191ae63640 10 librados: watch_flush enter 2024-03-25T18:36:14.529 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:36:14.528+0000 7f191ae63640 10 librados: watch_flush exit 2024-03-25T18:36:14.529 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:36:14.528+0000 7f191ae63640 20 client.4626.objecter shutdown clearing up homeless session... 2024-03-25T18:36:14.529 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:36:14.528+0000 7f191ae63640 10 client.4626.objecter successfully canceled tick 2024-03-25T18:36:14.529 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:36:14.528+0000 7f191ae63640 1 -- 172.21.15.1:0/919068291 >> v1:172.21.15.115:6800/4237379176 conn(0x7f18f806de80 legacy=0x7f18f8070340 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:36:14.530 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:36:14.528+0000 7f191ae63640 10 monclient: shutdown 2024-03-25T18:36:14.530 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:36:14.528+0000 7f191ae63640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:36:14.530 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:36:14.528+0000 7f191ae63640 1 -- 172.21.15.1:0/919068291 >> v1:172.21.15.1:6789/0 conn(0x7f191413c480 legacy=0x7f1914145c10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:36:14.530 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:36:14.528+0000 7f191ae63640 1 -- 172.21.15.1:0/919068291 shutdown_connections 2024-03-25T18:36:14.530 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:36:14.528+0000 7f191ae63640 1 -- 172.21.15.1:0/919068291 >> 172.21.15.1:0/919068291 conn(0x7f19140617a0 msgr2=0x7f19141425c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:36:14.530 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:36:14.528+0000 7f191ae63640 1 -- 172.21.15.1:0/919068291 shutdown_connections 2024-03-25T18:36:14.530 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:36:14.528+0000 7f191ae63640 1 -- 172.21.15.1:0/919068291 wait complete. 2024-03-25T18:36:14.530 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:36:14.528+0000 7f191ae63640 1 librados: shutdown 2024-03-25T18:36:14.542 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T18:36:21.349 DEBUG:teuthology.orchestra.run.smithi001:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T18:36:21.466 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:36:21.465+0000 7ff6da64e640 10 monclient: get_monmap_and_config 2024-03-25T18:36:21.466 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:36:21.465+0000 7ff6da64e640 10 monclient: build_initial_monmap 2024-03-25T18:36:21.466 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:36:21.465+0000 7ff6da64e640 10 monclient: monmap: 2024-03-25T18:36:21.466 INFO:teuthology.misc.health.smithi001.stderr:epoch 0 2024-03-25T18:36:21.466 INFO:teuthology.misc.health.smithi001.stderr:fsid c8ea72cf-9e99-4383-9b52-6c23d365fdfc 2024-03-25T18:36:21.466 INFO:teuthology.misc.health.smithi001.stderr:last_changed 2024-03-25T18:36:21.465916+0000 2024-03-25T18:36:21.466 INFO:teuthology.misc.health.smithi001.stderr:created 2024-03-25T18:36:21.465916+0000 2024-03-25T18:36:21.466 INFO:teuthology.misc.health.smithi001.stderr:min_mon_release 0 (unknown) 2024-03-25T18:36:21.466 INFO:teuthology.misc.health.smithi001.stderr:election_strategy: 3 2024-03-25T18:36:21.467 INFO:teuthology.misc.health.smithi001.stderr:0: v1:172.21.15.1:6789/0 mon.noname-a 2024-03-25T18:36:21.467 INFO:teuthology.misc.health.smithi001.stderr:1: v1:172.21.15.115:6789/0 mon.noname-c 2024-03-25T18:36:21.467 INFO:teuthology.misc.health.smithi001.stderr:2: v1:172.21.15.1:6790/0 mon.noname-b 2024-03-25T18:36:21.467 INFO:teuthology.misc.health.smithi001.stderr: 2024-03-25T18:36:21.469 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:36:21.466+0000 7ff6da64e640 1 Processor -- start 2024-03-25T18:36:21.469 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:36:21.466+0000 7ff6da64e640 1 -- start start 2024-03-25T18:36:21.469 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:36:21.466+0000 7ff6da64e640 10 monclient: init 2024-03-25T18:36:21.469 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:36:21.467+0000 7ff6da64e640 10 monclient: _reopen_session rank -1 2024-03-25T18:36:21.469 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:36:21.467+0000 7ff6da64e640 10 monclient: _add_conns ranks=[1,2,0] 2024-03-25T18:36:21.469 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:36:21.467+0000 7ff6da64e640 10 monclient(hunting): picked mon.noname-c con 0x7ff6d406ae10 addr v1:172.21.15.115:6789/0 2024-03-25T18:36:21.469 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:36:21.467+0000 7ff6da64e640 10 monclient(hunting): picked mon.noname-b con 0x7ff6d413da60 addr v1:172.21.15.1:6790/0 2024-03-25T18:36:21.469 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:36:21.467+0000 7ff6da64e640 10 monclient(hunting): picked mon.noname-a con 0x7ff6d4143d40 addr v1:172.21.15.1:6789/0 2024-03-25T18:36:21.469 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:36:21.467+0000 7ff6da64e640 1 -- --> v1:172.21.15.1:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff6d40afdc0 con 0x7ff6d4143d40 2024-03-25T18:36:21.469 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:36:21.467+0000 7ff6da64e640 1 -- --> v1:172.21.15.115:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff6d40b1160 con 0x7ff6d406ae10 2024-03-25T18:36:21.469 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:36:21.467+0000 7ff6da64e640 1 -- --> v1:172.21.15.1:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff6d40b1440 con 0x7ff6d413da60 2024-03-25T18:36:21.470 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:36:21.467+0000 7ff6da64e640 10 monclient(hunting): _renew_subs 2024-03-25T18:36:21.470 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:36:21.467+0000 7ff6da64e640 10 monclient(hunting): authenticate will time out at 8061.936928s 2024-03-25T18:36:21.471 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:36:21.470+0000 7ff6d37fe640 1 --1- >> v1:172.21.15.1:6790/0 conn(0x7ff6d413da60 0x7ff6d4143630 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.1:6790/0 says I am v1:172.21.15.1:56272/0 (socket says 172.21.15.1:56272) 2024-03-25T18:36:21.471 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:36:21.470+0000 7ff6d37fe640 1 -- 172.21.15.1:0/2991475687 learned_addr learned my addr 172.21.15.1:0/2991475687 (peer_addr_for_me v1:172.21.15.1:0/0) 2024-03-25T18:36:21.477 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:36:21.476+0000 7ff6d2ffd640 1 -- 172.21.15.1:0/2991475687 <== mon.2 v1:172.21.15.1:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2985753544 0 0) 0x7ff6d40b1440 con 0x7ff6d413da60 2024-03-25T18:36:21.477 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:36:21.476+0000 7ff6d2ffd640 10 monclient(hunting): _negotiate 2024-03-25T18:36:21.478 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:36:21.476+0000 7ff6d2ffd640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:36:21.478 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:36:21.476+0000 7ff6d2ffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:36:21.478 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:36:21.476+0000 7ff6d2ffd640 10 monclient(hunting): my global_id is 4640 2024-03-25T18:36:21.478 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:36:21.476+0000 7ff6d2ffd640 1 -- 172.21.15.1:0/2991475687 --> v1:172.21.15.1:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ff6bc003520 con 0x7ff6d413da60 2024-03-25T18:36:21.488 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:36:21.487+0000 7ff6d2ffd640 1 -- 172.21.15.1:0/2991475687 <== mon.2 v1:172.21.15.1:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2402439200 0 0) 0x7ff6bc003520 con 0x7ff6d413da60 2024-03-25T18:36:21.488 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:36:21.487+0000 7ff6d2ffd640 1 -- 172.21.15.1:0/2991475687 >> v1:172.21.15.115:6789/0 conn(0x7ff6d406ae10 legacy=0x7ff6d413d350 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:36:21.488 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:36:21.487+0000 7ff6d2ffd640 1 -- 172.21.15.1:0/2991475687 >> v1:172.21.15.1:6789/0 conn(0x7ff6d4143d40 legacy=0x7ff6d414b9d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:36:21.488 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:36:21.487+0000 7ff6d2ffd640 10 monclient: _finish_hunting 0 2024-03-25T18:36:21.488 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:36:21.487+0000 7ff6d2ffd640 1 monclient: found mon.noname-b 2024-03-25T18:36:21.488 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:36:21.487+0000 7ff6d2ffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:36:21.488 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:36:21.487+0000 7ff6d2ffd640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.1:6790/0 2024-03-25T18:36:21.489 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:36:21.487+0000 7ff6d2ffd640 1 -- 172.21.15.1:0/2991475687 --> v1:172.21.15.1:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff6d40b0250 con 0x7ff6d413da60 2024-03-25T18:36:21.489 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:36:21.487+0000 7ff6d2ffd640 10 monclient: _finish_auth 0 2024-03-25T18:36:21.489 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:36:21.487+0000 7ff6d2ffd640 10 monclient: _check_auth_tickets 2024-03-25T18:36:21.489 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:36:21.487+0000 7ff6d2ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:36:21.489 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:36:21.487+0000 7ff6d2ffd640 1 -- 172.21.15.1:0/2991475687 <== mon.2 v1:172.21.15.1:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1586965025 0 0) 0x7ff6c8001690 con 0x7ff6d413da60 2024-03-25T18:36:21.489 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:36:21.487+0000 7ff6d2ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:36:21.489 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:36:21.487+0000 7ff6d2ffd640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-03-25T18:36:21.489 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:36:21.487+0000 7ff6d2ffd640 10 monclient: dump: 2024-03-25T18:36:21.489 INFO:teuthology.misc.health.smithi001.stderr:epoch 1 2024-03-25T18:36:21.489 INFO:teuthology.misc.health.smithi001.stderr:fsid c8ea72cf-9e99-4383-9b52-6c23d365fdfc 2024-03-25T18:36:21.489 INFO:teuthology.misc.health.smithi001.stderr:last_changed 2024-03-25T18:25:50.510519+0000 2024-03-25T18:36:21.489 INFO:teuthology.misc.health.smithi001.stderr:created 2024-03-25T18:25:50.510519+0000 2024-03-25T18:36:21.489 INFO:teuthology.misc.health.smithi001.stderr:min_mon_release 19 (squid) 2024-03-25T18:36:21.489 INFO:teuthology.misc.health.smithi001.stderr:election_strategy: 3 2024-03-25T18:36:21.490 INFO:teuthology.misc.health.smithi001.stderr:0: v1:172.21.15.1:6789/0 mon.a 2024-03-25T18:36:21.490 INFO:teuthology.misc.health.smithi001.stderr:1: v1:172.21.15.115:6789/0 mon.b 2024-03-25T18:36:21.490 INFO:teuthology.misc.health.smithi001.stderr:2: v1:172.21.15.1:6790/0 mon.c 2024-03-25T18:36:21.490 INFO:teuthology.misc.health.smithi001.stderr: 2024-03-25T18:36:21.490 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:36:21.487+0000 7ff6da64e640 5 monclient: authenticate success, global_id 4640 2024-03-25T18:36:21.490 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:36:21.487+0000 7ff6da64e640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T18:36:21.494 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:36:21.494+0000 7ff6d2ffd640 1 -- 172.21.15.1:0/2991475687 <== mon.2 v1:172.21.15.1:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7ff6c8001870 con 0x7ff6d413da60 2024-03-25T18:36:21.495 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:36:21.494+0000 7ff6d2ffd640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:36:21.495 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:36:21.494+0000 7ff6d2ffd640 1 -- 172.21.15.1:0/2991475687 <== mon.2 v1:172.21.15.1:6790/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1586965025 0 0) 0x7ff6c8001c80 con 0x7ff6d413da60 2024-03-25T18:36:21.495 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:36:21.494+0000 7ff6d2ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:36:21.495 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:36:21.494+0000 7ff6d2ffd640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-25T18:36:21.495 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:36:21.494+0000 7ff6d2ffd640 10 monclient: dump: 2024-03-25T18:36:21.495 INFO:teuthology.misc.health.smithi001.stderr:epoch 1 2024-03-25T18:36:21.495 INFO:teuthology.misc.health.smithi001.stderr:fsid c8ea72cf-9e99-4383-9b52-6c23d365fdfc 2024-03-25T18:36:21.495 INFO:teuthology.misc.health.smithi001.stderr:last_changed 2024-03-25T18:25:50.510519+0000 2024-03-25T18:36:21.495 INFO:teuthology.misc.health.smithi001.stderr:created 2024-03-25T18:25:50.510519+0000 2024-03-25T18:36:21.495 INFO:teuthology.misc.health.smithi001.stderr:min_mon_release 19 (squid) 2024-03-25T18:36:21.495 INFO:teuthology.misc.health.smithi001.stderr:election_strategy: 3 2024-03-25T18:36:21.495 INFO:teuthology.misc.health.smithi001.stderr:0: v1:172.21.15.1:6789/0 mon.a 2024-03-25T18:36:21.496 INFO:teuthology.misc.health.smithi001.stderr:1: v1:172.21.15.115:6789/0 mon.b 2024-03-25T18:36:21.496 INFO:teuthology.misc.health.smithi001.stderr:2: v1:172.21.15.1:6790/0 mon.c 2024-03-25T18:36:21.496 INFO:teuthology.misc.health.smithi001.stderr: 2024-03-25T18:36:21.496 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:36:21.494+0000 7ff6da64e640 10 monclient: get_monmap_and_config success 2024-03-25T18:36:21.496 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:36:21.494+0000 7ff6da64e640 4 set_mon_vals no callback set 2024-03-25T18:36:21.499 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:36:21.495+0000 7ff6da64e640 10 monclient: shutdown 2024-03-25T18:36:21.499 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:36:21.495+0000 7ff6da64e640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:36:21.499 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:36:21.495+0000 7ff6da64e640 1 -- 172.21.15.1:0/2991475687 >> v1:172.21.15.1:6790/0 conn(0x7ff6d413da60 legacy=0x7ff6d4143630 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:36:21.499 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:36:21.495+0000 7ff6da64e640 1 -- 172.21.15.1:0/2991475687 shutdown_connections 2024-03-25T18:36:21.499 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:36:21.495+0000 7ff6da64e640 1 -- 172.21.15.1:0/2991475687 >> 172.21.15.1:0/2991475687 conn(0x7ff6d40b4bd0 msgr2=0x7ff6d40b4fe0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:36:21.500 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:36:21.495+0000 7ff6da64e640 1 -- 172.21.15.1:0/2991475687 shutdown_connections 2024-03-25T18:36:21.500 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:36:21.495+0000 7ff6da64e640 1 -- 172.21.15.1:0/2991475687 wait complete. 2024-03-25T18:36:21.500 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:36:21.496+0000 7ff6da64e640 10 monclient: build_initial_monmap 2024-03-25T18:36:21.500 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:36:21.496+0000 7ff6da64e640 10 monclient: monmap: 2024-03-25T18:36:21.500 INFO:teuthology.misc.health.smithi001.stderr:epoch 0 2024-03-25T18:36:21.500 INFO:teuthology.misc.health.smithi001.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T18:36:21.500 INFO:teuthology.misc.health.smithi001.stderr:last_changed 0.000000 2024-03-25T18:36:21.500 INFO:teuthology.misc.health.smithi001.stderr:created 0.000000 2024-03-25T18:36:21.500 INFO:teuthology.misc.health.smithi001.stderr:min_mon_release 0 (unknown) 2024-03-25T18:36:21.500 INFO:teuthology.misc.health.smithi001.stderr:election_strategy: 1 2024-03-25T18:36:21.500 INFO:teuthology.misc.health.smithi001.stderr:0: v1:172.21.15.1:6789/0 mon.noname-a 2024-03-25T18:36:21.501 INFO:teuthology.misc.health.smithi001.stderr:1: v1:172.21.15.115:6789/0 mon.noname-b 2024-03-25T18:36:21.501 INFO:teuthology.misc.health.smithi001.stderr:2: v1:172.21.15.1:6790/0 mon.noname-c 2024-03-25T18:36:21.501 INFO:teuthology.misc.health.smithi001.stderr: 2024-03-25T18:36:21.501 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:36:21.496+0000 7ff6da64e640 1 librados: starting msgr at 2024-03-25T18:36:21.501 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:36:21.496+0000 7ff6da64e640 1 librados: starting objecter 2024-03-25T18:36:21.501 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:36:21.496+0000 7ff6da64e640 1 Processor -- start 2024-03-25T18:36:21.501 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:36:21.497+0000 7ff6da64e640 1 -- start start 2024-03-25T18:36:21.501 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:36:21.497+0000 7ff6da64e640 1 librados: setting wanted keys 2024-03-25T18:36:21.501 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:36:21.497+0000 7ff6da64e640 1 librados: calling monclient init 2024-03-25T18:36:21.501 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:36:21.497+0000 7ff6da64e640 10 monclient: init 2024-03-25T18:36:21.501 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:36:21.497+0000 7ff6da64e640 10 monclient: _reopen_session rank -1 2024-03-25T18:36:21.501 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:36:21.497+0000 7ff6da64e640 10 monclient: _add_conns ranks=[2,1,0] 2024-03-25T18:36:21.501 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:36:21.497+0000 7ff6da64e640 10 monclient(hunting): picked mon.noname-c con 0x7ff6d406ae10 addr v1:172.21.15.1:6790/0 2024-03-25T18:36:21.501 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:36:21.497+0000 7ff6da64e640 10 monclient(hunting): picked mon.noname-b con 0x7ff6d413da60 addr v1:172.21.15.115:6789/0 2024-03-25T18:36:21.502 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:36:21.497+0000 7ff6da64e640 10 monclient(hunting): picked mon.noname-a con 0x7ff6d4143d40 addr v1:172.21.15.1:6789/0 2024-03-25T18:36:21.502 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:36:21.498+0000 7ff6da64e640 1 -- --> v1:172.21.15.1:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff6d40adff0 con 0x7ff6d4143d40 2024-03-25T18:36:21.502 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:36:21.498+0000 7ff6da64e640 1 -- --> v1:172.21.15.115:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff6d40ae2d0 con 0x7ff6d413da60 2024-03-25T18:36:21.502 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:36:21.498+0000 7ff6da64e640 1 -- --> v1:172.21.15.1:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff6d40ad230 con 0x7ff6d406ae10 2024-03-25T18:36:21.502 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:36:21.498+0000 7ff6da64e640 10 monclient(hunting): _renew_subs 2024-03-25T18:36:21.502 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:36:21.498+0000 7ff6da64e640 10 monclient(hunting): authenticate will time out at 8061.967705s 2024-03-25T18:36:21.508 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:36:21.507+0000 7ff6d37fe640 1 --1- >> v1:172.21.15.115:6789/0 conn(0x7ff6d413da60 0x7ff6d4138ba0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.115:6789/0 says I am v1:172.21.15.1:59478/0 (socket says 172.21.15.1:59478) 2024-03-25T18:36:21.508 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:36:21.507+0000 7ff6d37fe640 1 -- 172.21.15.1:0/3775684608 learned_addr learned my addr 172.21.15.1:0/3775684608 (peer_addr_for_me v1:172.21.15.1:0/0) 2024-03-25T18:36:21.508 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:36:21.508+0000 7ff6d0ff9640 10 client.?.objecter ms_handle_connect 0x7ff6d413da60 2024-03-25T18:36:21.509 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:36:21.508+0000 7ff6d0ff9640 10 client.?.objecter resend_mon_ops 2024-03-25T18:36:21.511 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:36:21.511+0000 7ff6d0ff9640 10 client.?.objecter ms_handle_connect 0x7ff6d4143d40 2024-03-25T18:36:21.511 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:36:21.511+0000 7ff6d0ff9640 10 client.?.objecter resend_mon_ops 2024-03-25T18:36:21.514 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:36:21.513+0000 7ff6d0ff9640 1 -- 172.21.15.1:0/3775684608 <== mon.1 v1:172.21.15.115:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2859365036 0 0) 0x7ff6d40ae2d0 con 0x7ff6d413da60 2024-03-25T18:36:21.515 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:36:21.513+0000 7ff6d0ff9640 10 monclient(hunting): _negotiate 2024-03-25T18:36:21.515 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:36:21.513+0000 7ff6d0ff9640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:36:21.515 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:36:21.513+0000 7ff6d0ff9640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:36:21.515 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:36:21.513+0000 7ff6d0ff9640 10 monclient(hunting): my global_id is 4645 2024-03-25T18:36:21.515 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:36:21.513+0000 7ff6d0ff9640 1 -- 172.21.15.1:0/3775684608 --> v1:172.21.15.115:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ff6c0004eb0 con 0x7ff6d413da60 2024-03-25T18:36:21.516 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:36:21.514+0000 7ff6d0ff9640 1 -- 172.21.15.1:0/3775684608 <== mon.0 v1:172.21.15.1:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1388525006 0 0) 0x7ff6d40adff0 con 0x7ff6d4143d40 2024-03-25T18:36:21.516 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:36:21.514+0000 7ff6d0ff9640 10 monclient(hunting): _negotiate 2024-03-25T18:36:21.516 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:36:21.514+0000 7ff6d0ff9640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:36:21.516 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:36:21.514+0000 7ff6d0ff9640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:36:21.516 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:36:21.514+0000 7ff6d0ff9640 10 monclient(hunting): my global_id is 4632 2024-03-25T18:36:21.516 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:36:21.514+0000 7ff6d0ff9640 1 -- 172.21.15.1:0/3775684608 --> v1:172.21.15.1:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ff6d40ae2d0 con 0x7ff6d4143d40 2024-03-25T18:36:21.516 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:36:21.515+0000 7ff6d0ff9640 10 client.?.objecter ms_handle_connect 0x7ff6d406ae10 2024-03-25T18:36:21.516 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:36:21.515+0000 7ff6d0ff9640 10 client.?.objecter resend_mon_ops 2024-03-25T18:36:21.519 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:36:21.519+0000 7ff6d0ff9640 1 -- 172.21.15.1:0/3775684608 <== mon.2 v1:172.21.15.1:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1500786217 0 0) 0x7ff6d40ad230 con 0x7ff6d406ae10 2024-03-25T18:36:21.519 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:36:21.519+0000 7ff6d0ff9640 10 monclient(hunting): _negotiate 2024-03-25T18:36:21.519 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:36:21.519+0000 7ff6d0ff9640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:36:21.519 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:36:21.519+0000 7ff6d0ff9640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:36:21.520 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:36:21.519+0000 7ff6d0ff9640 10 monclient(hunting): my global_id is 4643 2024-03-25T18:36:21.520 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:36:21.519+0000 7ff6d0ff9640 1 -- 172.21.15.1:0/3775684608 --> v1:172.21.15.1:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ff6d40adff0 con 0x7ff6d406ae10 2024-03-25T18:36:21.527 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:36:21.525+0000 7ff6d0ff9640 1 -- 172.21.15.1:0/3775684608 <== mon.1 v1:172.21.15.115:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3285868583 0 0) 0x7ff6c0004eb0 con 0x7ff6d413da60 2024-03-25T18:36:21.527 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:36:21.525+0000 7ff6d0ff9640 1 -- 172.21.15.1:0/3775684608 --> v1:172.21.15.115:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7ff6d40ad230 con 0x7ff6d413da60 2024-03-25T18:36:21.527 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:36:21.525+0000 7ff6d0ff9640 1 -- 172.21.15.1:0/3775684608 <== mon.1 v1:172.21.15.115:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1586965025 0 0) 0x7ff6c8001bc0 con 0x7ff6d413da60 2024-03-25T18:36:21.527 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:36:21.525+0000 7ff6d0ff9640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:36:21.527 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:36:21.525+0000 7ff6d0ff9640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-03-25T18:36:21.527 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:36:21.525+0000 7ff6d0ff9640 10 monclient(hunting): dump: 2024-03-25T18:36:21.527 INFO:teuthology.misc.health.smithi001.stderr:epoch 1 2024-03-25T18:36:21.527 INFO:teuthology.misc.health.smithi001.stderr:fsid c8ea72cf-9e99-4383-9b52-6c23d365fdfc 2024-03-25T18:36:21.527 INFO:teuthology.misc.health.smithi001.stderr:last_changed 2024-03-25T18:25:50.510519+0000 2024-03-25T18:36:21.527 INFO:teuthology.misc.health.smithi001.stderr:created 2024-03-25T18:25:50.510519+0000 2024-03-25T18:36:21.527 INFO:teuthology.misc.health.smithi001.stderr:min_mon_release 19 (squid) 2024-03-25T18:36:21.527 INFO:teuthology.misc.health.smithi001.stderr:election_strategy: 3 2024-03-25T18:36:21.528 INFO:teuthology.misc.health.smithi001.stderr:0: v1:172.21.15.1:6789/0 mon.a 2024-03-25T18:36:21.528 INFO:teuthology.misc.health.smithi001.stderr:1: v1:172.21.15.115:6789/0 mon.b 2024-03-25T18:36:21.528 INFO:teuthology.misc.health.smithi001.stderr:2: v1:172.21.15.1:6790/0 mon.c 2024-03-25T18:36:21.528 INFO:teuthology.misc.health.smithi001.stderr: 2024-03-25T18:36:21.528 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:36:21.525+0000 7ff6d0ff9640 10 monclient(hunting): _finish_auth 0 2024-03-25T18:36:21.533 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:36:21.532+0000 7ff6d0ff9640 1 -- 172.21.15.1:0/3775684608 <== mon.2 v1:172.21.15.1:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1130904264 0 0) 0x7ff6d40adff0 con 0x7ff6d406ae10 2024-03-25T18:36:21.533 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:36:21.532+0000 7ff6d0ff9640 1 -- 172.21.15.1:0/3775684608 --> v1:172.21.15.1:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7ff6c0004eb0 con 0x7ff6d406ae10 2024-03-25T18:36:21.533 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:36:21.532+0000 7ff6d0ff9640 1 -- 172.21.15.1:0/3775684608 <== mon.2 v1:172.21.15.1:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1586965025 0 0) 0x7ff6c40017e0 con 0x7ff6d406ae10 2024-03-25T18:36:21.533 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:36:21.532+0000 7ff6d0ff9640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:36:21.533 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:36:21.532+0000 7ff6d0ff9640 10 monclient(hunting): got monmap 1 from mon.c (according to old e1) 2024-03-25T18:36:21.533 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:36:21.532+0000 7ff6d0ff9640 10 monclient(hunting): dump: 2024-03-25T18:36:21.534 INFO:teuthology.misc.health.smithi001.stderr:epoch 1 2024-03-25T18:36:21.534 INFO:teuthology.misc.health.smithi001.stderr:fsid c8ea72cf-9e99-4383-9b52-6c23d365fdfc 2024-03-25T18:36:21.534 INFO:teuthology.misc.health.smithi001.stderr:last_changed 2024-03-25T18:25:50.510519+0000 2024-03-25T18:36:21.534 INFO:teuthology.misc.health.smithi001.stderr:created 2024-03-25T18:25:50.510519+0000 2024-03-25T18:36:21.534 INFO:teuthology.misc.health.smithi001.stderr:min_mon_release 19 (squid) 2024-03-25T18:36:21.534 INFO:teuthology.misc.health.smithi001.stderr:election_strategy: 3 2024-03-25T18:36:21.534 INFO:teuthology.misc.health.smithi001.stderr:0: v1:172.21.15.1:6789/0 mon.a 2024-03-25T18:36:21.534 INFO:teuthology.misc.health.smithi001.stderr:1: v1:172.21.15.115:6789/0 mon.b 2024-03-25T18:36:21.534 INFO:teuthology.misc.health.smithi001.stderr:2: v1:172.21.15.1:6790/0 mon.c 2024-03-25T18:36:21.534 INFO:teuthology.misc.health.smithi001.stderr: 2024-03-25T18:36:21.538 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:36:21.537+0000 7ff6d0ff9640 1 -- 172.21.15.1:0/3775684608 <== mon.1 v1:172.21.15.115:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 2250561096 0 0) 0x7ff6d40ad230 con 0x7ff6d413da60 2024-03-25T18:36:21.539 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:36:21.537+0000 7ff6d0ff9640 1 -- 172.21.15.1:0/3775684608 >> v1:172.21.15.1:6790/0 conn(0x7ff6d406ae10 legacy=0x7ff6d40acb20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:36:21.539 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:36:21.537+0000 7ff6d0ff9640 1 -- 172.21.15.1:0/3775684608 >> v1:172.21.15.1:6789/0 conn(0x7ff6d4143d40 legacy=0x7ff6d41d56c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:36:21.539 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:36:21.537+0000 7ff6d0ff9640 10 monclient: _finish_hunting 0 2024-03-25T18:36:21.539 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:36:21.537+0000 7ff6d0ff9640 1 monclient: found mon.b 2024-03-25T18:36:21.539 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:36:21.537+0000 7ff6d0ff9640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:36:21.539 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:36:21.537+0000 7ff6d0ff9640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.115:6789/0 2024-03-25T18:36:21.539 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:36:21.537+0000 7ff6d0ff9640 1 -- 172.21.15.1:0/3775684608 --> v1:172.21.15.115:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff6d40ac610 con 0x7ff6d413da60 2024-03-25T18:36:21.539 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:36:21.537+0000 7ff6d0ff9640 10 monclient: _finish_auth 0 2024-03-25T18:36:21.539 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:36:21.537+0000 7ff6d0ff9640 10 monclient: _check_auth_tickets 2024-03-25T18:36:21.539 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:36:21.537+0000 7ff6d0ff9640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:36:21.540 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:36:21.537+0000 7ff6da64e640 5 monclient: authenticate success, global_id 4645 2024-03-25T18:36:21.540 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:36:21.537+0000 7ff6da64e640 10 monclient: _renew_subs 2024-03-25T18:36:21.540 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:36:21.537+0000 7ff6da64e640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.115:6789/0 2024-03-25T18:36:21.540 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:36:21.537+0000 7ff6da64e640 1 -- 172.21.15.1:0/3775684608 --> v1:172.21.15.115:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff6d40ac130 con 0x7ff6d413da60 2024-03-25T18:36:21.540 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:36:21.537+0000 7ff6da64e640 10 client.4645.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T18:36:21.540 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:36:21.537+0000 7ff6da64e640 10 monclient: _renew_subs 2024-03-25T18:36:21.540 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:36:21.537+0000 7ff6da64e640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.115:6789/0 2024-03-25T18:36:21.540 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:36:21.537+0000 7ff6da64e640 1 -- 172.21.15.1:0/3775684608 --> v1:172.21.15.115:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7ff6d4139490 con 0x7ff6d413da60 2024-03-25T18:36:21.540 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:36:21.537+0000 7ff6da64e640 1 librados: init done 2024-03-25T18:36:21.541 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:36:21.539+0000 7ff6da64e640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T18:36:21.541 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:36:21.539+0000 7ff6da64e640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:36:21.541 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:36:21.539+0000 7ff6da64e640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.115:6789/0 2024-03-25T18:36:21.541 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:36:21.539+0000 7ff6da64e640 1 -- 172.21.15.1:0/3775684608 --> v1:172.21.15.115:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff6d40ac480 con 0x7ff6d413da60 2024-03-25T18:36:21.558 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:36:21.557+0000 7ff6d0ff9640 1 -- 172.21.15.1:0/3775684608 <== mon.1 v1:172.21.15.115:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7ff6c8004bf0 con 0x7ff6d413da60 2024-03-25T18:36:21.558 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:36:21.557+0000 7ff6d0ff9640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:36:21.558 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:36:21.557+0000 7ff6d0ff9640 1 -- 172.21.15.1:0/3775684608 <== mon.1 v1:172.21.15.115:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1586965025 0 0) 0x7ff6c8004f00 con 0x7ff6d413da60 2024-03-25T18:36:21.558 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:36:21.557+0000 7ff6d0ff9640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:36:21.558 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:36:21.557+0000 7ff6d0ff9640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-25T18:36:21.558 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:36:21.557+0000 7ff6d0ff9640 10 monclient: dump: 2024-03-25T18:36:21.558 INFO:teuthology.misc.health.smithi001.stderr:epoch 1 2024-03-25T18:36:21.558 INFO:teuthology.misc.health.smithi001.stderr:fsid c8ea72cf-9e99-4383-9b52-6c23d365fdfc 2024-03-25T18:36:21.558 INFO:teuthology.misc.health.smithi001.stderr:last_changed 2024-03-25T18:25:50.510519+0000 2024-03-25T18:36:21.558 INFO:teuthology.misc.health.smithi001.stderr:created 2024-03-25T18:25:50.510519+0000 2024-03-25T18:36:21.559 INFO:teuthology.misc.health.smithi001.stderr:min_mon_release 19 (squid) 2024-03-25T18:36:21.559 INFO:teuthology.misc.health.smithi001.stderr:election_strategy: 3 2024-03-25T18:36:21.559 INFO:teuthology.misc.health.smithi001.stderr:0: v1:172.21.15.1:6789/0 mon.a 2024-03-25T18:36:21.559 INFO:teuthology.misc.health.smithi001.stderr:1: v1:172.21.15.115:6789/0 mon.b 2024-03-25T18:36:21.559 INFO:teuthology.misc.health.smithi001.stderr:2: v1:172.21.15.1:6790/0 mon.c 2024-03-25T18:36:21.559 INFO:teuthology.misc.health.smithi001.stderr: 2024-03-25T18:36:21.559 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:36:21.557+0000 7ff6d27fc640 4 set_mon_vals no callback set 2024-03-25T18:36:21.560 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:36:21.558+0000 7ff6d0ff9640 1 -- 172.21.15.1:0/3775684608 <== mon.1 v1:172.21.15.115:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 21787975 0 0) 0x7ff6c801aed0 con 0x7ff6d413da60 2024-03-25T18:36:21.560 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:36:21.558+0000 7ff6d3fff640 10 monclient: get_auth_request con 0x7ff6c006df70 auth_method 0 2024-03-25T18:36:21.560 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:36:21.558+0000 7ff6d0ff9640 10 client.4645.objecter ms_handle_connect 0x7ff6c006df70 2024-03-25T18:36:21.563 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:36:21.562+0000 7ff6d0ff9640 1 -- 172.21.15.1:0/3775684608 <== mon.1 v1:172.21.15.115:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 996254230 0 0) 0x7ff6c8054930 con 0x7ff6d413da60 2024-03-25T18:36:21.563 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:36:21.562+0000 7ff6d0ff9640 10 client.4645.objecter ms_dispatch 0x7ff6d4000b90 osd_map(10..10 src has 1..10) v4 2024-03-25T18:36:21.563 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:36:21.562+0000 7ff6d0ff9640 3 client.4645.objecter handle_osd_map got epochs [10,10] > 0 2024-03-25T18:36:21.563 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:36:21.562+0000 7ff6d0ff9640 3 client.4645.objecter handle_osd_map decoding full epoch 10 2024-03-25T18:36:21.563 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:36:21.562+0000 7ff6d0ff9640 20 client.4645.objecter dump_active .. 0 homeless 2024-03-25T18:36:21.822 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:36:21.821+0000 7ff6d0ff9640 1 -- 172.21.15.1:0/3775684608 <== mon.1 v1:172.21.15.115:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7ff6c8050440 con 0x7ff6d413da60 2024-03-25T18:36:21.822 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:36:21.821+0000 7ff6d0ff9640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:36:21.822 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:36:21.821+0000 7ff6d0ff9640 10 monclient: _finish_command 1 = system:0 2024-03-25T18:36:22.031 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:36:22.030+0000 7ff6da64e640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:36:22.031 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:36:22.030+0000 7ff6da64e640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:36:22.031 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:36:22.030+0000 7ff6da64e640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.115:6789/0 2024-03-25T18:36:22.032 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:36:22.030+0000 7ff6da64e640 1 -- 172.21.15.1:0/3775684608 --> v1:172.21.15.115:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7ff6d4072d20 con 0x7ff6d413da60 2024-03-25T18:36:22.052 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:36:22.052+0000 7ff6d0ff9640 1 -- 172.21.15.1:0/3775684608 <== mon.1 v1:172.21.15.115:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 4092213348) 0x7ff6c8052370 con 0x7ff6d413da60 2024-03-25T18:36:22.052 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:36:22.052+0000 7ff6d0ff9640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:36:22.052 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:36:22.052+0000 7ff6d0ff9640 10 monclient: _finish_command 2 = system:0 2024-03-25T18:36:22.052 INFO:teuthology.misc.health.smithi001.stdout: 2024-03-25T18:36:22.053 INFO:teuthology.misc.health.smithi001.stdout:{"epoch":10,"fsid":"c8ea72cf-9e99-4383-9b52-6c23d365fdfc","created":"2024-03-25T18:26:38.856638+0000","modified":"2024-03-25T18:26:44.224208+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T18:26:44.224208+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"b9f34922-3a84-49be-8426-807c1eceaefc","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"42bf306f-55e3-4634-9874-8b4b73ac6644","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"292cb23a-18a9-4207-a7cc-4d5727feaec0","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"45e0e9ab-898d-4673-9590-b4117bc5e84a","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"831a5a09-73ab-4a61-9872-088e76bdbc66","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"88cd903d-3b55-4840-9396-6e3cfe005282","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"89db7355-bc9d-46f4-8ab0-e8d9e744e026","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"cac47bee-0914-4f08-896f-348838882866","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T18:36:22.055 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:36:22.054+0000 7ff6da64e640 10 librados: watch_flush enter 2024-03-25T18:36:22.055 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:36:22.054+0000 7ff6da64e640 10 librados: watch_flush exit 2024-03-25T18:36:22.055 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:36:22.054+0000 7ff6da64e640 20 client.4645.objecter shutdown clearing up homeless session... 2024-03-25T18:36:22.055 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:36:22.054+0000 7ff6da64e640 10 client.4645.objecter successfully canceled tick 2024-03-25T18:36:22.055 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:36:22.054+0000 7ff6da64e640 1 -- 172.21.15.1:0/3775684608 >> v1:172.21.15.115:6800/4237379176 conn(0x7ff6c006df70 legacy=0x7ff6c0070430 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:36:22.055 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:36:22.054+0000 7ff6da64e640 10 monclient: shutdown 2024-03-25T18:36:22.055 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:36:22.054+0000 7ff6da64e640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:36:22.056 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:36:22.054+0000 7ff6da64e640 1 -- 172.21.15.1:0/3775684608 >> v1:172.21.15.115:6789/0 conn(0x7ff6d413da60 legacy=0x7ff6d4138ba0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:36:22.056 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:36:22.054+0000 7ff6da64e640 1 -- 172.21.15.1:0/3775684608 shutdown_connections 2024-03-25T18:36:22.056 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:36:22.054+0000 7ff6da64e640 1 -- 172.21.15.1:0/3775684608 >> 172.21.15.1:0/3775684608 conn(0x7ff6d40b4bd0 msgr2=0x7ff6d413a610 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:36:22.056 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:36:22.054+0000 7ff6da64e640 1 -- 172.21.15.1:0/3775684608 shutdown_connections 2024-03-25T18:36:22.056 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:36:22.054+0000 7ff6da64e640 1 -- 172.21.15.1:0/3775684608 wait complete. 2024-03-25T18:36:22.056 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:36:22.054+0000 7ff6da64e640 1 librados: shutdown 2024-03-25T18:36:22.066 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T18:36:23.202 INFO:tasks.ceph.mon.a.smithi001.stderr:==00:00:10:01.761 34876== Warning: unimplemented fcntl command: 1036 2024-03-25T18:36:23.220 INFO:tasks.ceph.mon.a.smithi001.stderr:==00:00:10:01.779 34876== Warning: unimplemented fcntl command: 1036 2024-03-25T18:36:23.255 INFO:tasks.ceph.mon.a.smithi001.stderr:==00:00:10:01.813 34876== Warning: unimplemented fcntl command: 1036 2024-03-25T18:36:23.275 INFO:tasks.ceph.mon.c.smithi001.stderr:==00:00:10:01.833 34877== Warning: unimplemented fcntl command: 1036 2024-03-25T18:36:23.280 INFO:tasks.ceph.mon.c.smithi001.stderr:==00:00:10:01.839 34877== Warning: unimplemented fcntl command: 1036 2024-03-25T18:36:23.286 INFO:tasks.ceph.mon.b.smithi115.stderr:==00:00:10:01.770 34836== Warning: unimplemented fcntl command: 1036 2024-03-25T18:36:23.288 INFO:tasks.ceph.mon.b.smithi115.stderr:==00:00:10:01.772 34836== Warning: unimplemented fcntl command: 1036 2024-03-25T18:36:23.301 INFO:tasks.ceph.mon.c.smithi001.stderr:==00:00:10:01.860 34877== Warning: unimplemented fcntl command: 1036 2024-03-25T18:36:23.310 INFO:tasks.ceph.mon.b.smithi115.stderr:==00:00:10:01.794 34836== Warning: unimplemented fcntl command: 1036 2024-03-25T18:36:28.872 DEBUG:teuthology.orchestra.run.smithi001:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T18:36:28.990 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:36:28.990+0000 7f2e9ec31640 10 monclient: get_monmap_and_config 2024-03-25T18:36:28.991 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:36:28.990+0000 7f2e9ec31640 10 monclient: build_initial_monmap 2024-03-25T18:36:28.991 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:36:28.990+0000 7f2e9ec31640 10 monclient: monmap: 2024-03-25T18:36:28.991 INFO:teuthology.misc.health.smithi001.stderr:epoch 0 2024-03-25T18:36:28.991 INFO:teuthology.misc.health.smithi001.stderr:fsid c8ea72cf-9e99-4383-9b52-6c23d365fdfc 2024-03-25T18:36:28.991 INFO:teuthology.misc.health.smithi001.stderr:last_changed 2024-03-25T18:36:28.990771+0000 2024-03-25T18:36:28.991 INFO:teuthology.misc.health.smithi001.stderr:created 2024-03-25T18:36:28.990771+0000 2024-03-25T18:36:28.991 INFO:teuthology.misc.health.smithi001.stderr:min_mon_release 0 (unknown) 2024-03-25T18:36:28.991 INFO:teuthology.misc.health.smithi001.stderr:election_strategy: 3 2024-03-25T18:36:28.991 INFO:teuthology.misc.health.smithi001.stderr:0: v1:172.21.15.1:6789/0 mon.noname-a 2024-03-25T18:36:28.991 INFO:teuthology.misc.health.smithi001.stderr:1: v1:172.21.15.115:6789/0 mon.noname-c 2024-03-25T18:36:28.991 INFO:teuthology.misc.health.smithi001.stderr:2: v1:172.21.15.1:6790/0 mon.noname-b 2024-03-25T18:36:28.991 INFO:teuthology.misc.health.smithi001.stderr: 2024-03-25T18:36:28.992 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:36:28.991+0000 7f2e9ec31640 1 Processor -- start 2024-03-25T18:36:28.992 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:36:28.991+0000 7f2e9ec31640 1 -- start start 2024-03-25T18:36:28.992 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:36:28.991+0000 7f2e9ec31640 10 monclient: init 2024-03-25T18:36:28.993 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:36:28.992+0000 7f2e9ec31640 10 monclient: _reopen_session rank -1 2024-03-25T18:36:28.993 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:36:28.992+0000 7f2e9ec31640 10 monclient: _add_conns ranks=[1,0,2] 2024-03-25T18:36:28.993 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:36:28.992+0000 7f2e9ec31640 10 monclient(hunting): picked mon.noname-c con 0x7f2e980b4ff0 addr v1:172.21.15.115:6789/0 2024-03-25T18:36:28.993 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:36:28.992+0000 7f2e9ec31640 10 monclient(hunting): picked mon.noname-a con 0x7f2e980b4670 addr v1:172.21.15.1:6789/0 2024-03-25T18:36:28.993 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:36:28.992+0000 7f2e9ec31640 10 monclient(hunting): picked mon.noname-b con 0x7f2e9813c480 addr v1:172.21.15.1:6790/0 2024-03-25T18:36:28.993 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:36:28.992+0000 7f2e9ec31640 1 -- --> v1:172.21.15.1:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2e980ab0e0 con 0x7f2e980b4670 2024-03-25T18:36:28.993 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:36:28.992+0000 7f2e9ec31640 1 -- --> v1:172.21.15.115:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2e980ab3c0 con 0x7f2e980b4ff0 2024-03-25T18:36:28.994 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:36:28.992+0000 7f2e9ec31640 1 -- --> v1:172.21.15.1:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2e980a9f30 con 0x7f2e9813c480 2024-03-25T18:36:28.994 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:36:28.992+0000 7f2e9ec31640 10 monclient(hunting): _renew_subs 2024-03-25T18:36:28.994 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:36:28.992+0000 7f2e9ec31640 10 monclient(hunting): authenticate will time out at 8069.461973s 2024-03-25T18:36:29.006 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:36:29.006+0000 7f2e97fff640 1 --1- >> v1:172.21.15.1:6789/0 conn(0x7f2e980b4670 0x7f2e981425c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.1:6789/0 says I am v1:172.21.15.1:50098/0 (socket says 172.21.15.1:50098) 2024-03-25T18:36:29.006 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:36:29.006+0000 7f2e97fff640 1 -- 172.21.15.1:0/2898603635 learned_addr learned my addr 172.21.15.1:0/2898603635 (peer_addr_for_me v1:172.21.15.1:0/0) 2024-03-25T18:36:29.020 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:36:29.019+0000 7f2e977fe640 1 -- 172.21.15.1:0/2898603635 <== mon.0 v1:172.21.15.1:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 395305852 0 0) 0x7f2e980ab0e0 con 0x7f2e980b4670 2024-03-25T18:36:29.020 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:36:29.019+0000 7f2e977fe640 10 monclient(hunting): _negotiate 2024-03-25T18:36:29.020 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:36:29.019+0000 7f2e977fe640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:36:29.020 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:36:29.019+0000 7f2e977fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:36:29.020 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:36:29.019+0000 7f2e977fe640 10 monclient(hunting): my global_id is 4635 2024-03-25T18:36:29.020 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:36:29.019+0000 7f2e977fe640 1 -- 172.21.15.1:0/2898603635 --> v1:172.21.15.1:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f2e88003520 con 0x7f2e980b4670 2024-03-25T18:36:29.029 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:36:29.029+0000 7f2e977fe640 1 -- 172.21.15.1:0/2898603635 <== mon.0 v1:172.21.15.1:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3294811924 0 0) 0x7f2e88003520 con 0x7f2e980b4670 2024-03-25T18:36:29.030 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:36:29.029+0000 7f2e977fe640 1 -- 172.21.15.1:0/2898603635 >> v1:172.21.15.1:6790/0 conn(0x7f2e9813c480 legacy=0x7f2e9814dc00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:36:29.030 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:36:29.029+0000 7f2e977fe640 1 -- 172.21.15.1:0/2898603635 >> v1:172.21.15.115:6789/0 conn(0x7f2e980b4ff0 legacy=0x7f2e9813bd70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:36:29.030 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:36:29.029+0000 7f2e977fe640 10 monclient: _finish_hunting 0 2024-03-25T18:36:29.030 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:36:29.029+0000 7f2e977fe640 1 monclient: found mon.noname-a 2024-03-25T18:36:29.030 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:36:29.029+0000 7f2e977fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:36:29.030 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:36:29.029+0000 7f2e977fe640 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.1:6789/0 2024-03-25T18:36:29.030 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:36:29.029+0000 7f2e977fe640 1 -- 172.21.15.1:0/2898603635 --> v1:172.21.15.1:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2e980a9b70 con 0x7f2e980b4670 2024-03-25T18:36:29.030 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:36:29.029+0000 7f2e977fe640 10 monclient: _finish_auth 0 2024-03-25T18:36:29.030 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:36:29.029+0000 7f2e977fe640 10 monclient: _check_auth_tickets 2024-03-25T18:36:29.030 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:36:29.029+0000 7f2e977fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:36:29.030 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:36:29.029+0000 7f2e9ec31640 5 monclient: authenticate success, global_id 4635 2024-03-25T18:36:29.030 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:36:29.029+0000 7f2e9ec31640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T18:36:29.031 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:36:29.031+0000 7f2e977fe640 1 -- 172.21.15.1:0/2898603635 <== mon.0 v1:172.21.15.1:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1586965025 0 0) 0x7f2e84001690 con 0x7f2e980b4670 2024-03-25T18:36:29.031 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:36:29.031+0000 7f2e977fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:36:29.031 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:36:29.031+0000 7f2e977fe640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-03-25T18:36:29.031 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:36:29.031+0000 7f2e977fe640 10 monclient: dump: 2024-03-25T18:36:29.031 INFO:teuthology.misc.health.smithi001.stderr:epoch 1 2024-03-25T18:36:29.031 INFO:teuthology.misc.health.smithi001.stderr:fsid c8ea72cf-9e99-4383-9b52-6c23d365fdfc 2024-03-25T18:36:29.031 INFO:teuthology.misc.health.smithi001.stderr:last_changed 2024-03-25T18:25:50.510519+0000 2024-03-25T18:36:29.031 INFO:teuthology.misc.health.smithi001.stderr:created 2024-03-25T18:25:50.510519+0000 2024-03-25T18:36:29.032 INFO:teuthology.misc.health.smithi001.stderr:min_mon_release 19 (squid) 2024-03-25T18:36:29.032 INFO:teuthology.misc.health.smithi001.stderr:election_strategy: 3 2024-03-25T18:36:29.032 INFO:teuthology.misc.health.smithi001.stderr:0: v1:172.21.15.1:6789/0 mon.a 2024-03-25T18:36:29.032 INFO:teuthology.misc.health.smithi001.stderr:1: v1:172.21.15.115:6789/0 mon.b 2024-03-25T18:36:29.032 INFO:teuthology.misc.health.smithi001.stderr:2: v1:172.21.15.1:6790/0 mon.c 2024-03-25T18:36:29.032 INFO:teuthology.misc.health.smithi001.stderr: 2024-03-25T18:36:29.032 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:36:29.031+0000 7f2e9ec31640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T18:36:29.046 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:36:29.046+0000 7f2e977fe640 1 -- 172.21.15.1:0/2898603635 <== mon.0 v1:172.21.15.1:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f2e84001920 con 0x7f2e980b4670 2024-03-25T18:36:29.047 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:36:29.046+0000 7f2e977fe640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:36:29.047 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:36:29.046+0000 7f2e977fe640 1 -- 172.21.15.1:0/2898603635 <== mon.0 v1:172.21.15.1:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1586965025 0 0) 0x7f2e84001c30 con 0x7f2e980b4670 2024-03-25T18:36:29.047 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:36:29.046+0000 7f2e9ec31640 10 monclient: get_monmap_and_config success 2024-03-25T18:36:29.047 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:36:29.046+0000 7f2e9ec31640 4 set_mon_vals no callback set 2024-03-25T18:36:29.047 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:36:29.046+0000 7f2e977fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:36:29.047 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:36:29.046+0000 7f2e977fe640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-25T18:36:29.047 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:36:29.046+0000 7f2e977fe640 10 monclient: dump: 2024-03-25T18:36:29.047 INFO:teuthology.misc.health.smithi001.stderr:epoch 1 2024-03-25T18:36:29.047 INFO:teuthology.misc.health.smithi001.stderr:fsid c8ea72cf-9e99-4383-9b52-6c23d365fdfc 2024-03-25T18:36:29.047 INFO:teuthology.misc.health.smithi001.stderr:last_changed 2024-03-25T18:25:50.510519+0000 2024-03-25T18:36:29.047 INFO:teuthology.misc.health.smithi001.stderr:created 2024-03-25T18:25:50.510519+0000 2024-03-25T18:36:29.047 INFO:teuthology.misc.health.smithi001.stderr:min_mon_release 19 (squid) 2024-03-25T18:36:29.047 INFO:teuthology.misc.health.smithi001.stderr:election_strategy: 3 2024-03-25T18:36:29.047 INFO:teuthology.misc.health.smithi001.stderr:0: v1:172.21.15.1:6789/0 mon.a 2024-03-25T18:36:29.048 INFO:teuthology.misc.health.smithi001.stderr:1: v1:172.21.15.115:6789/0 mon.b 2024-03-25T18:36:29.048 INFO:teuthology.misc.health.smithi001.stderr:2: v1:172.21.15.1:6790/0 mon.c 2024-03-25T18:36:29.048 INFO:teuthology.misc.health.smithi001.stderr: 2024-03-25T18:36:29.049 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:36:29.047+0000 7f2e9ec31640 10 monclient: shutdown 2024-03-25T18:36:29.049 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:36:29.047+0000 7f2e9ec31640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:36:29.049 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:36:29.047+0000 7f2e9ec31640 1 -- 172.21.15.1:0/2898603635 >> v1:172.21.15.1:6789/0 conn(0x7f2e980b4670 legacy=0x7f2e981425c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:36:29.049 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:36:29.047+0000 7f2e9ec31640 1 -- 172.21.15.1:0/2898603635 shutdown_connections 2024-03-25T18:36:29.049 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:36:29.047+0000 7f2e9ec31640 1 -- 172.21.15.1:0/2898603635 >> 172.21.15.1:0/2898603635 conn(0x7f2e980617a0 msgr2=0x7f2e98061bb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:36:29.050 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:36:29.047+0000 7f2e9ec31640 1 -- 172.21.15.1:0/2898603635 shutdown_connections 2024-03-25T18:36:29.050 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:36:29.047+0000 7f2e9ec31640 1 -- 172.21.15.1:0/2898603635 wait complete. 2024-03-25T18:36:29.050 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:36:29.047+0000 7f2e9ec31640 10 monclient: build_initial_monmap 2024-03-25T18:36:29.050 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:36:29.047+0000 7f2e9ec31640 10 monclient: monmap: 2024-03-25T18:36:29.050 INFO:teuthology.misc.health.smithi001.stderr:epoch 0 2024-03-25T18:36:29.050 INFO:teuthology.misc.health.smithi001.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T18:36:29.050 INFO:teuthology.misc.health.smithi001.stderr:last_changed 0.000000 2024-03-25T18:36:29.050 INFO:teuthology.misc.health.smithi001.stderr:created 0.000000 2024-03-25T18:36:29.050 INFO:teuthology.misc.health.smithi001.stderr:min_mon_release 0 (unknown) 2024-03-25T18:36:29.050 INFO:teuthology.misc.health.smithi001.stderr:election_strategy: 1 2024-03-25T18:36:29.050 INFO:teuthology.misc.health.smithi001.stderr:0: v1:172.21.15.1:6789/0 mon.noname-a 2024-03-25T18:36:29.050 INFO:teuthology.misc.health.smithi001.stderr:1: v1:172.21.15.115:6789/0 mon.noname-b 2024-03-25T18:36:29.050 INFO:teuthology.misc.health.smithi001.stderr:2: v1:172.21.15.1:6790/0 mon.noname-c 2024-03-25T18:36:29.050 INFO:teuthology.misc.health.smithi001.stderr: 2024-03-25T18:36:29.050 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:36:29.048+0000 7f2e9ec31640 1 librados: starting msgr at 2024-03-25T18:36:29.051 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:36:29.048+0000 7f2e9ec31640 1 librados: starting objecter 2024-03-25T18:36:29.051 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:36:29.048+0000 7f2e9ec31640 1 Processor -- start 2024-03-25T18:36:29.051 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:36:29.048+0000 7f2e9ec31640 1 -- start start 2024-03-25T18:36:29.051 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:36:29.048+0000 7f2e9ec31640 1 librados: setting wanted keys 2024-03-25T18:36:29.051 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:36:29.048+0000 7f2e9ec31640 1 librados: calling monclient init 2024-03-25T18:36:29.051 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:36:29.048+0000 7f2e9ec31640 10 monclient: init 2024-03-25T18:36:29.052 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:36:29.049+0000 7f2e9ec31640 10 monclient: _reopen_session rank -1 2024-03-25T18:36:29.052 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:36:29.049+0000 7f2e9ec31640 10 monclient: _add_conns ranks=[0,2,1] 2024-03-25T18:36:29.052 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:36:29.049+0000 7f2e9ec31640 10 monclient(hunting): picked mon.noname-a con 0x7f2e980b4670 addr v1:172.21.15.1:6789/0 2024-03-25T18:36:29.052 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:36:29.049+0000 7f2e9ec31640 10 monclient(hunting): picked mon.noname-c con 0x7f2e980b4ff0 addr v1:172.21.15.1:6790/0 2024-03-25T18:36:29.052 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:36:29.049+0000 7f2e9ec31640 10 monclient(hunting): picked mon.noname-b con 0x7f2e9813c480 addr v1:172.21.15.115:6789/0 2024-03-25T18:36:29.052 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:36:29.049+0000 7f2e9ec31640 1 -- --> v1:172.21.15.1:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2e980b0680 con 0x7f2e980b4670 2024-03-25T18:36:29.052 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:36:29.050+0000 7f2e9ec31640 1 -- --> v1:172.21.15.115:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2e980b08d0 con 0x7f2e9813c480 2024-03-25T18:36:29.052 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:36:29.050+0000 7f2e9ec31640 1 -- --> v1:172.21.15.1:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2e980b0bb0 con 0x7f2e980b4ff0 2024-03-25T18:36:29.053 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:36:29.050+0000 7f2e9ec31640 10 monclient(hunting): _renew_subs 2024-03-25T18:36:29.053 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:36:29.050+0000 7f2e9ec31640 10 monclient(hunting): authenticate will time out at 8069.520057s 2024-03-25T18:36:29.070 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:36:29.069+0000 7f2e9d1ce640 1 --1- >> v1:172.21.15.115:6789/0 conn(0x7f2e9813c480 0x7f2e981e2380 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.115:6789/0 says I am v1:172.21.15.1:38210/0 (socket says 172.21.15.1:38210) 2024-03-25T18:36:29.070 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:36:29.069+0000 7f2e9d1ce640 1 -- 172.21.15.1:0/4252690076 learned_addr learned my addr 172.21.15.1:0/4252690076 (peer_addr_for_me v1:172.21.15.1:0/0) 2024-03-25T18:36:29.075 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:36:29.074+0000 7f2e957fa640 10 client.?.objecter ms_handle_connect 0x7f2e9813c480 2024-03-25T18:36:29.075 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:36:29.074+0000 7f2e957fa640 10 client.?.objecter resend_mon_ops 2024-03-25T18:36:29.078 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:36:29.077+0000 7f2e957fa640 10 client.?.objecter ms_handle_connect 0x7f2e980b4ff0 2024-03-25T18:36:29.078 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:36:29.077+0000 7f2e957fa640 10 client.?.objecter resend_mon_ops 2024-03-25T18:36:29.081 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:36:29.081+0000 7f2e957fa640 10 client.?.objecter ms_handle_connect 0x7f2e980b4670 2024-03-25T18:36:29.081 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:36:29.081+0000 7f2e957fa640 10 client.?.objecter resend_mon_ops 2024-03-25T18:36:29.085 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:36:29.085+0000 7f2e957fa640 1 -- 172.21.15.1:0/4252690076 <== mon.1 v1:172.21.15.115:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 95036538 0 0) 0x7f2e980b08d0 con 0x7f2e9813c480 2024-03-25T18:36:29.085 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:36:29.085+0000 7f2e957fa640 10 monclient(hunting): _negotiate 2024-03-25T18:36:29.085 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:36:29.085+0000 7f2e957fa640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:36:29.085 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:36:29.085+0000 7f2e957fa640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:36:29.085 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:36:29.085+0000 7f2e957fa640 10 monclient(hunting): my global_id is 4651 2024-03-25T18:36:29.085 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:36:29.085+0000 7f2e957fa640 1 -- 172.21.15.1:0/4252690076 --> v1:172.21.15.115:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f2e7c004eb0 con 0x7f2e9813c480 2024-03-25T18:36:29.089 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:36:29.088+0000 7f2e957fa640 1 -- 172.21.15.1:0/4252690076 <== mon.2 v1:172.21.15.1:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2200966665 0 0) 0x7f2e980b0bb0 con 0x7f2e980b4ff0 2024-03-25T18:36:29.089 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:36:29.088+0000 7f2e957fa640 10 monclient(hunting): _negotiate 2024-03-25T18:36:29.089 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:36:29.088+0000 7f2e957fa640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:36:29.089 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:36:29.088+0000 7f2e957fa640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:36:29.089 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:36:29.088+0000 7f2e957fa640 10 monclient(hunting): my global_id is 4649 2024-03-25T18:36:29.089 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:36:29.088+0000 7f2e957fa640 1 -- 172.21.15.1:0/4252690076 --> v1:172.21.15.1:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f2e980b08d0 con 0x7f2e980b4ff0 2024-03-25T18:36:29.101 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:36:29.100+0000 7f2e957fa640 1 -- 172.21.15.1:0/4252690076 <== mon.0 v1:172.21.15.1:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1288742208 0 0) 0x7f2e980b0680 con 0x7f2e980b4670 2024-03-25T18:36:29.101 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:36:29.100+0000 7f2e957fa640 10 monclient(hunting): _negotiate 2024-03-25T18:36:29.101 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:36:29.100+0000 7f2e957fa640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:36:29.101 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:36:29.100+0000 7f2e957fa640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:36:29.101 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:36:29.100+0000 7f2e957fa640 10 monclient(hunting): my global_id is 4638 2024-03-25T18:36:29.101 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:36:29.100+0000 7f2e957fa640 1 -- 172.21.15.1:0/4252690076 --> v1:172.21.15.1:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f2e980b0bb0 con 0x7f2e980b4670 2024-03-25T18:36:29.126 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:36:29.125+0000 7f2e957fa640 1 -- 172.21.15.1:0/4252690076 <== mon.1 v1:172.21.15.115:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2574119051 0 0) 0x7f2e7c004eb0 con 0x7f2e9813c480 2024-03-25T18:36:29.126 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:36:29.125+0000 7f2e957fa640 1 -- 172.21.15.1:0/4252690076 --> v1:172.21.15.115:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f2e980b0680 con 0x7f2e9813c480 2024-03-25T18:36:29.126 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:36:29.125+0000 7f2e957fa640 1 -- 172.21.15.1:0/4252690076 <== mon.1 v1:172.21.15.115:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1586965025 0 0) 0x7f2e90001c90 con 0x7f2e9813c480 2024-03-25T18:36:29.126 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:36:29.125+0000 7f2e957fa640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:36:29.126 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:36:29.125+0000 7f2e957fa640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-03-25T18:36:29.127 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:36:29.125+0000 7f2e957fa640 10 monclient(hunting): dump: 2024-03-25T18:36:29.127 INFO:teuthology.misc.health.smithi001.stderr:epoch 1 2024-03-25T18:36:29.127 INFO:teuthology.misc.health.smithi001.stderr:fsid c8ea72cf-9e99-4383-9b52-6c23d365fdfc 2024-03-25T18:36:29.127 INFO:teuthology.misc.health.smithi001.stderr:last_changed 2024-03-25T18:25:50.510519+0000 2024-03-25T18:36:29.127 INFO:teuthology.misc.health.smithi001.stderr:created 2024-03-25T18:25:50.510519+0000 2024-03-25T18:36:29.127 INFO:teuthology.misc.health.smithi001.stderr:min_mon_release 19 (squid) 2024-03-25T18:36:29.127 INFO:teuthology.misc.health.smithi001.stderr:election_strategy: 3 2024-03-25T18:36:29.127 INFO:teuthology.misc.health.smithi001.stderr:0: v1:172.21.15.1:6789/0 mon.a 2024-03-25T18:36:29.127 INFO:teuthology.misc.health.smithi001.stderr:1: v1:172.21.15.115:6789/0 mon.b 2024-03-25T18:36:29.127 INFO:teuthology.misc.health.smithi001.stderr:2: v1:172.21.15.1:6790/0 mon.c 2024-03-25T18:36:29.127 INFO:teuthology.misc.health.smithi001.stderr: 2024-03-25T18:36:29.127 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:36:29.125+0000 7f2e957fa640 10 monclient(hunting): _finish_auth 0 2024-03-25T18:36:29.133 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:36:29.132+0000 7f2e957fa640 1 -- 172.21.15.1:0/4252690076 <== mon.2 v1:172.21.15.1:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2771043797 0 0) 0x7f2e980b08d0 con 0x7f2e980b4ff0 2024-03-25T18:36:29.133 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:36:29.132+0000 7f2e957fa640 1 -- 172.21.15.1:0/4252690076 --> v1:172.21.15.1:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f2e7c004eb0 con 0x7f2e980b4ff0 2024-03-25T18:36:29.133 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:36:29.132+0000 7f2e957fa640 1 -- 172.21.15.1:0/4252690076 <== mon.2 v1:172.21.15.1:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1586965025 0 0) 0x7f2e840014b0 con 0x7f2e980b4ff0 2024-03-25T18:36:29.133 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:36:29.132+0000 7f2e957fa640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:36:29.133 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:36:29.132+0000 7f2e957fa640 10 monclient(hunting): got monmap 1 from mon.c (according to old e1) 2024-03-25T18:36:29.133 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:36:29.132+0000 7f2e957fa640 10 monclient(hunting): dump: 2024-03-25T18:36:29.133 INFO:teuthology.misc.health.smithi001.stderr:epoch 1 2024-03-25T18:36:29.133 INFO:teuthology.misc.health.smithi001.stderr:fsid c8ea72cf-9e99-4383-9b52-6c23d365fdfc 2024-03-25T18:36:29.133 INFO:teuthology.misc.health.smithi001.stderr:last_changed 2024-03-25T18:25:50.510519+0000 2024-03-25T18:36:29.133 INFO:teuthology.misc.health.smithi001.stderr:created 2024-03-25T18:25:50.510519+0000 2024-03-25T18:36:29.133 INFO:teuthology.misc.health.smithi001.stderr:min_mon_release 19 (squid) 2024-03-25T18:36:29.133 INFO:teuthology.misc.health.smithi001.stderr:election_strategy: 3 2024-03-25T18:36:29.134 INFO:teuthology.misc.health.smithi001.stderr:0: v1:172.21.15.1:6789/0 mon.a 2024-03-25T18:36:29.134 INFO:teuthology.misc.health.smithi001.stderr:1: v1:172.21.15.115:6789/0 mon.b 2024-03-25T18:36:29.134 INFO:teuthology.misc.health.smithi001.stderr:2: v1:172.21.15.1:6790/0 mon.c 2024-03-25T18:36:29.134 INFO:teuthology.misc.health.smithi001.stderr: 2024-03-25T18:36:29.134 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:36:29.132+0000 7f2e957fa640 1 -- 172.21.15.1:0/4252690076 <== mon.0 v1:172.21.15.1:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1438975456 0 0) 0x7f2e980b0bb0 con 0x7f2e980b4670 2024-03-25T18:36:29.134 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:36:29.132+0000 7f2e957fa640 1 -- 172.21.15.1:0/4252690076 --> v1:172.21.15.1:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f2e980b08d0 con 0x7f2e980b4670 2024-03-25T18:36:29.134 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:36:29.132+0000 7f2e957fa640 1 -- 172.21.15.1:0/4252690076 <== mon.0 v1:172.21.15.1:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1586965025 0 0) 0x7f2e8c001860 con 0x7f2e980b4670 2024-03-25T18:36:29.134 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:36:29.132+0000 7f2e957fa640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:36:29.134 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:36:29.132+0000 7f2e957fa640 10 monclient(hunting): got monmap 1 from mon.a (according to old e1) 2024-03-25T18:36:29.134 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:36:29.132+0000 7f2e957fa640 10 monclient(hunting): dump: 2024-03-25T18:36:29.134 INFO:teuthology.misc.health.smithi001.stderr:epoch 1 2024-03-25T18:36:29.134 INFO:teuthology.misc.health.smithi001.stderr:fsid c8ea72cf-9e99-4383-9b52-6c23d365fdfc 2024-03-25T18:36:29.134 INFO:teuthology.misc.health.smithi001.stderr:last_changed 2024-03-25T18:25:50.510519+0000 2024-03-25T18:36:29.134 INFO:teuthology.misc.health.smithi001.stderr:created 2024-03-25T18:25:50.510519+0000 2024-03-25T18:36:29.134 INFO:teuthology.misc.health.smithi001.stderr:min_mon_release 19 (squid) 2024-03-25T18:36:29.135 INFO:teuthology.misc.health.smithi001.stderr:election_strategy: 3 2024-03-25T18:36:29.135 INFO:teuthology.misc.health.smithi001.stderr:0: v1:172.21.15.1:6789/0 mon.a 2024-03-25T18:36:29.135 INFO:teuthology.misc.health.smithi001.stderr:1: v1:172.21.15.115:6789/0 mon.b 2024-03-25T18:36:29.135 INFO:teuthology.misc.health.smithi001.stderr:2: v1:172.21.15.1:6790/0 mon.c 2024-03-25T18:36:29.135 INFO:teuthology.misc.health.smithi001.stderr: 2024-03-25T18:36:29.157 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:36:29.156+0000 7f2e957fa640 1 -- 172.21.15.1:0/4252690076 <== mon.2 v1:172.21.15.1:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 1546274189 0 0) 0x7f2e7c004eb0 con 0x7f2e980b4ff0 2024-03-25T18:36:29.157 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:36:29.156+0000 7f2e957fa640 1 -- 172.21.15.1:0/4252690076 >> v1:172.21.15.115:6789/0 conn(0x7f2e9813c480 legacy=0x7f2e981e2380 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:36:29.158 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:36:29.156+0000 7f2e957fa640 1 -- 172.21.15.1:0/4252690076 >> v1:172.21.15.1:6789/0 conn(0x7f2e980b4670 legacy=0x7f2e980ae7c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:36:29.158 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:36:29.156+0000 7f2e957fa640 10 monclient: _finish_hunting 0 2024-03-25T18:36:29.158 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:36:29.156+0000 7f2e957fa640 1 monclient: found mon.c 2024-03-25T18:36:29.158 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:36:29.156+0000 7f2e957fa640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:36:29.158 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:36:29.156+0000 7f2e957fa640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.1:6790/0 2024-03-25T18:36:29.158 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:36:29.156+0000 7f2e957fa640 1 -- 172.21.15.1:0/4252690076 --> v1:172.21.15.1:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2e980b1520 con 0x7f2e980b4ff0 2024-03-25T18:36:29.158 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:36:29.156+0000 7f2e957fa640 10 monclient: _finish_auth 0 2024-03-25T18:36:29.158 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:36:29.156+0000 7f2e957fa640 10 monclient: _check_auth_tickets 2024-03-25T18:36:29.158 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:36:29.156+0000 7f2e957fa640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:36:29.158 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:36:29.156+0000 7f2e9ec31640 5 monclient: authenticate success, global_id 4649 2024-03-25T18:36:29.158 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:36:29.156+0000 7f2e9ec31640 10 monclient: _renew_subs 2024-03-25T18:36:29.158 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:36:29.156+0000 7f2e9ec31640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.1:6790/0 2024-03-25T18:36:29.158 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:36:29.156+0000 7f2e9ec31640 1 -- 172.21.15.1:0/4252690076 --> v1:172.21.15.1:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2e980b0fd0 con 0x7f2e980b4ff0 2024-03-25T18:36:29.159 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:36:29.156+0000 7f2e9ec31640 10 client.4649.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T18:36:29.159 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:36:29.156+0000 7f2e9ec31640 10 monclient: _renew_subs 2024-03-25T18:36:29.159 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:36:29.156+0000 7f2e9ec31640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.1:6790/0 2024-03-25T18:36:29.159 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:36:29.156+0000 7f2e9ec31640 1 -- 172.21.15.1:0/4252690076 --> v1:172.21.15.1:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f2e980af870 con 0x7f2e980b4ff0 2024-03-25T18:36:29.159 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:36:29.156+0000 7f2e9ec31640 1 librados: init done 2024-03-25T18:36:29.159 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:36:29.157+0000 7f2e9ec31640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T18:36:29.160 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:36:29.157+0000 7f2e9ec31640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:36:29.160 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:36:29.157+0000 7f2e9ec31640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.1:6790/0 2024-03-25T18:36:29.160 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:36:29.157+0000 7f2e9ec31640 1 -- 172.21.15.1:0/4252690076 --> v1:172.21.15.1:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f2e980b1760 con 0x7f2e980b4ff0 2024-03-25T18:36:29.510 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:36:29.509+0000 7f2e957fa640 1 -- 172.21.15.1:0/4252690076 <== mon.2 v1:172.21.15.1:6790/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f2e840016b0 con 0x7f2e980b4ff0 2024-03-25T18:36:29.511 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:36:29.509+0000 7f2e957fa640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:36:29.511 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:36:29.509+0000 7f2e957fa640 1 -- 172.21.15.1:0/4252690076 <== mon.2 v1:172.21.15.1:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1586965025 0 0) 0x7f2e84004bf0 con 0x7f2e980b4ff0 2024-03-25T18:36:29.511 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:36:29.509+0000 7f2e957fa640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:36:29.511 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:36:29.510+0000 7f2e957fa640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-25T18:36:29.511 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:36:29.510+0000 7f2e957fa640 10 monclient: dump: 2024-03-25T18:36:29.511 INFO:teuthology.misc.health.smithi001.stderr:epoch 1 2024-03-25T18:36:29.511 INFO:teuthology.misc.health.smithi001.stderr:fsid c8ea72cf-9e99-4383-9b52-6c23d365fdfc 2024-03-25T18:36:29.511 INFO:teuthology.misc.health.smithi001.stderr:last_changed 2024-03-25T18:25:50.510519+0000 2024-03-25T18:36:29.511 INFO:teuthology.misc.health.smithi001.stderr:created 2024-03-25T18:25:50.510519+0000 2024-03-25T18:36:29.511 INFO:teuthology.misc.health.smithi001.stderr:min_mon_release 19 (squid) 2024-03-25T18:36:29.511 INFO:teuthology.misc.health.smithi001.stderr:election_strategy: 3 2024-03-25T18:36:29.511 INFO:teuthology.misc.health.smithi001.stderr:0: v1:172.21.15.1:6789/0 mon.a 2024-03-25T18:36:29.512 INFO:teuthology.misc.health.smithi001.stderr:1: v1:172.21.15.115:6789/0 mon.b 2024-03-25T18:36:29.512 INFO:teuthology.misc.health.smithi001.stderr:2: v1:172.21.15.1:6790/0 mon.c 2024-03-25T18:36:29.512 INFO:teuthology.misc.health.smithi001.stderr: 2024-03-25T18:36:29.512 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:36:29.510+0000 7f2e96ffd640 4 set_mon_vals no callback set 2024-03-25T18:36:29.512 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:36:29.510+0000 7f2e957fa640 1 -- 172.21.15.1:0/4252690076 <== mon.2 v1:172.21.15.1:6790/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 21787975 0 0) 0x7f2e8401abc0 con 0x7f2e980b4ff0 2024-03-25T18:36:29.512 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:36:29.511+0000 7f2e9c9cd640 10 monclient: get_auth_request con 0x7f2e7c06e160 auth_method 0 2024-03-25T18:36:29.512 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:36:29.511+0000 7f2e957fa640 10 client.4649.objecter ms_handle_connect 0x7f2e7c06e160 2024-03-25T18:36:29.519 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:36:29.518+0000 7f2e957fa640 1 -- 172.21.15.1:0/4252690076 <== mon.2 v1:172.21.15.1:6790/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 996254230 0 0) 0x7f2e84054630 con 0x7f2e980b4ff0 2024-03-25T18:36:29.519 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:36:29.518+0000 7f2e957fa640 10 client.4649.objecter ms_dispatch 0x7f2e98000b90 osd_map(10..10 src has 1..10) v4 2024-03-25T18:36:29.519 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:36:29.518+0000 7f2e957fa640 3 client.4649.objecter handle_osd_map got epochs [10,10] > 0 2024-03-25T18:36:29.519 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:36:29.518+0000 7f2e957fa640 3 client.4649.objecter handle_osd_map decoding full epoch 10 2024-03-25T18:36:29.519 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:36:29.518+0000 7f2e957fa640 20 client.4649.objecter dump_active .. 0 homeless 2024-03-25T18:36:29.519 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:36:29.518+0000 7f2e957fa640 1 -- 172.21.15.1:0/4252690076 <== mon.2 v1:172.21.15.1:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f2e84050140 con 0x7f2e980b4ff0 2024-03-25T18:36:29.519 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:36:29.518+0000 7f2e957fa640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:36:29.519 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:36:29.518+0000 7f2e957fa640 10 monclient: _finish_command 1 = system:0 2024-03-25T18:36:29.771 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:36:29.770+0000 7f2e9ec31640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:36:29.771 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:36:29.770+0000 7f2e9ec31640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:36:29.771 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:36:29.770+0000 7f2e9ec31640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.1:6790/0 2024-03-25T18:36:29.771 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:36:29.770+0000 7f2e9ec31640 1 -- 172.21.15.1:0/4252690076 --> v1:172.21.15.1:6790/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f2e980688c0 con 0x7f2e980b4ff0 2024-03-25T18:36:29.821 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:36:29.820+0000 7f2e957fa640 1 -- 172.21.15.1:0/4252690076 <== mon.2 v1:172.21.15.1:6790/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 4092213348) 0x7f2e84052070 con 0x7f2e980b4ff0 2024-03-25T18:36:29.821 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:36:29.820+0000 7f2e957fa640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:36:29.821 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:36:29.820+0000 7f2e957fa640 10 monclient: _finish_command 2 = system:0 2024-03-25T18:36:29.821 INFO:teuthology.misc.health.smithi001.stdout: 2024-03-25T18:36:29.821 INFO:teuthology.misc.health.smithi001.stdout:{"epoch":10,"fsid":"c8ea72cf-9e99-4383-9b52-6c23d365fdfc","created":"2024-03-25T18:26:38.856638+0000","modified":"2024-03-25T18:26:44.224208+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T18:26:44.224208+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"b9f34922-3a84-49be-8426-807c1eceaefc","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"42bf306f-55e3-4634-9874-8b4b73ac6644","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"292cb23a-18a9-4207-a7cc-4d5727feaec0","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"45e0e9ab-898d-4673-9590-b4117bc5e84a","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"831a5a09-73ab-4a61-9872-088e76bdbc66","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"88cd903d-3b55-4840-9396-6e3cfe005282","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"89db7355-bc9d-46f4-8ab0-e8d9e744e026","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"cac47bee-0914-4f08-896f-348838882866","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T18:36:29.824 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:36:29.823+0000 7f2e9ec31640 10 librados: watch_flush enter 2024-03-25T18:36:29.824 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:36:29.823+0000 7f2e9ec31640 10 librados: watch_flush exit 2024-03-25T18:36:29.824 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:36:29.823+0000 7f2e9ec31640 20 client.4649.objecter shutdown clearing up homeless session... 2024-03-25T18:36:29.824 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:36:29.823+0000 7f2e9ec31640 10 client.4649.objecter successfully canceled tick 2024-03-25T18:36:29.824 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:36:29.823+0000 7f2e9ec31640 1 -- 172.21.15.1:0/4252690076 >> v1:172.21.15.115:6800/4237379176 conn(0x7f2e7c06e160 legacy=0x7f2e7c070620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:36:29.825 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:36:29.823+0000 7f2e9ec31640 10 monclient: shutdown 2024-03-25T18:36:29.825 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:36:29.823+0000 7f2e9ec31640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:36:29.825 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:36:29.823+0000 7f2e9ec31640 1 -- 172.21.15.1:0/4252690076 >> v1:172.21.15.1:6790/0 conn(0x7f2e980b4ff0 legacy=0x7f2e980aef80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:36:29.825 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:36:29.823+0000 7f2e9ec31640 1 -- 172.21.15.1:0/4252690076 shutdown_connections 2024-03-25T18:36:29.825 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:36:29.823+0000 7f2e9ec31640 1 -- 172.21.15.1:0/4252690076 >> 172.21.15.1:0/4252690076 conn(0x7f2e980617a0 msgr2=0x7f2e981455d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:36:29.825 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:36:29.823+0000 7f2e9ec31640 1 -- 172.21.15.1:0/4252690076 shutdown_connections 2024-03-25T18:36:29.825 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:36:29.823+0000 7f2e9ec31640 1 -- 172.21.15.1:0/4252690076 wait complete. 2024-03-25T18:36:29.825 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:36:29.823+0000 7f2e9ec31640 1 librados: shutdown 2024-03-25T18:36:29.834 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T18:36:36.639 DEBUG:teuthology.orchestra.run.smithi001:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T18:36:36.754 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:36:36.753+0000 7fa7d2be8640 10 monclient: get_monmap_and_config 2024-03-25T18:36:36.754 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:36:36.753+0000 7fa7d2be8640 10 monclient: build_initial_monmap 2024-03-25T18:36:36.754 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:36:36.753+0000 7fa7d2be8640 10 monclient: monmap: 2024-03-25T18:36:36.754 INFO:teuthology.misc.health.smithi001.stderr:epoch 0 2024-03-25T18:36:36.754 INFO:teuthology.misc.health.smithi001.stderr:fsid c8ea72cf-9e99-4383-9b52-6c23d365fdfc 2024-03-25T18:36:36.754 INFO:teuthology.misc.health.smithi001.stderr:last_changed 2024-03-25T18:36:36.754283+0000 2024-03-25T18:36:36.754 INFO:teuthology.misc.health.smithi001.stderr:created 2024-03-25T18:36:36.754283+0000 2024-03-25T18:36:36.754 INFO:teuthology.misc.health.smithi001.stderr:min_mon_release 0 (unknown) 2024-03-25T18:36:36.754 INFO:teuthology.misc.health.smithi001.stderr:election_strategy: 3 2024-03-25T18:36:36.755 INFO:teuthology.misc.health.smithi001.stderr:0: v1:172.21.15.1:6789/0 mon.noname-a 2024-03-25T18:36:36.755 INFO:teuthology.misc.health.smithi001.stderr:1: v1:172.21.15.115:6789/0 mon.noname-c 2024-03-25T18:36:36.755 INFO:teuthology.misc.health.smithi001.stderr:2: v1:172.21.15.1:6790/0 mon.noname-b 2024-03-25T18:36:36.755 INFO:teuthology.misc.health.smithi001.stderr: 2024-03-25T18:36:36.756 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:36:36.754+0000 7fa7d2be8640 1 Processor -- start 2024-03-25T18:36:36.756 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:36:36.754+0000 7fa7d2be8640 1 -- start start 2024-03-25T18:36:36.756 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:36:36.754+0000 7fa7d2be8640 10 monclient: init 2024-03-25T18:36:36.756 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:36:36.755+0000 7fa7d2be8640 10 monclient: _reopen_session rank -1 2024-03-25T18:36:36.756 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:36:36.755+0000 7fa7d2be8640 10 monclient: _add_conns ranks=[2,1,0] 2024-03-25T18:36:36.756 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:36:36.755+0000 7fa7d2be8640 10 monclient(hunting): picked mon.noname-b con 0x7fa7cc0b4ff0 addr v1:172.21.15.1:6790/0 2024-03-25T18:36:36.756 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:36:36.755+0000 7fa7d2be8640 10 monclient(hunting): picked mon.noname-c con 0x7fa7cc0b4670 addr v1:172.21.15.115:6789/0 2024-03-25T18:36:36.756 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:36:36.755+0000 7fa7d2be8640 10 monclient(hunting): picked mon.noname-a con 0x7fa7cc13c480 addr v1:172.21.15.1:6789/0 2024-03-25T18:36:36.757 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:36:36.755+0000 7fa7d2be8640 1 -- --> v1:172.21.15.1:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa7cc0ab0e0 con 0x7fa7cc13c480 2024-03-25T18:36:36.757 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:36:36.755+0000 7fa7d2be8640 1 -- --> v1:172.21.15.115:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa7cc0ab3c0 con 0x7fa7cc0b4670 2024-03-25T18:36:36.757 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:36:36.755+0000 7fa7d2be8640 1 -- --> v1:172.21.15.1:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa7cc0a9f30 con 0x7fa7cc0b4ff0 2024-03-25T18:36:36.757 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:36:36.755+0000 7fa7d2be8640 10 monclient(hunting): _renew_subs 2024-03-25T18:36:36.757 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:36:36.755+0000 7fa7d2be8640 10 monclient(hunting): authenticate will time out at 8077.225263s 2024-03-25T18:36:36.757 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:36:36.756+0000 7fa7cbfff640 1 --1- >> v1:172.21.15.115:6789/0 conn(0x7fa7cc0b4670 0x7fa7cc1425c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.115:6789/0 says I am v1:172.21.15.1:45100/0 (socket says 172.21.15.1:45100) 2024-03-25T18:36:36.757 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:36:36.756+0000 7fa7cbfff640 1 -- 172.21.15.1:0/2619606166 learned_addr learned my addr 172.21.15.1:0/2619606166 (peer_addr_for_me v1:172.21.15.1:0/0) 2024-03-25T18:36:36.846 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:36:36.845+0000 7fa7cb7fe640 1 -- 172.21.15.1:0/2619606166 <== mon.0 v1:172.21.15.1:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 379332453 0 0) 0x7fa7cc0ab0e0 con 0x7fa7cc13c480 2024-03-25T18:36:36.846 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:36:36.845+0000 7fa7cb7fe640 10 monclient(hunting): _negotiate 2024-03-25T18:36:36.846 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:36:36.845+0000 7fa7cb7fe640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:36:36.846 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:36:36.845+0000 7fa7cb7fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:36:36.846 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:36:36.845+0000 7fa7cb7fe640 10 monclient(hunting): my global_id is 4641 2024-03-25T18:36:36.846 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:36:36.845+0000 7fa7cb7fe640 1 -- 172.21.15.1:0/2619606166 --> v1:172.21.15.1:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fa7bc003520 con 0x7fa7cc13c480 2024-03-25T18:36:36.868 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:36:36.867+0000 7fa7cb7fe640 1 -- 172.21.15.1:0/2619606166 <== mon.0 v1:172.21.15.1:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2545952830 0 0) 0x7fa7bc003520 con 0x7fa7cc13c480 2024-03-25T18:36:36.868 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:36:36.867+0000 7fa7cb7fe640 1 -- 172.21.15.1:0/2619606166 >> v1:172.21.15.1:6790/0 conn(0x7fa7cc0b4ff0 legacy=0x7fa7cc13bd70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:36:36.868 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:36:36.867+0000 7fa7cb7fe640 1 -- 172.21.15.1:0/2619606166 >> v1:172.21.15.115:6789/0 conn(0x7fa7cc0b4670 legacy=0x7fa7cc1425c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:36:36.868 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:36:36.867+0000 7fa7cb7fe640 10 monclient: _finish_hunting 0 2024-03-25T18:36:36.868 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:36:36.867+0000 7fa7cb7fe640 1 monclient: found mon.noname-a 2024-03-25T18:36:36.868 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:36:36.867+0000 7fa7cb7fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:36:36.868 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:36:36.867+0000 7fa7cb7fe640 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.1:6789/0 2024-03-25T18:36:36.869 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:36:36.867+0000 7fa7cb7fe640 1 -- 172.21.15.1:0/2619606166 --> v1:172.21.15.1:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa7cc0a9b70 con 0x7fa7cc13c480 2024-03-25T18:36:36.869 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:36:36.867+0000 7fa7cb7fe640 10 monclient: _finish_auth 0 2024-03-25T18:36:36.869 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:36:36.867+0000 7fa7cb7fe640 10 monclient: _check_auth_tickets 2024-03-25T18:36:36.869 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:36:36.867+0000 7fa7cb7fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:36:36.869 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:36:36.867+0000 7fa7cb7fe640 1 -- 172.21.15.1:0/2619606166 <== mon.0 v1:172.21.15.1:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1586965025 0 0) 0x7fa7c4001a10 con 0x7fa7cc13c480 2024-03-25T18:36:36.869 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:36:36.867+0000 7fa7d2be8640 5 monclient: authenticate success, global_id 4641 2024-03-25T18:36:36.869 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:36:36.867+0000 7fa7d2be8640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T18:36:36.869 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:36:36.867+0000 7fa7cb7fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:36:36.869 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:36:36.867+0000 7fa7cb7fe640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-03-25T18:36:36.869 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:36:36.867+0000 7fa7cb7fe640 10 monclient: dump: 2024-03-25T18:36:36.869 INFO:teuthology.misc.health.smithi001.stderr:epoch 1 2024-03-25T18:36:36.869 INFO:teuthology.misc.health.smithi001.stderr:fsid c8ea72cf-9e99-4383-9b52-6c23d365fdfc 2024-03-25T18:36:36.869 INFO:teuthology.misc.health.smithi001.stderr:last_changed 2024-03-25T18:25:50.510519+0000 2024-03-25T18:36:36.869 INFO:teuthology.misc.health.smithi001.stderr:created 2024-03-25T18:25:50.510519+0000 2024-03-25T18:36:36.870 INFO:teuthology.misc.health.smithi001.stderr:min_mon_release 19 (squid) 2024-03-25T18:36:36.870 INFO:teuthology.misc.health.smithi001.stderr:election_strategy: 3 2024-03-25T18:36:36.870 INFO:teuthology.misc.health.smithi001.stderr:0: v1:172.21.15.1:6789/0 mon.a 2024-03-25T18:36:36.870 INFO:teuthology.misc.health.smithi001.stderr:1: v1:172.21.15.115:6789/0 mon.b 2024-03-25T18:36:36.870 INFO:teuthology.misc.health.smithi001.stderr:2: v1:172.21.15.1:6790/0 mon.c 2024-03-25T18:36:36.870 INFO:teuthology.misc.health.smithi001.stderr: 2024-03-25T18:36:36.870 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:36:36.867+0000 7fa7d2be8640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T18:36:36.880 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:36:36.880+0000 7fa7cb7fe640 1 -- 172.21.15.1:0/2619606166 <== mon.0 v1:172.21.15.1:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fa7c4001b70 con 0x7fa7cc13c480 2024-03-25T18:36:36.881 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:36:36.880+0000 7fa7cb7fe640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:36:36.881 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:36:36.880+0000 7fa7cb7fe640 1 -- 172.21.15.1:0/2619606166 <== mon.0 v1:172.21.15.1:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1586965025 0 0) 0x7fa7c4001f80 con 0x7fa7cc13c480 2024-03-25T18:36:36.881 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:36:36.880+0000 7fa7cb7fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:36:36.881 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:36:36.880+0000 7fa7cb7fe640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-25T18:36:36.881 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:36:36.880+0000 7fa7cb7fe640 10 monclient: dump: 2024-03-25T18:36:36.881 INFO:teuthology.misc.health.smithi001.stderr:epoch 1 2024-03-25T18:36:36.881 INFO:teuthology.misc.health.smithi001.stderr:fsid c8ea72cf-9e99-4383-9b52-6c23d365fdfc 2024-03-25T18:36:36.881 INFO:teuthology.misc.health.smithi001.stderr:last_changed 2024-03-25T18:25:50.510519+0000 2024-03-25T18:36:36.881 INFO:teuthology.misc.health.smithi001.stderr:created 2024-03-25T18:25:50.510519+0000 2024-03-25T18:36:36.881 INFO:teuthology.misc.health.smithi001.stderr:min_mon_release 19 (squid) 2024-03-25T18:36:36.881 INFO:teuthology.misc.health.smithi001.stderr:election_strategy: 3 2024-03-25T18:36:36.881 INFO:teuthology.misc.health.smithi001.stderr:0: v1:172.21.15.1:6789/0 mon.a 2024-03-25T18:36:36.881 INFO:teuthology.misc.health.smithi001.stderr:1: v1:172.21.15.115:6789/0 mon.b 2024-03-25T18:36:36.881 INFO:teuthology.misc.health.smithi001.stderr:2: v1:172.21.15.1:6790/0 mon.c 2024-03-25T18:36:36.881 INFO:teuthology.misc.health.smithi001.stderr: 2024-03-25T18:36:36.881 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:36:36.880+0000 7fa7d2be8640 10 monclient: get_monmap_and_config success 2024-03-25T18:36:36.882 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:36:36.880+0000 7fa7d2be8640 4 set_mon_vals no callback set 2024-03-25T18:36:36.883 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:36:36.881+0000 7fa7d2be8640 10 monclient: shutdown 2024-03-25T18:36:36.884 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:36:36.881+0000 7fa7d2be8640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:36:36.884 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:36:36.881+0000 7fa7d2be8640 1 -- 172.21.15.1:0/2619606166 >> v1:172.21.15.1:6789/0 conn(0x7fa7cc13c480 legacy=0x7fa7cc14dc00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:36:36.884 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:36:36.881+0000 7fa7d2be8640 1 -- 172.21.15.1:0/2619606166 shutdown_connections 2024-03-25T18:36:36.884 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:36:36.881+0000 7fa7d2be8640 1 -- 172.21.15.1:0/2619606166 >> 172.21.15.1:0/2619606166 conn(0x7fa7cc0617a0 msgr2=0x7fa7cc061bb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:36:36.884 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:36:36.881+0000 7fa7d2be8640 1 -- 172.21.15.1:0/2619606166 shutdown_connections 2024-03-25T18:36:36.884 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:36:36.881+0000 7fa7d2be8640 1 -- 172.21.15.1:0/2619606166 wait complete. 2024-03-25T18:36:36.884 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:36:36.881+0000 7fa7d2be8640 10 monclient: build_initial_monmap 2024-03-25T18:36:36.884 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:36:36.881+0000 7fa7d2be8640 10 monclient: monmap: 2024-03-25T18:36:36.884 INFO:teuthology.misc.health.smithi001.stderr:epoch 0 2024-03-25T18:36:36.884 INFO:teuthology.misc.health.smithi001.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T18:36:36.884 INFO:teuthology.misc.health.smithi001.stderr:last_changed 0.000000 2024-03-25T18:36:36.884 INFO:teuthology.misc.health.smithi001.stderr:created 0.000000 2024-03-25T18:36:36.884 INFO:teuthology.misc.health.smithi001.stderr:min_mon_release 0 (unknown) 2024-03-25T18:36:36.884 INFO:teuthology.misc.health.smithi001.stderr:election_strategy: 1 2024-03-25T18:36:36.884 INFO:teuthology.misc.health.smithi001.stderr:0: v1:172.21.15.1:6789/0 mon.noname-a 2024-03-25T18:36:36.884 INFO:teuthology.misc.health.smithi001.stderr:1: v1:172.21.15.115:6789/0 mon.noname-b 2024-03-25T18:36:36.885 INFO:teuthology.misc.health.smithi001.stderr:2: v1:172.21.15.1:6790/0 mon.noname-c 2024-03-25T18:36:36.885 INFO:teuthology.misc.health.smithi001.stderr: 2024-03-25T18:36:36.885 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:36:36.882+0000 7fa7d2be8640 1 librados: starting msgr at 2024-03-25T18:36:36.885 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:36:36.882+0000 7fa7d2be8640 1 librados: starting objecter 2024-03-25T18:36:36.885 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:36:36.882+0000 7fa7d2be8640 1 Processor -- start 2024-03-25T18:36:36.885 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:36:36.882+0000 7fa7d2be8640 1 -- start start 2024-03-25T18:36:36.885 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:36:36.882+0000 7fa7d2be8640 1 librados: setting wanted keys 2024-03-25T18:36:36.885 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:36:36.882+0000 7fa7d2be8640 1 librados: calling monclient init 2024-03-25T18:36:36.885 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:36:36.882+0000 7fa7d2be8640 10 monclient: init 2024-03-25T18:36:36.885 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:36:36.882+0000 7fa7d2be8640 10 monclient: _reopen_session rank -1 2024-03-25T18:36:36.885 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:36:36.882+0000 7fa7d2be8640 10 monclient: _add_conns ranks=[2,1,0] 2024-03-25T18:36:36.885 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:36:36.883+0000 7fa7d2be8640 10 monclient(hunting): picked mon.noname-c con 0x7fa7cc0b4670 addr v1:172.21.15.1:6790/0 2024-03-25T18:36:36.885 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:36:36.883+0000 7fa7d2be8640 10 monclient(hunting): picked mon.noname-b con 0x7fa7cc0b4ff0 addr v1:172.21.15.115:6789/0 2024-03-25T18:36:36.885 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:36:36.883+0000 7fa7d2be8640 10 monclient(hunting): picked mon.noname-a con 0x7fa7cc13c480 addr v1:172.21.15.1:6789/0 2024-03-25T18:36:36.885 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:36:36.883+0000 7fa7d2be8640 1 -- --> v1:172.21.15.1:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa7cc13deb0 con 0x7fa7cc13c480 2024-03-25T18:36:36.885 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:36:36.883+0000 7fa7d2be8640 1 -- --> v1:172.21.15.115:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa7cc13e190 con 0x7fa7cc0b4ff0 2024-03-25T18:36:36.886 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:36:36.883+0000 7fa7d2be8640 1 -- --> v1:172.21.15.1:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa7cc13e470 con 0x7fa7cc0b4670 2024-03-25T18:36:36.886 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:36:36.883+0000 7fa7d2be8640 10 monclient(hunting): _renew_subs 2024-03-25T18:36:36.886 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:36:36.883+0000 7fa7d2be8640 10 monclient(hunting): authenticate will time out at 8077.352841s 2024-03-25T18:36:36.897 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:36:36.897+0000 7fa7d1185640 1 --1- >> v1:172.21.15.1:6789/0 conn(0x7fa7cc13c480 0x7fa7cc1e26e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.1:6789/0 says I am v1:172.21.15.1:58482/0 (socket says 172.21.15.1:58482) 2024-03-25T18:36:36.897 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:36:36.897+0000 7fa7d1185640 1 -- 172.21.15.1:0/3759053879 learned_addr learned my addr 172.21.15.1:0/3759053879 (peer_addr_for_me v1:172.21.15.1:0/0) 2024-03-25T18:36:36.898 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:36:36.897+0000 7fa7c97fa640 10 client.?.objecter ms_handle_connect 0x7fa7cc13c480 2024-03-25T18:36:36.898 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:36:36.897+0000 7fa7c97fa640 10 client.?.objecter resend_mon_ops 2024-03-25T18:36:36.905 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:36:36.904+0000 7fa7c97fa640 10 client.?.objecter ms_handle_connect 0x7fa7cc0b4ff0 2024-03-25T18:36:36.905 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:36:36.904+0000 7fa7c97fa640 10 client.?.objecter resend_mon_ops 2024-03-25T18:36:36.913 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:36:36.913+0000 7fa7c97fa640 10 client.?.objecter ms_handle_connect 0x7fa7cc0b4670 2024-03-25T18:36:36.913 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:36:36.913+0000 7fa7c97fa640 10 client.?.objecter resend_mon_ops 2024-03-25T18:36:36.914 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:36:36.914+0000 7fa7c97fa640 1 -- 172.21.15.1:0/3759053879 <== mon.0 v1:172.21.15.1:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2554430862 0 0) 0x7fa7cc13deb0 con 0x7fa7cc13c480 2024-03-25T18:36:36.914 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:36:36.914+0000 7fa7c97fa640 10 monclient(hunting): _negotiate 2024-03-25T18:36:36.915 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:36:36.914+0000 7fa7c97fa640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:36:36.915 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:36:36.914+0000 7fa7c97fa640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:36:36.915 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:36:36.914+0000 7fa7c97fa640 10 monclient(hunting): my global_id is 4644 2024-03-25T18:36:36.915 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:36:36.914+0000 7fa7c97fa640 1 -- 172.21.15.1:0/3759053879 --> v1:172.21.15.1:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fa7b0004eb0 con 0x7fa7cc13c480 2024-03-25T18:36:36.922 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:36:36.921+0000 7fa7c97fa640 1 -- 172.21.15.1:0/3759053879 <== mon.1 v1:172.21.15.115:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 779857408 0 0) 0x7fa7cc13e190 con 0x7fa7cc0b4ff0 2024-03-25T18:36:36.922 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:36:36.921+0000 7fa7c97fa640 10 monclient(hunting): _negotiate 2024-03-25T18:36:36.922 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:36:36.921+0000 7fa7c97fa640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:36:36.922 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:36:36.921+0000 7fa7c97fa640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:36:36.922 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:36:36.921+0000 7fa7c97fa640 10 monclient(hunting): my global_id is 4657 2024-03-25T18:36:36.922 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:36:36.921+0000 7fa7c97fa640 1 -- 172.21.15.1:0/3759053879 --> v1:172.21.15.115:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fa7cc13deb0 con 0x7fa7cc0b4ff0 2024-03-25T18:36:36.935 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:36:36.935+0000 7fa7c97fa640 1 -- 172.21.15.1:0/3759053879 <== mon.2 v1:172.21.15.1:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1499391659 0 0) 0x7fa7cc13e470 con 0x7fa7cc0b4670 2024-03-25T18:36:36.936 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:36:36.935+0000 7fa7c97fa640 10 monclient(hunting): _negotiate 2024-03-25T18:36:36.936 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:36:36.935+0000 7fa7c97fa640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:36:36.936 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:36:36.935+0000 7fa7c97fa640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:36:36.936 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:36:36.935+0000 7fa7c97fa640 10 monclient(hunting): my global_id is 4655 2024-03-25T18:36:36.936 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:36:36.935+0000 7fa7c97fa640 1 -- 172.21.15.1:0/3759053879 --> v1:172.21.15.1:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fa7cc13e190 con 0x7fa7cc0b4670 2024-03-25T18:36:36.940 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:36:36.940+0000 7fa7c97fa640 1 -- 172.21.15.1:0/3759053879 <== mon.0 v1:172.21.15.1:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1420716545 0 0) 0x7fa7b0004eb0 con 0x7fa7cc13c480 2024-03-25T18:36:36.941 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:36:36.940+0000 7fa7c97fa640 1 -- 172.21.15.1:0/3759053879 --> v1:172.21.15.1:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fa7cc13e470 con 0x7fa7cc13c480 2024-03-25T18:36:36.941 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:36:36.940+0000 7fa7c97fa640 1 -- 172.21.15.1:0/3759053879 <== mon.0 v1:172.21.15.1:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1586965025 0 0) 0x7fa7c4001fc0 con 0x7fa7cc13c480 2024-03-25T18:36:36.941 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:36:36.940+0000 7fa7c97fa640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:36:36.941 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:36:36.940+0000 7fa7c97fa640 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2024-03-25T18:36:36.941 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:36:36.940+0000 7fa7c97fa640 10 monclient(hunting): dump: 2024-03-25T18:36:36.941 INFO:teuthology.misc.health.smithi001.stderr:epoch 1 2024-03-25T18:36:36.941 INFO:teuthology.misc.health.smithi001.stderr:fsid c8ea72cf-9e99-4383-9b52-6c23d365fdfc 2024-03-25T18:36:36.941 INFO:teuthology.misc.health.smithi001.stderr:last_changed 2024-03-25T18:25:50.510519+0000 2024-03-25T18:36:36.941 INFO:teuthology.misc.health.smithi001.stderr:created 2024-03-25T18:25:50.510519+0000 2024-03-25T18:36:36.941 INFO:teuthology.misc.health.smithi001.stderr:min_mon_release 19 (squid) 2024-03-25T18:36:36.941 INFO:teuthology.misc.health.smithi001.stderr:election_strategy: 3 2024-03-25T18:36:36.941 INFO:teuthology.misc.health.smithi001.stderr:0: v1:172.21.15.1:6789/0 mon.a 2024-03-25T18:36:36.941 INFO:teuthology.misc.health.smithi001.stderr:1: v1:172.21.15.115:6789/0 mon.b 2024-03-25T18:36:36.941 INFO:teuthology.misc.health.smithi001.stderr:2: v1:172.21.15.1:6790/0 mon.c 2024-03-25T18:36:36.941 INFO:teuthology.misc.health.smithi001.stderr: 2024-03-25T18:36:36.942 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:36:36.940+0000 7fa7c97fa640 10 monclient(hunting): _finish_auth 0 2024-03-25T18:36:36.949 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:36:36.948+0000 7fa7c97fa640 1 -- 172.21.15.1:0/3759053879 <== mon.2 v1:172.21.15.1:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 798617962 0 0) 0x7fa7cc13e190 con 0x7fa7cc0b4670 2024-03-25T18:36:36.949 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:36:36.948+0000 7fa7c97fa640 1 -- 172.21.15.1:0/3759053879 --> v1:172.21.15.1:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fa7b0004eb0 con 0x7fa7cc0b4670 2024-03-25T18:36:36.949 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:36:36.948+0000 7fa7c97fa640 1 -- 172.21.15.1:0/3759053879 <== mon.2 v1:172.21.15.1:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1586965025 0 0) 0x7fa7c0001860 con 0x7fa7cc0b4670 2024-03-25T18:36:36.949 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:36:36.948+0000 7fa7c97fa640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:36:36.949 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:36:36.948+0000 7fa7c97fa640 10 monclient(hunting): got monmap 1 from mon.c (according to old e1) 2024-03-25T18:36:36.949 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:36:36.948+0000 7fa7c97fa640 10 monclient(hunting): dump: 2024-03-25T18:36:36.949 INFO:teuthology.misc.health.smithi001.stderr:epoch 1 2024-03-25T18:36:36.949 INFO:teuthology.misc.health.smithi001.stderr:fsid c8ea72cf-9e99-4383-9b52-6c23d365fdfc 2024-03-25T18:36:36.949 INFO:teuthology.misc.health.smithi001.stderr:last_changed 2024-03-25T18:25:50.510519+0000 2024-03-25T18:36:36.949 INFO:teuthology.misc.health.smithi001.stderr:created 2024-03-25T18:25:50.510519+0000 2024-03-25T18:36:36.949 INFO:teuthology.misc.health.smithi001.stderr:min_mon_release 19 (squid) 2024-03-25T18:36:36.949 INFO:teuthology.misc.health.smithi001.stderr:election_strategy: 3 2024-03-25T18:36:36.949 INFO:teuthology.misc.health.smithi001.stderr:0: v1:172.21.15.1:6789/0 mon.a 2024-03-25T18:36:36.949 INFO:teuthology.misc.health.smithi001.stderr:1: v1:172.21.15.115:6789/0 mon.b 2024-03-25T18:36:36.950 INFO:teuthology.misc.health.smithi001.stderr:2: v1:172.21.15.1:6790/0 mon.c 2024-03-25T18:36:36.950 INFO:teuthology.misc.health.smithi001.stderr: 2024-03-25T18:36:36.954 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:36:36.953+0000 7fa7c97fa640 1 -- 172.21.15.1:0/3759053879 <== mon.0 v1:172.21.15.1:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 3883544660 0 0) 0x7fa7cc13e470 con 0x7fa7cc13c480 2024-03-25T18:36:36.954 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:36:36.953+0000 7fa7c97fa640 1 -- 172.21.15.1:0/3759053879 >> v1:172.21.15.1:6790/0 conn(0x7fa7cc0b4670 legacy=0x7fa7cc1db410 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:36:36.954 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:36:36.953+0000 7fa7c97fa640 1 -- 172.21.15.1:0/3759053879 >> v1:172.21.15.115:6789/0 conn(0x7fa7cc0b4ff0 legacy=0x7fa7cc1dbb20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:36:36.954 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:36:36.953+0000 7fa7c97fa640 10 monclient: _finish_hunting 0 2024-03-25T18:36:36.955 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:36:36.953+0000 7fa7c97fa640 1 monclient: found mon.a 2024-03-25T18:36:36.955 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:36:36.953+0000 7fa7c97fa640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:36:36.955 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:36:36.953+0000 7fa7c97fa640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.1:6789/0 2024-03-25T18:36:36.955 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:36:36.953+0000 7fa7c97fa640 1 -- 172.21.15.1:0/3759053879 --> v1:172.21.15.1:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa7cc13ce60 con 0x7fa7cc13c480 2024-03-25T18:36:36.955 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:36:36.953+0000 7fa7c97fa640 10 monclient: _finish_auth 0 2024-03-25T18:36:36.955 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:36:36.953+0000 7fa7c97fa640 10 monclient: _check_auth_tickets 2024-03-25T18:36:36.955 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:36:36.953+0000 7fa7c97fa640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:36:36.955 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:36:36.953+0000 7fa7d2be8640 5 monclient: authenticate success, global_id 4644 2024-03-25T18:36:36.955 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:36:36.953+0000 7fa7d2be8640 10 monclient: _renew_subs 2024-03-25T18:36:36.955 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:36:36.953+0000 7fa7d2be8640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.1:6789/0 2024-03-25T18:36:36.955 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:36:36.953+0000 7fa7d2be8640 1 -- 172.21.15.1:0/3759053879 --> v1:172.21.15.1:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa7cc13d130 con 0x7fa7cc13c480 2024-03-25T18:36:36.955 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:36:36.953+0000 7fa7d2be8640 10 client.4644.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T18:36:36.955 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:36:36.953+0000 7fa7d2be8640 10 monclient: _renew_subs 2024-03-25T18:36:36.955 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:36:36.953+0000 7fa7d2be8640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.1:6789/0 2024-03-25T18:36:36.955 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:36:36.953+0000 7fa7d2be8640 1 -- 172.21.15.1:0/3759053879 --> v1:172.21.15.1:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7fa7cc143840 con 0x7fa7cc13c480 2024-03-25T18:36:36.956 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:36:36.953+0000 7fa7d2be8640 1 librados: init done 2024-03-25T18:36:36.956 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:36:36.955+0000 7fa7d2be8640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T18:36:36.956 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:36:36.955+0000 7fa7d2be8640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:36:36.956 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:36:36.955+0000 7fa7d2be8640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.1:6789/0 2024-03-25T18:36:36.956 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:36:36.955+0000 7fa7d2be8640 1 -- 172.21.15.1:0/3759053879 --> v1:172.21.15.1:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa7cc06dbd0 con 0x7fa7cc13c480 2024-03-25T18:36:37.281 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:36:37.280+0000 7fa7c97fa640 1 -- 172.21.15.1:0/3759053879 <== mon.0 v1:172.21.15.1:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fa7c4003c20 con 0x7fa7cc13c480 2024-03-25T18:36:37.281 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:36:37.280+0000 7fa7c97fa640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:36:37.281 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:36:37.280+0000 7fa7c97fa640 1 -- 172.21.15.1:0/3759053879 <== mon.0 v1:172.21.15.1:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1586965025 0 0) 0x7fa7c4003ff0 con 0x7fa7cc13c480 2024-03-25T18:36:37.281 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:36:37.280+0000 7fa7c97fa640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:36:37.281 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:36:37.280+0000 7fa7c97fa640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-25T18:36:37.281 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:36:37.280+0000 7fa7c97fa640 10 monclient: dump: 2024-03-25T18:36:37.281 INFO:teuthology.misc.health.smithi001.stderr:epoch 1 2024-03-25T18:36:37.281 INFO:teuthology.misc.health.smithi001.stderr:fsid c8ea72cf-9e99-4383-9b52-6c23d365fdfc 2024-03-25T18:36:37.281 INFO:teuthology.misc.health.smithi001.stderr:last_changed 2024-03-25T18:25:50.510519+0000 2024-03-25T18:36:37.282 INFO:teuthology.misc.health.smithi001.stderr:created 2024-03-25T18:25:50.510519+0000 2024-03-25T18:36:37.282 INFO:teuthology.misc.health.smithi001.stderr:min_mon_release 19 (squid) 2024-03-25T18:36:37.282 INFO:teuthology.misc.health.smithi001.stderr:election_strategy: 3 2024-03-25T18:36:37.282 INFO:teuthology.misc.health.smithi001.stderr:0: v1:172.21.15.1:6789/0 mon.a 2024-03-25T18:36:37.282 INFO:teuthology.misc.health.smithi001.stderr:1: v1:172.21.15.115:6789/0 mon.b 2024-03-25T18:36:37.282 INFO:teuthology.misc.health.smithi001.stderr:2: v1:172.21.15.1:6790/0 mon.c 2024-03-25T18:36:37.282 INFO:teuthology.misc.health.smithi001.stderr: 2024-03-25T18:36:37.282 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:36:37.280+0000 7fa7caffd640 4 set_mon_vals no callback set 2024-03-25T18:36:37.282 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:36:37.281+0000 7fa7c97fa640 1 -- 172.21.15.1:0/3759053879 <== mon.0 v1:172.21.15.1:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 21787975 0 0) 0x7fa7c4004210 con 0x7fa7cc13c480 2024-03-25T18:36:37.282 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:36:37.282+0000 7fa7d0984640 10 monclient: get_auth_request con 0x7fa7b006df70 auth_method 0 2024-03-25T18:36:37.283 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:36:37.282+0000 7fa7c97fa640 10 client.4644.objecter ms_handle_connect 0x7fa7b006df70 2024-03-25T18:36:37.289 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:36:37.288+0000 7fa7c97fa640 1 -- 172.21.15.1:0/3759053879 <== mon.0 v1:172.21.15.1:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 996254230 0 0) 0x7fa7c4054bf0 con 0x7fa7cc13c480 2024-03-25T18:36:37.289 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:36:37.288+0000 7fa7c97fa640 10 client.4644.objecter ms_dispatch 0x7fa7cc000b90 osd_map(10..10 src has 1..10) v4 2024-03-25T18:36:37.289 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:36:37.288+0000 7fa7c97fa640 3 client.4644.objecter handle_osd_map got epochs [10,10] > 0 2024-03-25T18:36:37.289 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:36:37.288+0000 7fa7c97fa640 3 client.4644.objecter handle_osd_map decoding full epoch 10 2024-03-25T18:36:37.289 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:36:37.288+0000 7fa7c97fa640 20 client.4644.objecter dump_active .. 0 homeless 2024-03-25T18:36:37.289 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:36:37.288+0000 7fa7c97fa640 1 -- 172.21.15.1:0/3759053879 <== mon.0 v1:172.21.15.1:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7fa7c4050700 con 0x7fa7cc13c480 2024-03-25T18:36:37.289 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:36:37.288+0000 7fa7c97fa640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:36:37.289 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:36:37.288+0000 7fa7c97fa640 10 monclient: _finish_command 1 = system:0 2024-03-25T18:36:37.557 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:36:37.556+0000 7fa7d2be8640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:36:37.557 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:36:37.556+0000 7fa7d2be8640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:36:37.557 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:36:37.556+0000 7fa7d2be8640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.1:6789/0 2024-03-25T18:36:37.557 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:36:37.556+0000 7fa7d2be8640 1 -- 172.21.15.1:0/3759053879 --> v1:172.21.15.1:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fa7cc0688c0 con 0x7fa7cc13c480 2024-03-25T18:36:37.594 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:36:37.594+0000 7fa7c97fa640 1 -- 172.21.15.1:0/3759053879 <== mon.0 v1:172.21.15.1:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 4092213348) 0x7fa7c4052630 con 0x7fa7cc13c480 2024-03-25T18:36:37.594 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:36:37.594+0000 7fa7c97fa640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:36:37.595 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:36:37.594+0000 7fa7c97fa640 10 monclient: _finish_command 2 = system:0 2024-03-25T18:36:37.595 INFO:teuthology.misc.health.smithi001.stdout: 2024-03-25T18:36:37.595 INFO:teuthology.misc.health.smithi001.stdout:{"epoch":10,"fsid":"c8ea72cf-9e99-4383-9b52-6c23d365fdfc","created":"2024-03-25T18:26:38.856638+0000","modified":"2024-03-25T18:26:44.224208+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T18:26:44.224208+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"b9f34922-3a84-49be-8426-807c1eceaefc","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"42bf306f-55e3-4634-9874-8b4b73ac6644","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"292cb23a-18a9-4207-a7cc-4d5727feaec0","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"45e0e9ab-898d-4673-9590-b4117bc5e84a","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"831a5a09-73ab-4a61-9872-088e76bdbc66","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"88cd903d-3b55-4840-9396-6e3cfe005282","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"89db7355-bc9d-46f4-8ab0-e8d9e744e026","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"cac47bee-0914-4f08-896f-348838882866","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T18:36:37.597 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:36:37.596+0000 7fa7d2be8640 10 librados: watch_flush enter 2024-03-25T18:36:37.597 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:36:37.596+0000 7fa7d2be8640 10 librados: watch_flush exit 2024-03-25T18:36:37.597 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:36:37.596+0000 7fa7d2be8640 20 client.4644.objecter shutdown clearing up homeless session... 2024-03-25T18:36:37.597 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:36:37.596+0000 7fa7d2be8640 10 client.4644.objecter successfully canceled tick 2024-03-25T18:36:37.597 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:36:37.596+0000 7fa7d2be8640 1 -- 172.21.15.1:0/3759053879 >> v1:172.21.15.115:6800/4237379176 conn(0x7fa7b006df70 legacy=0x7fa7b0070430 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:36:37.598 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:36:37.596+0000 7fa7d2be8640 10 monclient: shutdown 2024-03-25T18:36:37.598 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:36:37.596+0000 7fa7d2be8640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:36:37.598 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:36:37.596+0000 7fa7d2be8640 1 -- 172.21.15.1:0/3759053879 >> v1:172.21.15.1:6789/0 conn(0x7fa7cc13c480 legacy=0x7fa7cc1e26e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:36:37.598 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:36:37.596+0000 7fa7d2be8640 1 -- 172.21.15.1:0/3759053879 shutdown_connections 2024-03-25T18:36:37.598 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:36:37.596+0000 7fa7d2be8640 1 -- 172.21.15.1:0/3759053879 >> 172.21.15.1:0/3759053879 conn(0x7fa7cc0617a0 msgr2=0x7fa7cc140760 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:36:37.598 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:36:37.596+0000 7fa7d2be8640 1 -- 172.21.15.1:0/3759053879 shutdown_connections 2024-03-25T18:36:37.598 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:36:37.596+0000 7fa7d2be8640 1 -- 172.21.15.1:0/3759053879 wait complete. 2024-03-25T18:36:37.598 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:36:37.596+0000 7fa7d2be8640 1 librados: shutdown 2024-03-25T18:36:37.609 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T18:36:43.256 INFO:tasks.ceph.mon.a.smithi001.stderr:==00:00:10:21.814 34876== Warning: unimplemented fcntl command: 1036 2024-03-25T18:36:43.308 INFO:tasks.ceph.mon.a.smithi001.stderr:==00:00:10:21.867 34876== Warning: unimplemented fcntl command: 1036 2024-03-25T18:36:43.339 INFO:tasks.ceph.mon.a.smithi001.stderr:==00:00:10:21.897 34876== Warning: unimplemented fcntl command: 1036 2024-03-25T18:36:43.377 INFO:tasks.ceph.mon.b.smithi115.stderr:==00:00:10:21.861 34836== Warning: unimplemented fcntl command: 1036 2024-03-25T18:36:43.380 INFO:tasks.ceph.mon.b.smithi115.stderr:==00:00:10:21.864 34836== Warning: unimplemented fcntl command: 1036 2024-03-25T18:36:43.390 INFO:tasks.ceph.mon.c.smithi001.stderr:==00:00:10:21.949 34877== Warning: unimplemented fcntl command: 1036 2024-03-25T18:36:43.395 INFO:tasks.ceph.mon.c.smithi001.stderr:==00:00:10:21.954 34877== Warning: unimplemented fcntl command: 1036 2024-03-25T18:36:43.408 INFO:tasks.ceph.mon.b.smithi115.stderr:==00:00:10:21.892 34836== Warning: unimplemented fcntl command: 1036 2024-03-25T18:36:43.421 INFO:tasks.ceph.mon.c.smithi001.stderr:==00:00:10:21.979 34877== Warning: unimplemented fcntl command: 1036 2024-03-25T18:36:44.415 DEBUG:teuthology.orchestra.run.smithi001:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T18:36:44.530 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:36:44.530+0000 7f0185235640 10 monclient: get_monmap_and_config 2024-03-25T18:36:44.531 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:36:44.530+0000 7f0185235640 10 monclient: build_initial_monmap 2024-03-25T18:36:44.531 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:36:44.530+0000 7f0185235640 10 monclient: monmap: 2024-03-25T18:36:44.531 INFO:teuthology.misc.health.smithi001.stderr:epoch 0 2024-03-25T18:36:44.531 INFO:teuthology.misc.health.smithi001.stderr:fsid c8ea72cf-9e99-4383-9b52-6c23d365fdfc 2024-03-25T18:36:44.531 INFO:teuthology.misc.health.smithi001.stderr:last_changed 2024-03-25T18:36:44.530778+0000 2024-03-25T18:36:44.531 INFO:teuthology.misc.health.smithi001.stderr:created 2024-03-25T18:36:44.530778+0000 2024-03-25T18:36:44.531 INFO:teuthology.misc.health.smithi001.stderr:min_mon_release 0 (unknown) 2024-03-25T18:36:44.531 INFO:teuthology.misc.health.smithi001.stderr:election_strategy: 3 2024-03-25T18:36:44.531 INFO:teuthology.misc.health.smithi001.stderr:0: v1:172.21.15.1:6789/0 mon.noname-a 2024-03-25T18:36:44.531 INFO:teuthology.misc.health.smithi001.stderr:1: v1:172.21.15.115:6789/0 mon.noname-c 2024-03-25T18:36:44.531 INFO:teuthology.misc.health.smithi001.stderr:2: v1:172.21.15.1:6790/0 mon.noname-b 2024-03-25T18:36:44.531 INFO:teuthology.misc.health.smithi001.stderr: 2024-03-25T18:36:44.533 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:36:44.531+0000 7f0185235640 1 Processor -- start 2024-03-25T18:36:44.533 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:36:44.531+0000 7f0185235640 1 -- start start 2024-03-25T18:36:44.533 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:36:44.531+0000 7f0185235640 10 monclient: init 2024-03-25T18:36:44.533 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:36:44.531+0000 7f0185235640 10 monclient: _reopen_session rank -1 2024-03-25T18:36:44.533 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:36:44.531+0000 7f0185235640 10 monclient: _add_conns ranks=[1,2,0] 2024-03-25T18:36:44.533 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:36:44.531+0000 7f0185235640 10 monclient(hunting): picked mon.noname-c con 0x7f01800b4e70 addr v1:172.21.15.115:6789/0 2024-03-25T18:36:44.533 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:36:44.531+0000 7f0185235640 10 monclient(hunting): picked mon.noname-b con 0x7f01800b44b0 addr v1:172.21.15.1:6790/0 2024-03-25T18:36:44.533 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:36:44.531+0000 7f0185235640 10 monclient(hunting): picked mon.noname-a con 0x7f01800b1b00 addr v1:172.21.15.1:6789/0 2024-03-25T18:36:44.533 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:36:44.531+0000 7f0185235640 1 -- --> v1:172.21.15.1:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f01800a96f0 con 0x7f01800b1b00 2024-03-25T18:36:44.533 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:36:44.531+0000 7f0185235640 1 -- --> v1:172.21.15.115:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f01800ab0e0 con 0x7f01800b4e70 2024-03-25T18:36:44.533 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:36:44.531+0000 7f0185235640 1 -- --> v1:172.21.15.1:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0180145f70 con 0x7f01800b44b0 2024-03-25T18:36:44.534 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:36:44.531+0000 7f0185235640 10 monclient(hunting): _renew_subs 2024-03-25T18:36:44.534 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:36:44.531+0000 7f0185235640 10 monclient(hunting): authenticate will time out at 8085.001414s 2024-03-25T18:36:44.542 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:36:44.541+0000 7f017ed9d640 1 --1- >> v1:172.21.15.115:6789/0 conn(0x7f01800b4e70 0x7f018013f940 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.115:6789/0 says I am v1:172.21.15.1:55728/0 (socket says 172.21.15.1:55728) 2024-03-25T18:36:44.542 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:36:44.541+0000 7f017ed9d640 1 -- 172.21.15.1:0/716371401 learned_addr learned my addr 172.21.15.1:0/716371401 (peer_addr_for_me v1:172.21.15.1:0/0) 2024-03-25T18:36:44.548 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:36:44.547+0000 7f017e59c640 1 -- 172.21.15.1:0/716371401 <== mon.1 v1:172.21.15.115:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2820038810 0 0) 0x7f01800ab0e0 con 0x7f01800b4e70 2024-03-25T18:36:44.548 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:36:44.547+0000 7f017e59c640 10 monclient(hunting): _negotiate 2024-03-25T18:36:44.548 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:36:44.547+0000 7f017e59c640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:36:44.548 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:36:44.547+0000 7f017e59c640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:36:44.548 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:36:44.547+0000 7f017e59c640 10 monclient(hunting): my global_id is 4660 2024-03-25T18:36:44.548 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:36:44.547+0000 7f017e59c640 1 -- 172.21.15.1:0/716371401 --> v1:172.21.15.115:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f016c003520 con 0x7f01800b4e70 2024-03-25T18:36:44.556 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:36:44.555+0000 7f017e59c640 1 -- 172.21.15.1:0/716371401 <== mon.1 v1:172.21.15.115:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2613819565 0 0) 0x7f016c003520 con 0x7f01800b4e70 2024-03-25T18:36:44.556 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:36:44.555+0000 7f017e59c640 1 -- 172.21.15.1:0/716371401 >> v1:172.21.15.1:6790/0 conn(0x7f01800b44b0 legacy=0x7f0180145860 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:36:44.556 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:36:44.555+0000 7f017e59c640 1 -- 172.21.15.1:0/716371401 >> v1:172.21.15.1:6789/0 conn(0x7f01800b1b00 legacy=0x7f018014dc00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:36:44.556 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:36:44.555+0000 7f017e59c640 10 monclient: _finish_hunting 0 2024-03-25T18:36:44.556 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:36:44.555+0000 7f017e59c640 1 monclient: found mon.noname-c 2024-03-25T18:36:44.556 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:36:44.555+0000 7f017e59c640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:36:44.556 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:36:44.555+0000 7f017e59c640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.115:6789/0 2024-03-25T18:36:44.556 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:36:44.555+0000 7f017e59c640 1 -- 172.21.15.1:0/716371401 --> v1:172.21.15.115:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f01800a9b80 con 0x7f01800b4e70 2024-03-25T18:36:44.556 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:36:44.555+0000 7f017e59c640 10 monclient: _finish_auth 0 2024-03-25T18:36:44.556 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:36:44.555+0000 7f017e59c640 10 monclient: _check_auth_tickets 2024-03-25T18:36:44.556 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:36:44.555+0000 7f017e59c640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:36:44.556 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:36:44.555+0000 7f017e59c640 1 -- 172.21.15.1:0/716371401 <== mon.1 v1:172.21.15.115:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1586965025 0 0) 0x7f0170001690 con 0x7f01800b4e70 2024-03-25T18:36:44.557 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:36:44.555+0000 7f017e59c640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:36:44.557 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:36:44.555+0000 7f017e59c640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-03-25T18:36:44.557 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:36:44.555+0000 7f017e59c640 10 monclient: dump: 2024-03-25T18:36:44.557 INFO:teuthology.misc.health.smithi001.stderr:epoch 1 2024-03-25T18:36:44.557 INFO:teuthology.misc.health.smithi001.stderr:fsid c8ea72cf-9e99-4383-9b52-6c23d365fdfc 2024-03-25T18:36:44.557 INFO:teuthology.misc.health.smithi001.stderr:last_changed 2024-03-25T18:25:50.510519+0000 2024-03-25T18:36:44.557 INFO:teuthology.misc.health.smithi001.stderr:created 2024-03-25T18:25:50.510519+0000 2024-03-25T18:36:44.557 INFO:teuthology.misc.health.smithi001.stderr:min_mon_release 19 (squid) 2024-03-25T18:36:44.557 INFO:teuthology.misc.health.smithi001.stderr:election_strategy: 3 2024-03-25T18:36:44.557 INFO:teuthology.misc.health.smithi001.stderr:0: v1:172.21.15.1:6789/0 mon.a 2024-03-25T18:36:44.557 INFO:teuthology.misc.health.smithi001.stderr:1: v1:172.21.15.115:6789/0 mon.b 2024-03-25T18:36:44.557 INFO:teuthology.misc.health.smithi001.stderr:2: v1:172.21.15.1:6790/0 mon.c 2024-03-25T18:36:44.557 INFO:teuthology.misc.health.smithi001.stderr: 2024-03-25T18:36:44.557 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:36:44.555+0000 7f0185235640 5 monclient: authenticate success, global_id 4660 2024-03-25T18:36:44.558 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:36:44.555+0000 7f0185235640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T18:36:44.561 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:36:44.561+0000 7f017e59c640 1 -- 172.21.15.1:0/716371401 <== mon.1 v1:172.21.15.115:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f01700017f0 con 0x7f01800b4e70 2024-03-25T18:36:44.561 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:36:44.561+0000 7f017e59c640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:36:44.561 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:36:44.561+0000 7f017e59c640 1 -- 172.21.15.1:0/716371401 <== mon.1 v1:172.21.15.115:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1586965025 0 0) 0x7f0170001c00 con 0x7f01800b4e70 2024-03-25T18:36:44.562 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:36:44.561+0000 7f017e59c640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:36:44.562 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:36:44.561+0000 7f017e59c640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-25T18:36:44.562 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:36:44.561+0000 7f017e59c640 10 monclient: dump: 2024-03-25T18:36:44.562 INFO:teuthology.misc.health.smithi001.stderr:epoch 1 2024-03-25T18:36:44.562 INFO:teuthology.misc.health.smithi001.stderr:fsid c8ea72cf-9e99-4383-9b52-6c23d365fdfc 2024-03-25T18:36:44.562 INFO:teuthology.misc.health.smithi001.stderr:last_changed 2024-03-25T18:25:50.510519+0000 2024-03-25T18:36:44.562 INFO:teuthology.misc.health.smithi001.stderr:created 2024-03-25T18:25:50.510519+0000 2024-03-25T18:36:44.562 INFO:teuthology.misc.health.smithi001.stderr:min_mon_release 19 (squid) 2024-03-25T18:36:44.562 INFO:teuthology.misc.health.smithi001.stderr:election_strategy: 3 2024-03-25T18:36:44.562 INFO:teuthology.misc.health.smithi001.stderr:0: v1:172.21.15.1:6789/0 mon.a 2024-03-25T18:36:44.562 INFO:teuthology.misc.health.smithi001.stderr:1: v1:172.21.15.115:6789/0 mon.b 2024-03-25T18:36:44.562 INFO:teuthology.misc.health.smithi001.stderr:2: v1:172.21.15.1:6790/0 mon.c 2024-03-25T18:36:44.562 INFO:teuthology.misc.health.smithi001.stderr: 2024-03-25T18:36:44.563 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:36:44.561+0000 7f0185235640 10 monclient: get_monmap_and_config success 2024-03-25T18:36:44.563 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:36:44.561+0000 7f0185235640 4 set_mon_vals no callback set 2024-03-25T18:36:44.564 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:36:44.562+0000 7f0185235640 10 monclient: shutdown 2024-03-25T18:36:44.564 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:36:44.562+0000 7f0185235640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:36:44.564 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:36:44.562+0000 7f0185235640 1 -- 172.21.15.1:0/716371401 >> v1:172.21.15.115:6789/0 conn(0x7f01800b4e70 legacy=0x7f018013f940 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:36:44.564 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:36:44.562+0000 7f0185235640 1 -- 172.21.15.1:0/716371401 shutdown_connections 2024-03-25T18:36:44.564 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:36:44.562+0000 7f0185235640 1 -- 172.21.15.1:0/716371401 >> 172.21.15.1:0/716371401 conn(0x7f01800617a0 msgr2=0x7f0180061bb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:36:44.564 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:36:44.562+0000 7f0185235640 1 -- 172.21.15.1:0/716371401 shutdown_connections 2024-03-25T18:36:44.564 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:36:44.562+0000 7f0185235640 1 -- 172.21.15.1:0/716371401 wait complete. 2024-03-25T18:36:44.564 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:36:44.563+0000 7f0185235640 10 monclient: build_initial_monmap 2024-03-25T18:36:44.564 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:36:44.563+0000 7f0185235640 10 monclient: monmap: 2024-03-25T18:36:44.564 INFO:teuthology.misc.health.smithi001.stderr:epoch 0 2024-03-25T18:36:44.564 INFO:teuthology.misc.health.smithi001.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T18:36:44.565 INFO:teuthology.misc.health.smithi001.stderr:last_changed 0.000000 2024-03-25T18:36:44.565 INFO:teuthology.misc.health.smithi001.stderr:created 0.000000 2024-03-25T18:36:44.565 INFO:teuthology.misc.health.smithi001.stderr:min_mon_release 0 (unknown) 2024-03-25T18:36:44.565 INFO:teuthology.misc.health.smithi001.stderr:election_strategy: 1 2024-03-25T18:36:44.565 INFO:teuthology.misc.health.smithi001.stderr:0: v1:172.21.15.1:6789/0 mon.noname-a 2024-03-25T18:36:44.565 INFO:teuthology.misc.health.smithi001.stderr:1: v1:172.21.15.115:6789/0 mon.noname-b 2024-03-25T18:36:44.565 INFO:teuthology.misc.health.smithi001.stderr:2: v1:172.21.15.1:6790/0 mon.noname-c 2024-03-25T18:36:44.565 INFO:teuthology.misc.health.smithi001.stderr: 2024-03-25T18:36:44.565 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:36:44.563+0000 7f0185235640 1 librados: starting msgr at 2024-03-25T18:36:44.565 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:36:44.563+0000 7f0185235640 1 librados: starting objecter 2024-03-25T18:36:44.567 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:36:44.563+0000 7f0185235640 1 Processor -- start 2024-03-25T18:36:44.567 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:36:44.563+0000 7f0185235640 1 -- start start 2024-03-25T18:36:44.567 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:36:44.563+0000 7f0185235640 1 librados: setting wanted keys 2024-03-25T18:36:44.567 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:36:44.563+0000 7f0185235640 1 librados: calling monclient init 2024-03-25T18:36:44.567 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:36:44.563+0000 7f0185235640 10 monclient: init 2024-03-25T18:36:44.567 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:36:44.564+0000 7f0185235640 10 monclient: _reopen_session rank -1 2024-03-25T18:36:44.567 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:36:44.564+0000 7f0185235640 10 monclient: _add_conns ranks=[0,2,1] 2024-03-25T18:36:44.567 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:36:44.564+0000 7f0185235640 10 monclient(hunting): picked mon.noname-a con 0x7f01800b1b00 addr v1:172.21.15.1:6789/0 2024-03-25T18:36:44.567 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:36:44.564+0000 7f0185235640 10 monclient(hunting): picked mon.noname-c con 0x7f01800b44b0 addr v1:172.21.15.1:6790/0 2024-03-25T18:36:44.567 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:36:44.565+0000 7f0185235640 10 monclient(hunting): picked mon.noname-b con 0x7f01800b4e70 addr v1:172.21.15.115:6789/0 2024-03-25T18:36:44.567 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:36:44.565+0000 7f0185235640 1 -- --> v1:172.21.15.1:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f01800700b0 con 0x7f01800b1b00 2024-03-25T18:36:44.568 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:36:44.565+0000 7f0185235640 1 -- --> v1:172.21.15.115:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0180070390 con 0x7f01800b4e70 2024-03-25T18:36:44.568 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:36:44.565+0000 7f0185235640 1 -- --> v1:172.21.15.1:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0180072000 con 0x7f01800b44b0 2024-03-25T18:36:44.568 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:36:44.565+0000 7f0185235640 10 monclient(hunting): _renew_subs 2024-03-25T18:36:44.568 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:36:44.565+0000 7f0185235640 10 monclient(hunting): authenticate will time out at 8085.034909s 2024-03-25T18:36:44.573 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:36:44.572+0000 7f017f59e640 1 --1- >> v1:172.21.15.115:6789/0 conn(0x7f01800b4e70 0x7f01801e63d0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.115:6789/0 says I am v1:172.21.15.1:55756/0 (socket says 172.21.15.1:55756) 2024-03-25T18:36:44.573 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:36:44.572+0000 7f017f59e640 1 -- 172.21.15.1:0/3734663137 learned_addr learned my addr 172.21.15.1:0/3734663137 (peer_addr_for_me v1:172.21.15.1:0/0) 2024-03-25T18:36:44.575 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:36:44.575+0000 7f01777fe640 10 client.?.objecter ms_handle_connect 0x7f01800b44b0 2024-03-25T18:36:44.575 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:36:44.575+0000 7f01777fe640 10 client.?.objecter resend_mon_ops 2024-03-25T18:36:44.577 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:36:44.576+0000 7f01777fe640 10 client.?.objecter ms_handle_connect 0x7f01800b4e70 2024-03-25T18:36:44.577 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:36:44.576+0000 7f01777fe640 10 client.?.objecter resend_mon_ops 2024-03-25T18:36:44.585 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:36:44.584+0000 7f01777fe640 1 -- 172.21.15.1:0/3734663137 <== mon.1 v1:172.21.15.115:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3950651764 0 0) 0x7f0180070390 con 0x7f01800b4e70 2024-03-25T18:36:44.585 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:36:44.584+0000 7f01777fe640 10 monclient(hunting): _negotiate 2024-03-25T18:36:44.585 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:36:44.584+0000 7f01777fe640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:36:44.585 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:36:44.584+0000 7f01777fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:36:44.585 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:36:44.584+0000 7f01777fe640 10 monclient(hunting): my global_id is 4663 2024-03-25T18:36:44.585 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:36:44.584+0000 7f01777fe640 1 -- 172.21.15.1:0/3734663137 --> v1:172.21.15.115:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f0160004eb0 con 0x7f01800b4e70 2024-03-25T18:36:44.588 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:36:44.587+0000 7f01777fe640 1 -- 172.21.15.1:0/3734663137 <== mon.2 v1:172.21.15.1:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2543544847 0 0) 0x7f0180072000 con 0x7f01800b44b0 2024-03-25T18:36:44.588 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:36:44.587+0000 7f01777fe640 10 monclient(hunting): _negotiate 2024-03-25T18:36:44.588 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:36:44.587+0000 7f01777fe640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:36:44.588 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:36:44.587+0000 7f01777fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:36:44.588 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:36:44.587+0000 7f01777fe640 10 monclient(hunting): my global_id is 4661 2024-03-25T18:36:44.588 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:36:44.587+0000 7f01777fe640 1 -- 172.21.15.1:0/3734663137 --> v1:172.21.15.1:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f0180070390 con 0x7f01800b44b0 2024-03-25T18:36:44.597 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:36:44.596+0000 7f01777fe640 10 client.?.objecter ms_handle_connect 0x7f01800b1b00 2024-03-25T18:36:44.597 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:36:44.596+0000 7f01777fe640 10 client.?.objecter resend_mon_ops 2024-03-25T18:36:44.603 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:36:44.602+0000 7f01777fe640 1 -- 172.21.15.1:0/3734663137 <== mon.1 v1:172.21.15.115:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 359994516 0 0) 0x7f0160004eb0 con 0x7f01800b4e70 2024-03-25T18:36:44.603 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:36:44.602+0000 7f01777fe640 1 -- 172.21.15.1:0/3734663137 --> v1:172.21.15.115:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f0180072000 con 0x7f01800b4e70 2024-03-25T18:36:44.603 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:36:44.602+0000 7f01777fe640 1 -- 172.21.15.1:0/3734663137 <== mon.1 v1:172.21.15.115:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1586965025 0 0) 0x7f0178001d30 con 0x7f01800b4e70 2024-03-25T18:36:44.603 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:36:44.602+0000 7f01777fe640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:36:44.603 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:36:44.602+0000 7f01777fe640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-03-25T18:36:44.603 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:36:44.602+0000 7f01777fe640 10 monclient(hunting): dump: 2024-03-25T18:36:44.603 INFO:teuthology.misc.health.smithi001.stderr:epoch 1 2024-03-25T18:36:44.603 INFO:teuthology.misc.health.smithi001.stderr:fsid c8ea72cf-9e99-4383-9b52-6c23d365fdfc 2024-03-25T18:36:44.603 INFO:teuthology.misc.health.smithi001.stderr:last_changed 2024-03-25T18:25:50.510519+0000 2024-03-25T18:36:44.603 INFO:teuthology.misc.health.smithi001.stderr:created 2024-03-25T18:25:50.510519+0000 2024-03-25T18:36:44.603 INFO:teuthology.misc.health.smithi001.stderr:min_mon_release 19 (squid) 2024-03-25T18:36:44.603 INFO:teuthology.misc.health.smithi001.stderr:election_strategy: 3 2024-03-25T18:36:44.603 INFO:teuthology.misc.health.smithi001.stderr:0: v1:172.21.15.1:6789/0 mon.a 2024-03-25T18:36:44.603 INFO:teuthology.misc.health.smithi001.stderr:1: v1:172.21.15.115:6789/0 mon.b 2024-03-25T18:36:44.604 INFO:teuthology.misc.health.smithi001.stderr:2: v1:172.21.15.1:6790/0 mon.c 2024-03-25T18:36:44.604 INFO:teuthology.misc.health.smithi001.stderr: 2024-03-25T18:36:44.604 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:36:44.602+0000 7f01777fe640 10 monclient(hunting): _finish_auth 0 2024-03-25T18:36:44.607 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:36:44.605+0000 7f01777fe640 1 -- 172.21.15.1:0/3734663137 <== mon.0 v1:172.21.15.1:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1535449957 0 0) 0x7f01800700b0 con 0x7f01800b1b00 2024-03-25T18:36:44.607 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:36:44.605+0000 7f01777fe640 10 monclient(hunting): _negotiate 2024-03-25T18:36:44.607 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:36:44.605+0000 7f01777fe640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:36:44.607 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:36:44.605+0000 7f01777fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:36:44.607 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:36:44.605+0000 7f01777fe640 10 monclient(hunting): my global_id is 4650 2024-03-25T18:36:44.607 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:36:44.606+0000 7f01777fe640 1 -- 172.21.15.1:0/3734663137 --> v1:172.21.15.1:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f0160004eb0 con 0x7f01800b1b00 2024-03-25T18:36:44.607 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:36:44.606+0000 7f01777fe640 1 -- 172.21.15.1:0/3734663137 <== mon.2 v1:172.21.15.1:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 745919261 0 0) 0x7f0180070390 con 0x7f01800b44b0 2024-03-25T18:36:44.607 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:36:44.606+0000 7f01777fe640 1 -- 172.21.15.1:0/3734663137 --> v1:172.21.15.1:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f01800700b0 con 0x7f01800b44b0 2024-03-25T18:36:44.607 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:36:44.606+0000 7f01777fe640 1 -- 172.21.15.1:0/3734663137 <== mon.2 v1:172.21.15.1:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1586965025 0 0) 0x7f0168001930 con 0x7f01800b44b0 2024-03-25T18:36:44.607 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:36:44.606+0000 7f01777fe640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:36:44.607 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:36:44.606+0000 7f01777fe640 10 monclient(hunting): got monmap 1 from mon.c (according to old e1) 2024-03-25T18:36:44.607 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:36:44.606+0000 7f01777fe640 10 monclient(hunting): dump: 2024-03-25T18:36:44.608 INFO:teuthology.misc.health.smithi001.stderr:epoch 1 2024-03-25T18:36:44.608 INFO:teuthology.misc.health.smithi001.stderr:fsid c8ea72cf-9e99-4383-9b52-6c23d365fdfc 2024-03-25T18:36:44.608 INFO:teuthology.misc.health.smithi001.stderr:last_changed 2024-03-25T18:25:50.510519+0000 2024-03-25T18:36:44.608 INFO:teuthology.misc.health.smithi001.stderr:created 2024-03-25T18:25:50.510519+0000 2024-03-25T18:36:44.608 INFO:teuthology.misc.health.smithi001.stderr:min_mon_release 19 (squid) 2024-03-25T18:36:44.608 INFO:teuthology.misc.health.smithi001.stderr:election_strategy: 3 2024-03-25T18:36:44.608 INFO:teuthology.misc.health.smithi001.stderr:0: v1:172.21.15.1:6789/0 mon.a 2024-03-25T18:36:44.608 INFO:teuthology.misc.health.smithi001.stderr:1: v1:172.21.15.115:6789/0 mon.b 2024-03-25T18:36:44.608 INFO:teuthology.misc.health.smithi001.stderr:2: v1:172.21.15.1:6790/0 mon.c 2024-03-25T18:36:44.608 INFO:teuthology.misc.health.smithi001.stderr: 2024-03-25T18:36:44.612 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:36:44.611+0000 7f01777fe640 1 -- 172.21.15.1:0/3734663137 <== mon.1 v1:172.21.15.115:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 3241799798 0 0) 0x7f0180072000 con 0x7f01800b4e70 2024-03-25T18:36:44.612 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:36:44.611+0000 7f01777fe640 1 -- 172.21.15.1:0/3734663137 >> v1:172.21.15.1:6790/0 conn(0x7f01800b44b0 legacy=0x7f0180071280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:36:44.613 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:36:44.611+0000 7f01777fe640 1 -- 172.21.15.1:0/3734663137 >> v1:172.21.15.1:6789/0 conn(0x7f01800b1b00 legacy=0x7f0180070b70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:36:44.613 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:36:44.611+0000 7f01777fe640 10 monclient: _finish_hunting 0 2024-03-25T18:36:44.613 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:36:44.611+0000 7f01777fe640 1 monclient: found mon.b 2024-03-25T18:36:44.613 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:36:44.611+0000 7f01777fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:36:44.613 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:36:44.611+0000 7f01777fe640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.115:6789/0 2024-03-25T18:36:44.613 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:36:44.611+0000 7f01777fe640 1 -- 172.21.15.1:0/3734663137 --> v1:172.21.15.115:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f018006f1c0 con 0x7f01800b4e70 2024-03-25T18:36:44.613 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:36:44.611+0000 7f01777fe640 10 monclient: _finish_auth 0 2024-03-25T18:36:44.613 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:36:44.611+0000 7f01777fe640 10 monclient: _check_auth_tickets 2024-03-25T18:36:44.613 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:36:44.611+0000 7f01777fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:36:44.613 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:36:44.611+0000 7f0185235640 5 monclient: authenticate success, global_id 4663 2024-03-25T18:36:44.613 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:36:44.611+0000 7f0185235640 10 monclient: _renew_subs 2024-03-25T18:36:44.613 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:36:44.611+0000 7f0185235640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.115:6789/0 2024-03-25T18:36:44.613 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:36:44.611+0000 7f0185235640 1 -- 172.21.15.1:0/3734663137 --> v1:172.21.15.115:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f018014ec10 con 0x7f01800b4e70 2024-03-25T18:36:44.614 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:36:44.611+0000 7f0185235640 10 client.4663.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T18:36:44.614 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:36:44.611+0000 7f0185235640 10 monclient: _renew_subs 2024-03-25T18:36:44.614 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:36:44.611+0000 7f0185235640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.115:6789/0 2024-03-25T18:36:44.614 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:36:44.611+0000 7f0185235640 1 -- 172.21.15.1:0/3734663137 --> v1:172.21.15.115:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f01801e7af0 con 0x7f01800b4e70 2024-03-25T18:36:44.614 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:36:44.611+0000 7f0185235640 1 librados: init done 2024-03-25T18:36:44.614 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:36:44.612+0000 7f0185235640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T18:36:44.614 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:36:44.612+0000 7f0185235640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:36:44.615 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:36:44.612+0000 7f0185235640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.115:6789/0 2024-03-25T18:36:44.615 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:36:44.612+0000 7f0185235640 1 -- 172.21.15.1:0/3734663137 --> v1:172.21.15.115:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f018004f730 con 0x7f01800b4e70 2024-03-25T18:36:44.633 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:36:44.632+0000 7f01777fe640 1 -- 172.21.15.1:0/3734663137 <== mon.1 v1:172.21.15.115:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f0178003990 con 0x7f01800b4e70 2024-03-25T18:36:44.633 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:36:44.632+0000 7f01777fe640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:36:44.633 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:36:44.632+0000 7f01777fe640 1 -- 172.21.15.1:0/3734663137 <== mon.1 v1:172.21.15.115:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1586965025 0 0) 0x7f0178003ca0 con 0x7f01800b4e70 2024-03-25T18:36:44.633 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:36:44.632+0000 7f01777fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:36:44.633 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:36:44.632+0000 7f01777fe640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-25T18:36:44.633 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:36:44.632+0000 7f01777fe640 10 monclient: dump: 2024-03-25T18:36:44.633 INFO:teuthology.misc.health.smithi001.stderr:epoch 1 2024-03-25T18:36:44.633 INFO:teuthology.misc.health.smithi001.stderr:fsid c8ea72cf-9e99-4383-9b52-6c23d365fdfc 2024-03-25T18:36:44.634 INFO:teuthology.misc.health.smithi001.stderr:last_changed 2024-03-25T18:25:50.510519+0000 2024-03-25T18:36:44.634 INFO:teuthology.misc.health.smithi001.stderr:created 2024-03-25T18:25:50.510519+0000 2024-03-25T18:36:44.634 INFO:teuthology.misc.health.smithi001.stderr:min_mon_release 19 (squid) 2024-03-25T18:36:44.634 INFO:teuthology.misc.health.smithi001.stderr:election_strategy: 3 2024-03-25T18:36:44.634 INFO:teuthology.misc.health.smithi001.stderr:0: v1:172.21.15.1:6789/0 mon.a 2024-03-25T18:36:44.634 INFO:teuthology.misc.health.smithi001.stderr:1: v1:172.21.15.115:6789/0 mon.b 2024-03-25T18:36:44.634 INFO:teuthology.misc.health.smithi001.stderr:2: v1:172.21.15.1:6790/0 mon.c 2024-03-25T18:36:44.634 INFO:teuthology.misc.health.smithi001.stderr: 2024-03-25T18:36:44.634 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:36:44.632+0000 7f017dd9b640 4 set_mon_vals no callback set 2024-03-25T18:36:44.635 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:36:44.633+0000 7f01777fe640 1 -- 172.21.15.1:0/3734663137 <== mon.1 v1:172.21.15.115:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 21787975 0 0) 0x7f0178003e80 con 0x7f01800b4e70 2024-03-25T18:36:44.635 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:36:44.634+0000 7f017ed9d640 10 monclient: get_auth_request con 0x7f016006df70 auth_method 0 2024-03-25T18:36:44.635 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:36:44.634+0000 7f01777fe640 10 client.4663.objecter ms_handle_connect 0x7f016006df70 2024-03-25T18:36:44.636 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:36:44.635+0000 7f01777fe640 1 -- 172.21.15.1:0/3734663137 <== mon.1 v1:172.21.15.115:6789/0 8 ==== osd_map(11..11 src has 1..11) v4 ==== 1645+0+0 (unknown 2504898309 0 0) 0x7f0178054af0 con 0x7f01800b4e70 2024-03-25T18:36:44.636 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:36:44.635+0000 7f01777fe640 10 client.4663.objecter ms_dispatch 0x7f0180000b90 osd_map(11..11 src has 1..11) v4 2024-03-25T18:36:44.636 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:36:44.635+0000 7f01777fe640 3 client.4663.objecter handle_osd_map got epochs [11,11] > 0 2024-03-25T18:36:44.636 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:36:44.635+0000 7f01777fe640 3 client.4663.objecter handle_osd_map decoding full epoch 11 2024-03-25T18:36:44.636 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:36:44.635+0000 7f01777fe640 20 client.4663.objecter dump_active .. 0 homeless 2024-03-25T18:36:44.887 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:36:44.887+0000 7f01777fe640 1 -- 172.21.15.1:0/3734663137 <== mon.1 v1:172.21.15.115:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f0178050430 con 0x7f01800b4e70 2024-03-25T18:36:44.888 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:36:44.887+0000 7f01777fe640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:36:44.888 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:36:44.887+0000 7f01777fe640 10 monclient: _finish_command 1 = system:0 2024-03-25T18:36:45.104 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:36:45.104+0000 7f0185235640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:36:45.105 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:36:45.104+0000 7f0185235640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:36:45.105 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:36:45.104+0000 7f0185235640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.115:6789/0 2024-03-25T18:36:45.105 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:36:45.104+0000 7f0185235640 1 -- 172.21.15.1:0/3734663137 --> v1:172.21.15.115:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f01800aa090 con 0x7f01800b4e70 2024-03-25T18:36:45.126 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:36:45.125+0000 7f01777fe640 1 -- 172.21.15.1:0/3734663137 <== mon.1 v1:172.21.15.115:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v11) v1 ==== 74+0+7100 (unknown 238102980 0 2126609903) 0x7f0178052530 con 0x7f01800b4e70 2024-03-25T18:36:45.126 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:36:45.125+0000 7f01777fe640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:36:45.126 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:36:45.125+0000 7f01777fe640 10 monclient: _finish_command 2 = system:0 2024-03-25T18:36:45.127 INFO:teuthology.misc.health.smithi001.stdout: 2024-03-25T18:36:45.127 INFO:teuthology.misc.health.smithi001.stdout:{"epoch":11,"fsid":"c8ea72cf-9e99-4383-9b52-6c23d365fdfc","created":"2024-03-25T18:26:38.856638+0000","modified":"2024-03-25T18:36:43.033551+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T18:36:43.033551+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"b9f34922-3a84-49be-8426-807c1eceaefc","up":0,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["autoout","exists","new"]},{"osd":1,"uuid":"42bf306f-55e3-4634-9874-8b4b73ac6644","up":0,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["autoout","exists","new"]},{"osd":2,"uuid":"292cb23a-18a9-4207-a7cc-4d5727feaec0","up":0,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["autoout","exists","new"]},{"osd":3,"uuid":"45e0e9ab-898d-4673-9590-b4117bc5e84a","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"831a5a09-73ab-4a61-9872-088e76bdbc66","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"88cd903d-3b55-4840-9396-6e3cfe005282","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"89db7355-bc9d-46f4-8ab0-e8d9e744e026","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"cac47bee-0914-4f08-896f-348838882866","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":65536,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":65536,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":65536,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T18:36:45.129 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:36:45.128+0000 7f0185235640 10 librados: watch_flush enter 2024-03-25T18:36:45.129 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:36:45.128+0000 7f0185235640 10 librados: watch_flush exit 2024-03-25T18:36:45.129 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:36:45.128+0000 7f0185235640 20 client.4663.objecter shutdown clearing up homeless session... 2024-03-25T18:36:45.129 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:36:45.128+0000 7f0185235640 10 client.4663.objecter successfully canceled tick 2024-03-25T18:36:45.130 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:36:45.128+0000 7f0185235640 1 -- 172.21.15.1:0/3734663137 >> v1:172.21.15.115:6800/4237379176 conn(0x7f016006df70 legacy=0x7f0160070430 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:36:45.130 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:36:45.128+0000 7f0185235640 10 monclient: shutdown 2024-03-25T18:36:45.130 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:36:45.128+0000 7f0185235640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:36:45.130 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:36:45.128+0000 7f0185235640 1 -- 172.21.15.1:0/3734663137 >> v1:172.21.15.115:6789/0 conn(0x7f01800b4e70 legacy=0x7f01801e63d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:36:45.130 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:36:45.128+0000 7f0185235640 1 -- 172.21.15.1:0/3734663137 shutdown_connections 2024-03-25T18:36:45.130 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:36:45.128+0000 7f0185235640 1 -- 172.21.15.1:0/3734663137 >> 172.21.15.1:0/3734663137 conn(0x7f01800617a0 msgr2=0x7f01800b2710 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:36:45.130 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:36:45.128+0000 7f0185235640 1 -- 172.21.15.1:0/3734663137 shutdown_connections 2024-03-25T18:36:45.130 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:36:45.128+0000 7f0185235640 1 -- 172.21.15.1:0/3734663137 wait complete. 2024-03-25T18:36:45.130 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:36:45.128+0000 7f0185235640 1 librados: shutdown 2024-03-25T18:36:45.142 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T18:36:51.949 DEBUG:teuthology.orchestra.run.smithi001:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T18:36:52.064 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:36:52.063+0000 7f297fbcf640 10 monclient: get_monmap_and_config 2024-03-25T18:36:52.064 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:36:52.063+0000 7f297fbcf640 10 monclient: build_initial_monmap 2024-03-25T18:36:52.064 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:36:52.063+0000 7f297fbcf640 10 monclient: monmap: 2024-03-25T18:36:52.064 INFO:teuthology.misc.health.smithi001.stderr:epoch 0 2024-03-25T18:36:52.064 INFO:teuthology.misc.health.smithi001.stderr:fsid c8ea72cf-9e99-4383-9b52-6c23d365fdfc 2024-03-25T18:36:52.064 INFO:teuthology.misc.health.smithi001.stderr:last_changed 2024-03-25T18:36:52.063975+0000 2024-03-25T18:36:52.064 INFO:teuthology.misc.health.smithi001.stderr:created 2024-03-25T18:36:52.063975+0000 2024-03-25T18:36:52.064 INFO:teuthology.misc.health.smithi001.stderr:min_mon_release 0 (unknown) 2024-03-25T18:36:52.064 INFO:teuthology.misc.health.smithi001.stderr:election_strategy: 3 2024-03-25T18:36:52.064 INFO:teuthology.misc.health.smithi001.stderr:0: v1:172.21.15.1:6789/0 mon.noname-a 2024-03-25T18:36:52.065 INFO:teuthology.misc.health.smithi001.stderr:1: v1:172.21.15.115:6789/0 mon.noname-c 2024-03-25T18:36:52.065 INFO:teuthology.misc.health.smithi001.stderr:2: v1:172.21.15.1:6790/0 mon.noname-b 2024-03-25T18:36:52.065 INFO:teuthology.misc.health.smithi001.stderr: 2024-03-25T18:36:52.066 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:36:52.064+0000 7f297fbcf640 1 Processor -- start 2024-03-25T18:36:52.066 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:36:52.064+0000 7f297fbcf640 1 -- start start 2024-03-25T18:36:52.066 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:36:52.064+0000 7f297fbcf640 10 monclient: init 2024-03-25T18:36:52.066 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:36:52.065+0000 7f297fbcf640 10 monclient: _reopen_session rank -1 2024-03-25T18:36:52.066 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:36:52.065+0000 7f297fbcf640 10 monclient: _add_conns ranks=[2,1,0] 2024-03-25T18:36:52.066 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:36:52.065+0000 7f297fbcf640 10 monclient(hunting): picked mon.noname-b con 0x7f29780b4ff0 addr v1:172.21.15.1:6790/0 2024-03-25T18:36:52.066 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:36:52.065+0000 7f297fbcf640 10 monclient(hunting): picked mon.noname-c con 0x7f29780b4670 addr v1:172.21.15.115:6789/0 2024-03-25T18:36:52.066 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:36:52.065+0000 7f297fbcf640 10 monclient(hunting): picked mon.noname-a con 0x7f297813c480 addr v1:172.21.15.1:6789/0 2024-03-25T18:36:52.066 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:36:52.065+0000 7f297fbcf640 1 -- --> v1:172.21.15.1:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f29780ab0e0 con 0x7f297813c480 2024-03-25T18:36:52.066 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:36:52.065+0000 7f297fbcf640 1 -- --> v1:172.21.15.115:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f29780ab3c0 con 0x7f29780b4670 2024-03-25T18:36:52.066 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:36:52.065+0000 7f297fbcf640 1 -- --> v1:172.21.15.1:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f29780a9f30 con 0x7f29780b4ff0 2024-03-25T18:36:52.067 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:36:52.065+0000 7f297fbcf640 10 monclient(hunting): _renew_subs 2024-03-25T18:36:52.067 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:36:52.065+0000 7f297fbcf640 10 monclient(hunting): authenticate will time out at 8092.534977s 2024-03-25T18:36:52.078 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:36:52.078+0000 7f297d16a640 1 --1- >> v1:172.21.15.115:6789/0 conn(0x7f29780b4670 0x7f29781425c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.115:6789/0 says I am v1:172.21.15.1:60530/0 (socket says 172.21.15.1:60530) 2024-03-25T18:36:52.078 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:36:52.078+0000 7f297d16a640 1 -- 172.21.15.1:0/1626482793 learned_addr learned my addr 172.21.15.1:0/1626482793 (peer_addr_for_me v1:172.21.15.1:0/0) 2024-03-25T18:36:52.087 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:36:52.086+0000 7f297c969640 1 -- 172.21.15.1:0/1626482793 <== mon.1 v1:172.21.15.115:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3925505173 0 0) 0x7f29780ab3c0 con 0x7f29780b4670 2024-03-25T18:36:52.087 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:36:52.087+0000 7f297c969640 10 monclient(hunting): _negotiate 2024-03-25T18:36:52.087 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:36:52.087+0000 7f297c969640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:36:52.087 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:36:52.087+0000 7f297c969640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:36:52.087 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:36:52.087+0000 7f297c969640 10 monclient(hunting): my global_id is 4666 2024-03-25T18:36:52.087 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:36:52.087+0000 7f297c969640 1 -- 172.21.15.1:0/1626482793 --> v1:172.21.15.115:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f2964003520 con 0x7f29780b4670 2024-03-25T18:36:52.091 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:36:52.090+0000 7f297c969640 1 -- 172.21.15.1:0/1626482793 <== mon.1 v1:172.21.15.115:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3272991503 0 0) 0x7f2964003520 con 0x7f29780b4670 2024-03-25T18:36:52.091 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:36:52.090+0000 7f297c969640 1 -- 172.21.15.1:0/1626482793 >> v1:172.21.15.1:6790/0 conn(0x7f29780b4ff0 legacy=0x7f297813bd70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-25T18:36:52.091 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:36:52.090+0000 7f297c969640 1 -- 172.21.15.1:0/1626482793 >> v1:172.21.15.1:6789/0 conn(0x7f297813c480 legacy=0x7f297814dc00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:36:52.091 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:36:52.090+0000 7f297c969640 10 monclient: _finish_hunting 0 2024-03-25T18:36:52.091 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:36:52.090+0000 7f297c969640 1 monclient: found mon.noname-c 2024-03-25T18:36:52.091 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:36:52.090+0000 7f297c969640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:36:52.091 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:36:52.090+0000 7f297c969640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.115:6789/0 2024-03-25T18:36:52.091 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:36:52.090+0000 7f297c969640 1 -- 172.21.15.1:0/1626482793 --> v1:172.21.15.115:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f29780a9b70 con 0x7f29780b4670 2024-03-25T18:36:52.091 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:36:52.090+0000 7f297c969640 10 monclient: _finish_auth 0 2024-03-25T18:36:52.092 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:36:52.090+0000 7f297c969640 10 monclient: _check_auth_tickets 2024-03-25T18:36:52.092 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:36:52.090+0000 7f297c969640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:36:52.092 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:36:52.090+0000 7f297fbcf640 5 monclient: authenticate success, global_id 4666 2024-03-25T18:36:52.092 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:36:52.090+0000 7f297fbcf640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T18:36:52.092 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:36:52.090+0000 7f297c969640 1 -- 172.21.15.1:0/1626482793 <== mon.1 v1:172.21.15.115:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1586965025 0 0) 0x7f2968001690 con 0x7f29780b4670 2024-03-25T18:36:52.092 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:36:52.090+0000 7f297c969640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:36:52.092 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:36:52.090+0000 7f297c969640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-03-25T18:36:52.092 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:36:52.090+0000 7f297c969640 10 monclient: dump: 2024-03-25T18:36:52.092 INFO:teuthology.misc.health.smithi001.stderr:epoch 1 2024-03-25T18:36:52.092 INFO:teuthology.misc.health.smithi001.stderr:fsid c8ea72cf-9e99-4383-9b52-6c23d365fdfc 2024-03-25T18:36:52.092 INFO:teuthology.misc.health.smithi001.stderr:last_changed 2024-03-25T18:25:50.510519+0000 2024-03-25T18:36:52.092 INFO:teuthology.misc.health.smithi001.stderr:created 2024-03-25T18:25:50.510519+0000 2024-03-25T18:36:52.092 INFO:teuthology.misc.health.smithi001.stderr:min_mon_release 19 (squid) 2024-03-25T18:36:52.092 INFO:teuthology.misc.health.smithi001.stderr:election_strategy: 3 2024-03-25T18:36:52.092 INFO:teuthology.misc.health.smithi001.stderr:0: v1:172.21.15.1:6789/0 mon.a 2024-03-25T18:36:52.093 INFO:teuthology.misc.health.smithi001.stderr:1: v1:172.21.15.115:6789/0 mon.b 2024-03-25T18:36:52.093 INFO:teuthology.misc.health.smithi001.stderr:2: v1:172.21.15.1:6790/0 mon.c 2024-03-25T18:36:52.093 INFO:teuthology.misc.health.smithi001.stderr: 2024-03-25T18:36:52.093 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:36:52.090+0000 7f297fbcf640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T18:36:52.094 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:36:52.094+0000 7f297c969640 1 -- 172.21.15.1:0/1626482793 <== mon.1 v1:172.21.15.115:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f2968001960 con 0x7f29780b4670 2024-03-25T18:36:52.094 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:36:52.094+0000 7f297c969640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:36:52.095 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:36:52.094+0000 7f297c969640 1 -- 172.21.15.1:0/1626482793 <== mon.1 v1:172.21.15.115:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1586965025 0 0) 0x7f2968001cf0 con 0x7f29780b4670 2024-03-25T18:36:52.095 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:36:52.094+0000 7f297c969640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:36:52.095 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:36:52.094+0000 7f297c969640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-25T18:36:52.095 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:36:52.094+0000 7f297c969640 10 monclient: dump: 2024-03-25T18:36:52.095 INFO:teuthology.misc.health.smithi001.stderr:epoch 1 2024-03-25T18:36:52.095 INFO:teuthology.misc.health.smithi001.stderr:fsid c8ea72cf-9e99-4383-9b52-6c23d365fdfc 2024-03-25T18:36:52.095 INFO:teuthology.misc.health.smithi001.stderr:last_changed 2024-03-25T18:25:50.510519+0000 2024-03-25T18:36:52.095 INFO:teuthology.misc.health.smithi001.stderr:created 2024-03-25T18:25:50.510519+0000 2024-03-25T18:36:52.095 INFO:teuthology.misc.health.smithi001.stderr:min_mon_release 19 (squid) 2024-03-25T18:36:52.095 INFO:teuthology.misc.health.smithi001.stderr:election_strategy: 3 2024-03-25T18:36:52.095 INFO:teuthology.misc.health.smithi001.stderr:0: v1:172.21.15.1:6789/0 mon.a 2024-03-25T18:36:52.095 INFO:teuthology.misc.health.smithi001.stderr:1: v1:172.21.15.115:6789/0 mon.b 2024-03-25T18:36:52.095 INFO:teuthology.misc.health.smithi001.stderr:2: v1:172.21.15.1:6790/0 mon.c 2024-03-25T18:36:52.095 INFO:teuthology.misc.health.smithi001.stderr: 2024-03-25T18:36:52.096 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:36:52.094+0000 7f297fbcf640 10 monclient: get_monmap_and_config success 2024-03-25T18:36:52.096 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:36:52.094+0000 7f297fbcf640 4 set_mon_vals no callback set 2024-03-25T18:36:52.097 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:36:52.094+0000 7f297fbcf640 10 monclient: shutdown 2024-03-25T18:36:52.097 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:36:52.094+0000 7f297fbcf640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:36:52.097 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:36:52.094+0000 7f297fbcf640 1 -- 172.21.15.1:0/1626482793 >> v1:172.21.15.115:6789/0 conn(0x7f29780b4670 legacy=0x7f29781425c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:36:52.097 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:36:52.095+0000 7f297fbcf640 1 -- 172.21.15.1:0/1626482793 shutdown_connections 2024-03-25T18:36:52.097 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:36:52.095+0000 7f297fbcf640 1 -- 172.21.15.1:0/1626482793 >> 172.21.15.1:0/1626482793 conn(0x7f29780617a0 msgr2=0x7f2978061bb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:36:52.097 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:36:52.095+0000 7f297fbcf640 1 -- 172.21.15.1:0/1626482793 shutdown_connections 2024-03-25T18:36:52.097 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:36:52.095+0000 7f297fbcf640 1 -- 172.21.15.1:0/1626482793 wait complete. 2024-03-25T18:36:52.097 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:36:52.096+0000 7f297fbcf640 10 monclient: build_initial_monmap 2024-03-25T18:36:52.098 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:36:52.096+0000 7f297fbcf640 10 monclient: monmap: 2024-03-25T18:36:52.098 INFO:teuthology.misc.health.smithi001.stderr:epoch 0 2024-03-25T18:36:52.098 INFO:teuthology.misc.health.smithi001.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T18:36:52.098 INFO:teuthology.misc.health.smithi001.stderr:last_changed 0.000000 2024-03-25T18:36:52.098 INFO:teuthology.misc.health.smithi001.stderr:created 0.000000 2024-03-25T18:36:52.098 INFO:teuthology.misc.health.smithi001.stderr:min_mon_release 0 (unknown) 2024-03-25T18:36:52.098 INFO:teuthology.misc.health.smithi001.stderr:election_strategy: 1 2024-03-25T18:36:52.098 INFO:teuthology.misc.health.smithi001.stderr:0: v1:172.21.15.1:6789/0 mon.noname-a 2024-03-25T18:36:52.098 INFO:teuthology.misc.health.smithi001.stderr:1: v1:172.21.15.115:6789/0 mon.noname-b 2024-03-25T18:36:52.098 INFO:teuthology.misc.health.smithi001.stderr:2: v1:172.21.15.1:6790/0 mon.noname-c 2024-03-25T18:36:52.098 INFO:teuthology.misc.health.smithi001.stderr: 2024-03-25T18:36:52.098 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:36:52.096+0000 7f297fbcf640 1 librados: starting msgr at 2024-03-25T18:36:52.098 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:36:52.096+0000 7f297fbcf640 1 librados: starting objecter 2024-03-25T18:36:52.099 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:36:52.096+0000 7f297fbcf640 1 Processor -- start 2024-03-25T18:36:52.099 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:36:52.096+0000 7f297fbcf640 1 -- start start 2024-03-25T18:36:52.099 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:36:52.096+0000 7f297fbcf640 1 librados: setting wanted keys 2024-03-25T18:36:52.099 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:36:52.096+0000 7f297fbcf640 1 librados: calling monclient init 2024-03-25T18:36:52.099 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:36:52.096+0000 7f297fbcf640 10 monclient: init 2024-03-25T18:36:52.100 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:36:52.097+0000 7f297fbcf640 10 monclient: _reopen_session rank -1 2024-03-25T18:36:52.100 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:36:52.097+0000 7f297fbcf640 10 monclient: _add_conns ranks=[2,0,1] 2024-03-25T18:36:52.100 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:36:52.097+0000 7f297fbcf640 10 monclient(hunting): picked mon.noname-c con 0x7f29780b4670 addr v1:172.21.15.1:6790/0 2024-03-25T18:36:52.100 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:36:52.097+0000 7f297fbcf640 10 monclient(hunting): picked mon.noname-a con 0x7f29780b4ff0 addr v1:172.21.15.1:6789/0 2024-03-25T18:36:52.100 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:36:52.097+0000 7f297fbcf640 10 monclient(hunting): picked mon.noname-b con 0x7f297813c480 addr v1:172.21.15.115:6789/0 2024-03-25T18:36:52.100 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:36:52.097+0000 7f297fbcf640 1 -- --> v1:172.21.15.1:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2978145830 con 0x7f29780b4ff0 2024-03-25T18:36:52.100 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:36:52.097+0000 7f297fbcf640 1 -- --> v1:172.21.15.115:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2978137ca0 con 0x7f297813c480 2024-03-25T18:36:52.100 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:36:52.097+0000 7f297fbcf640 1 -- --> v1:172.21.15.1:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2978145cc0 con 0x7f29780b4670 2024-03-25T18:36:52.100 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:36:52.097+0000 7f297fbcf640 10 monclient(hunting): _renew_subs 2024-03-25T18:36:52.100 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:36:52.097+0000 7f297fbcf640 10 monclient(hunting): authenticate will time out at 8092.567525s 2024-03-25T18:36:52.107 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:36:52.107+0000 7f297e16c640 1 --1- >> v1:172.21.15.115:6789/0 conn(0x7f297813c480 0x7f29781e2600 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.115:6789/0 says I am v1:172.21.15.1:60548/0 (socket says 172.21.15.1:60548) 2024-03-25T18:36:52.108 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:36:52.107+0000 7f297e16c640 1 -- 172.21.15.1:0/726347465 learned_addr learned my addr 172.21.15.1:0/726347465 (peer_addr_for_me v1:172.21.15.1:0/0) 2024-03-25T18:36:52.111 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:36:52.110+0000 7f296e7fc640 10 client.?.objecter ms_handle_connect 0x7f297813c480 2024-03-25T18:36:52.111 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:36:52.111+0000 7f296e7fc640 10 client.?.objecter resend_mon_ops 2024-03-25T18:36:52.116 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:36:52.116+0000 7f296e7fc640 10 client.?.objecter ms_handle_connect 0x7f29780b4ff0 2024-03-25T18:36:52.116 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:36:52.116+0000 7f296e7fc640 10 client.?.objecter resend_mon_ops 2024-03-25T18:36:52.127 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:36:52.127+0000 7f296e7fc640 10 client.?.objecter ms_handle_connect 0x7f29780b4670 2024-03-25T18:36:52.127 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:36:52.127+0000 7f296e7fc640 10 client.?.objecter resend_mon_ops 2024-03-25T18:36:52.128 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:36:52.127+0000 7f296e7fc640 1 -- 172.21.15.1:0/726347465 <== mon.1 v1:172.21.15.115:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2117456367 0 0) 0x7f2978137ca0 con 0x7f297813c480 2024-03-25T18:36:52.128 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:36:52.127+0000 7f296e7fc640 10 monclient(hunting): _negotiate 2024-03-25T18:36:52.128 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:36:52.127+0000 7f296e7fc640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:36:52.128 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:36:52.127+0000 7f296e7fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:36:52.128 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:36:52.127+0000 7f296e7fc640 10 monclient(hunting): my global_id is 4669 2024-03-25T18:36:52.128 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:36:52.128+0000 7f296e7fc640 1 -- 172.21.15.1:0/726347465 --> v1:172.21.15.115:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f295c004eb0 con 0x7f297813c480 2024-03-25T18:36:52.140 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:36:52.139+0000 7f296e7fc640 1 -- 172.21.15.1:0/726347465 <== mon.0 v1:172.21.15.1:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 446319306 0 0) 0x7f2978145830 con 0x7f29780b4ff0 2024-03-25T18:36:52.140 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:36:52.139+0000 7f296e7fc640 10 monclient(hunting): _negotiate 2024-03-25T18:36:52.140 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:36:52.139+0000 7f296e7fc640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:36:52.140 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:36:52.139+0000 7f296e7fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:36:52.140 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:36:52.139+0000 7f296e7fc640 10 monclient(hunting): my global_id is 4656 2024-03-25T18:36:52.140 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:36:52.139+0000 7f296e7fc640 1 -- 172.21.15.1:0/726347465 --> v1:172.21.15.1:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f2978137ca0 con 0x7f29780b4ff0 2024-03-25T18:36:52.143 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:36:52.142+0000 7f296e7fc640 1 -- 172.21.15.1:0/726347465 <== mon.2 v1:172.21.15.1:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2520441935 0 0) 0x7f2978145cc0 con 0x7f29780b4670 2024-03-25T18:36:52.143 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:36:52.142+0000 7f296e7fc640 10 monclient(hunting): _negotiate 2024-03-25T18:36:52.143 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:36:52.142+0000 7f296e7fc640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:36:52.143 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:36:52.142+0000 7f296e7fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:36:52.143 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:36:52.142+0000 7f296e7fc640 10 monclient(hunting): my global_id is 4664 2024-03-25T18:36:52.143 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:36:52.142+0000 7f296e7fc640 1 -- 172.21.15.1:0/726347465 --> v1:172.21.15.1:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f2978145830 con 0x7f29780b4670 2024-03-25T18:36:52.152 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:36:52.152+0000 7f296e7fc640 1 -- 172.21.15.1:0/726347465 <== mon.0 v1:172.21.15.1:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1439835416 0 0) 0x7f2978137ca0 con 0x7f29780b4ff0 2024-03-25T18:36:52.153 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:36:52.152+0000 7f296e7fc640 1 -- 172.21.15.1:0/726347465 --> v1:172.21.15.1:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f2978145cc0 con 0x7f29780b4ff0 2024-03-25T18:36:52.153 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:36:52.152+0000 7f296e7fc640 1 -- 172.21.15.1:0/726347465 <== mon.0 v1:172.21.15.1:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1586965025 0 0) 0x7f2968001650 con 0x7f29780b4ff0 2024-03-25T18:36:52.153 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:36:52.152+0000 7f296e7fc640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:36:52.153 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:36:52.152+0000 7f296e7fc640 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2024-03-25T18:36:52.153 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:36:52.152+0000 7f296e7fc640 10 monclient(hunting): dump: 2024-03-25T18:36:52.153 INFO:teuthology.misc.health.smithi001.stderr:epoch 1 2024-03-25T18:36:52.153 INFO:teuthology.misc.health.smithi001.stderr:fsid c8ea72cf-9e99-4383-9b52-6c23d365fdfc 2024-03-25T18:36:52.153 INFO:teuthology.misc.health.smithi001.stderr:last_changed 2024-03-25T18:25:50.510519+0000 2024-03-25T18:36:52.153 INFO:teuthology.misc.health.smithi001.stderr:created 2024-03-25T18:25:50.510519+0000 2024-03-25T18:36:52.153 INFO:teuthology.misc.health.smithi001.stderr:min_mon_release 19 (squid) 2024-03-25T18:36:52.153 INFO:teuthology.misc.health.smithi001.stderr:election_strategy: 3 2024-03-25T18:36:52.153 INFO:teuthology.misc.health.smithi001.stderr:0: v1:172.21.15.1:6789/0 mon.a 2024-03-25T18:36:52.153 INFO:teuthology.misc.health.smithi001.stderr:1: v1:172.21.15.115:6789/0 mon.b 2024-03-25T18:36:52.153 INFO:teuthology.misc.health.smithi001.stderr:2: v1:172.21.15.1:6790/0 mon.c 2024-03-25T18:36:52.154 INFO:teuthology.misc.health.smithi001.stderr: 2024-03-25T18:36:52.154 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:36:52.152+0000 7f296e7fc640 10 monclient(hunting): _finish_auth 0 2024-03-25T18:36:52.155 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:36:52.155+0000 7f296e7fc640 1 -- 172.21.15.1:0/726347465 <== mon.1 v1:172.21.15.115:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2901001912 0 0) 0x7f295c004eb0 con 0x7f297813c480 2024-03-25T18:36:52.155 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:36:52.155+0000 7f296e7fc640 1 -- 172.21.15.1:0/726347465 --> v1:172.21.15.115:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f2978137ca0 con 0x7f297813c480 2024-03-25T18:36:52.156 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:36:52.155+0000 7f296e7fc640 1 -- 172.21.15.1:0/726347465 <== mon.1 v1:172.21.15.115:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1586965025 0 0) 0x7f2970001c30 con 0x7f297813c480 2024-03-25T18:36:52.156 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:36:52.155+0000 7f296e7fc640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:36:52.156 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:36:52.155+0000 7f296e7fc640 10 monclient(hunting): got monmap 1 from mon.b (according to old e1) 2024-03-25T18:36:52.156 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:36:52.155+0000 7f296e7fc640 10 monclient(hunting): dump: 2024-03-25T18:36:52.156 INFO:teuthology.misc.health.smithi001.stderr:epoch 1 2024-03-25T18:36:52.156 INFO:teuthology.misc.health.smithi001.stderr:fsid c8ea72cf-9e99-4383-9b52-6c23d365fdfc 2024-03-25T18:36:52.156 INFO:teuthology.misc.health.smithi001.stderr:last_changed 2024-03-25T18:25:50.510519+0000 2024-03-25T18:36:52.156 INFO:teuthology.misc.health.smithi001.stderr:created 2024-03-25T18:25:50.510519+0000 2024-03-25T18:36:52.156 INFO:teuthology.misc.health.smithi001.stderr:min_mon_release 19 (squid) 2024-03-25T18:36:52.156 INFO:teuthology.misc.health.smithi001.stderr:election_strategy: 3 2024-03-25T18:36:52.156 INFO:teuthology.misc.health.smithi001.stderr:0: v1:172.21.15.1:6789/0 mon.a 2024-03-25T18:36:52.156 INFO:teuthology.misc.health.smithi001.stderr:1: v1:172.21.15.115:6789/0 mon.b 2024-03-25T18:36:52.156 INFO:teuthology.misc.health.smithi001.stderr:2: v1:172.21.15.1:6790/0 mon.c 2024-03-25T18:36:52.156 INFO:teuthology.misc.health.smithi001.stderr: 2024-03-25T18:36:52.161 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:36:52.160+0000 7f296e7fc640 1 -- 172.21.15.1:0/726347465 <== mon.0 v1:172.21.15.1:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 3246367930 0 0) 0x7f2978145cc0 con 0x7f29780b4ff0 2024-03-25T18:36:52.161 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:36:52.160+0000 7f296e7fc640 1 -- 172.21.15.1:0/726347465 >> v1:172.21.15.1:6790/0 conn(0x7f29780b4670 legacy=0x7f2978144cb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:36:52.161 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:36:52.160+0000 7f296e7fc640 1 -- 172.21.15.1:0/726347465 >> v1:172.21.15.115:6789/0 conn(0x7f297813c480 legacy=0x7f29781e2600 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:36:52.161 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:36:52.160+0000 7f296e7fc640 10 monclient: _finish_hunting 0 2024-03-25T18:36:52.161 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:36:52.160+0000 7f296e7fc640 1 monclient: found mon.a 2024-03-25T18:36:52.161 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:36:52.160+0000 7f296e7fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:36:52.161 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:36:52.160+0000 7f296e7fc640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.1:6789/0 2024-03-25T18:36:52.161 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:36:52.160+0000 7f296e7fc640 1 -- 172.21.15.1:0/726347465 --> v1:172.21.15.1:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f29781453c0 con 0x7f29780b4ff0 2024-03-25T18:36:52.161 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:36:52.160+0000 7f296e7fc640 10 monclient: _finish_auth 0 2024-03-25T18:36:52.161 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:36:52.160+0000 7f296e7fc640 10 monclient: _check_auth_tickets 2024-03-25T18:36:52.161 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:36:52.160+0000 7f296e7fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:36:52.161 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:36:52.160+0000 7f297fbcf640 5 monclient: authenticate success, global_id 4656 2024-03-25T18:36:52.161 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:36:52.160+0000 7f297fbcf640 10 monclient: _renew_subs 2024-03-25T18:36:52.162 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:36:52.160+0000 7f297fbcf640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.1:6789/0 2024-03-25T18:36:52.162 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:36:52.160+0000 7f297fbcf640 1 -- 172.21.15.1:0/726347465 --> v1:172.21.15.1:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2978138b20 con 0x7f29780b4ff0 2024-03-25T18:36:52.162 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:36:52.160+0000 7f297fbcf640 10 client.4656.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T18:36:52.162 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:36:52.160+0000 7f297fbcf640 10 monclient: _renew_subs 2024-03-25T18:36:52.162 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:36:52.160+0000 7f297fbcf640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.1:6789/0 2024-03-25T18:36:52.162 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:36:52.160+0000 7f297fbcf640 1 -- 172.21.15.1:0/726347465 --> v1:172.21.15.1:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f297813eac0 con 0x7f29780b4ff0 2024-03-25T18:36:52.162 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:36:52.160+0000 7f297fbcf640 1 librados: init done 2024-03-25T18:36:52.163 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:36:52.161+0000 7f297fbcf640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T18:36:52.163 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:36:52.161+0000 7f297fbcf640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:36:52.163 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:36:52.161+0000 7f297fbcf640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.1:6789/0 2024-03-25T18:36:52.163 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:36:52.161+0000 7f297fbcf640 1 -- 172.21.15.1:0/726347465 --> v1:172.21.15.1:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f297814f190 con 0x7f29780b4ff0 2024-03-25T18:36:52.473 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:36:52.472+0000 7f296e7fc640 1 -- 172.21.15.1:0/726347465 <== mon.0 v1:172.21.15.1:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f2968001270 con 0x7f29780b4ff0 2024-03-25T18:36:52.473 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:36:52.472+0000 7f296e7fc640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:36:52.473 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:36:52.472+0000 7f296e7fc640 1 -- 172.21.15.1:0/726347465 <== mon.0 v1:172.21.15.1:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1586965025 0 0) 0x7f2968004d90 con 0x7f29780b4ff0 2024-03-25T18:36:52.473 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:36:52.472+0000 7f296e7fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:36:52.473 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:36:52.472+0000 7f296e7fc640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-25T18:36:52.474 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:36:52.472+0000 7f296e7fc640 10 monclient: dump: 2024-03-25T18:36:52.474 INFO:teuthology.misc.health.smithi001.stderr:epoch 1 2024-03-25T18:36:52.474 INFO:teuthology.misc.health.smithi001.stderr:fsid c8ea72cf-9e99-4383-9b52-6c23d365fdfc 2024-03-25T18:36:52.474 INFO:teuthology.misc.health.smithi001.stderr:last_changed 2024-03-25T18:25:50.510519+0000 2024-03-25T18:36:52.474 INFO:teuthology.misc.health.smithi001.stderr:created 2024-03-25T18:25:50.510519+0000 2024-03-25T18:36:52.474 INFO:teuthology.misc.health.smithi001.stderr:min_mon_release 19 (squid) 2024-03-25T18:36:52.474 INFO:teuthology.misc.health.smithi001.stderr:election_strategy: 3 2024-03-25T18:36:52.474 INFO:teuthology.misc.health.smithi001.stderr:0: v1:172.21.15.1:6789/0 mon.a 2024-03-25T18:36:52.474 INFO:teuthology.misc.health.smithi001.stderr:1: v1:172.21.15.115:6789/0 mon.b 2024-03-25T18:36:52.475 INFO:teuthology.misc.health.smithi001.stderr:2: v1:172.21.15.1:6790/0 mon.c 2024-03-25T18:36:52.475 INFO:teuthology.misc.health.smithi001.stderr: 2024-03-25T18:36:52.475 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:36:52.472+0000 7f296ffff640 4 set_mon_vals no callback set 2024-03-25T18:36:52.475 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:36:52.473+0000 7f296e7fc640 1 -- 172.21.15.1:0/726347465 <== mon.0 v1:172.21.15.1:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 21787975 0 0) 0x7f296801ad60 con 0x7f29780b4ff0 2024-03-25T18:36:52.475 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:36:52.474+0000 7f297d96b640 10 monclient: get_auth_request con 0x7f295c06df20 auth_method 0 2024-03-25T18:36:52.475 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:36:52.474+0000 7f296e7fc640 10 client.4656.objecter ms_handle_connect 0x7f295c06df20 2024-03-25T18:36:52.495 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:36:52.494+0000 7f296e7fc640 1 -- 172.21.15.1:0/726347465 <== mon.0 v1:172.21.15.1:6789/0 8 ==== osd_map(11..11 src has 1..11) v4 ==== 1645+0+0 (unknown 2504898309 0 0) 0x7f2968054930 con 0x7f29780b4ff0 2024-03-25T18:36:52.496 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:36:52.495+0000 7f296e7fc640 10 client.4656.objecter ms_dispatch 0x7f2978000b90 osd_map(11..11 src has 1..11) v4 2024-03-25T18:36:52.496 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:36:52.495+0000 7f296e7fc640 3 client.4656.objecter handle_osd_map got epochs [11,11] > 0 2024-03-25T18:36:52.496 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:36:52.495+0000 7f296e7fc640 3 client.4656.objecter handle_osd_map decoding full epoch 11 2024-03-25T18:36:52.496 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:36:52.495+0000 7f296e7fc640 20 client.4656.objecter dump_active .. 0 homeless 2024-03-25T18:36:52.496 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:36:52.495+0000 7f296e7fc640 1 -- 172.21.15.1:0/726347465 <== mon.0 v1:172.21.15.1:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f2968050270 con 0x7f29780b4ff0 2024-03-25T18:36:52.496 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:36:52.495+0000 7f296e7fc640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:36:52.496 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:36:52.495+0000 7f296e7fc640 10 monclient: _finish_command 1 = system:0 2024-03-25T18:36:52.749 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:36:52.748+0000 7f297fbcf640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:36:52.749 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:36:52.748+0000 7f297fbcf640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:36:52.749 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:36:52.748+0000 7f297fbcf640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.1:6789/0 2024-03-25T18:36:52.749 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:36:52.748+0000 7f297fbcf640 1 -- 172.21.15.1:0/726347465 --> v1:172.21.15.1:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f2978138150 con 0x7f29780b4ff0 2024-03-25T18:36:52.781 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:36:52.781+0000 7f296e7fc640 1 -- 172.21.15.1:0/726347465 <== mon.0 v1:172.21.15.1:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v11) v1 ==== 74+0+7100 (unknown 238102980 0 2126609903) 0x7f2978138150 con 0x7f29780b4ff0 2024-03-25T18:36:52.781 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:36:52.781+0000 7f296e7fc640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:36:52.781 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:36:52.781+0000 7f296e7fc640 10 monclient: _finish_command 2 = system:0 2024-03-25T18:36:52.781 INFO:teuthology.misc.health.smithi001.stdout: 2024-03-25T18:36:52.782 INFO:teuthology.misc.health.smithi001.stdout:{"epoch":11,"fsid":"c8ea72cf-9e99-4383-9b52-6c23d365fdfc","created":"2024-03-25T18:26:38.856638+0000","modified":"2024-03-25T18:36:43.033551+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T18:36:43.033551+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"b9f34922-3a84-49be-8426-807c1eceaefc","up":0,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["autoout","exists","new"]},{"osd":1,"uuid":"42bf306f-55e3-4634-9874-8b4b73ac6644","up":0,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["autoout","exists","new"]},{"osd":2,"uuid":"292cb23a-18a9-4207-a7cc-4d5727feaec0","up":0,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["autoout","exists","new"]},{"osd":3,"uuid":"45e0e9ab-898d-4673-9590-b4117bc5e84a","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"831a5a09-73ab-4a61-9872-088e76bdbc66","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"88cd903d-3b55-4840-9396-6e3cfe005282","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"89db7355-bc9d-46f4-8ab0-e8d9e744e026","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"cac47bee-0914-4f08-896f-348838882866","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":65536,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":65536,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":65536,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T18:36:52.784 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:36:52.783+0000 7f297fbcf640 10 librados: watch_flush enter 2024-03-25T18:36:52.784 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:36:52.783+0000 7f297fbcf640 10 librados: watch_flush exit 2024-03-25T18:36:52.784 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:36:52.783+0000 7f297fbcf640 20 client.4656.objecter shutdown clearing up homeless session... 2024-03-25T18:36:52.784 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:36:52.783+0000 7f297fbcf640 10 client.4656.objecter successfully canceled tick 2024-03-25T18:36:52.784 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:36:52.783+0000 7f297fbcf640 1 -- 172.21.15.1:0/726347465 >> v1:172.21.15.115:6800/4237379176 conn(0x7f295c06df20 legacy=0x7f295c0703e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:36:52.784 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:36:52.783+0000 7f297fbcf640 10 monclient: shutdown 2024-03-25T18:36:52.784 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:36:52.783+0000 7f297fbcf640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:36:52.784 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:36:52.783+0000 7f297fbcf640 1 -- 172.21.15.1:0/726347465 >> v1:172.21.15.1:6789/0 conn(0x7f29780b4ff0 legacy=0x7f2978138410 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:36:52.784 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:36:52.783+0000 7f297fbcf640 1 -- 172.21.15.1:0/726347465 shutdown_connections 2024-03-25T18:36:52.785 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:36:52.783+0000 7f297fbcf640 1 -- 172.21.15.1:0/726347465 >> 172.21.15.1:0/726347465 conn(0x7f29780617a0 msgr2=0x7f29780b8690 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:36:52.785 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:36:52.783+0000 7f297fbcf640 1 -- 172.21.15.1:0/726347465 shutdown_connections 2024-03-25T18:36:52.785 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:36:52.783+0000 7f297fbcf640 1 -- 172.21.15.1:0/726347465 wait complete. 2024-03-25T18:36:52.785 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:36:52.783+0000 7f297fbcf640 1 librados: shutdown 2024-03-25T18:36:52.796 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T18:36:59.603 DEBUG:teuthology.orchestra.run.smithi001:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T18:36:59.720 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:36:59.718+0000 7ff780782640 10 monclient: get_monmap_and_config 2024-03-25T18:36:59.720 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:36:59.718+0000 7ff780782640 10 monclient: build_initial_monmap 2024-03-25T18:36:59.720 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:36:59.718+0000 7ff780782640 10 monclient: monmap: 2024-03-25T18:36:59.720 INFO:teuthology.misc.health.smithi001.stderr:epoch 0 2024-03-25T18:36:59.720 INFO:teuthology.misc.health.smithi001.stderr:fsid c8ea72cf-9e99-4383-9b52-6c23d365fdfc 2024-03-25T18:36:59.720 INFO:teuthology.misc.health.smithi001.stderr:last_changed 2024-03-25T18:36:59.720081+0000 2024-03-25T18:36:59.720 INFO:teuthology.misc.health.smithi001.stderr:created 2024-03-25T18:36:59.720081+0000 2024-03-25T18:36:59.720 INFO:teuthology.misc.health.smithi001.stderr:min_mon_release 0 (unknown) 2024-03-25T18:36:59.720 INFO:teuthology.misc.health.smithi001.stderr:election_strategy: 3 2024-03-25T18:36:59.720 INFO:teuthology.misc.health.smithi001.stderr:0: v1:172.21.15.1:6789/0 mon.noname-a 2024-03-25T18:36:59.720 INFO:teuthology.misc.health.smithi001.stderr:1: v1:172.21.15.115:6789/0 mon.noname-c 2024-03-25T18:36:59.721 INFO:teuthology.misc.health.smithi001.stderr:2: v1:172.21.15.1:6790/0 mon.noname-b 2024-03-25T18:36:59.721 INFO:teuthology.misc.health.smithi001.stderr: 2024-03-25T18:36:59.722 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:36:59.719+0000 7ff780782640 1 Processor -- start 2024-03-25T18:36:59.722 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:36:59.719+0000 7ff780782640 1 -- start start 2024-03-25T18:36:59.722 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:36:59.719+0000 7ff780782640 10 monclient: init 2024-03-25T18:36:59.722 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:36:59.719+0000 7ff780782640 10 monclient: _reopen_session rank -1 2024-03-25T18:36:59.722 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:36:59.719+0000 7ff780782640 10 monclient: _add_conns ranks=[2,1,0] 2024-03-25T18:36:59.722 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:36:59.719+0000 7ff780782640 10 monclient(hunting): picked mon.noname-b con 0x7ff7780b49f0 addr v1:172.21.15.1:6790/0 2024-03-25T18:36:59.722 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:36:59.719+0000 7ff780782640 10 monclient(hunting): picked mon.noname-c con 0x7ff7780b4dd0 addr v1:172.21.15.115:6789/0 2024-03-25T18:36:59.722 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:36:59.719+0000 7ff780782640 10 monclient(hunting): picked mon.noname-a con 0x7ff7780a9a90 addr v1:172.21.15.1:6789/0 2024-03-25T18:36:59.722 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:36:59.719+0000 7ff780782640 1 -- --> v1:172.21.15.1:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff7780aa070 con 0x7ff7780a9a90 2024-03-25T18:36:59.722 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:36:59.719+0000 7ff780782640 1 -- --> v1:172.21.15.115:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff7780aaf40 con 0x7ff7780b4dd0 2024-03-25T18:36:59.722 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:36:59.719+0000 7ff780782640 1 -- --> v1:172.21.15.1:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff7780ab220 con 0x7ff7780b49f0 2024-03-25T18:36:59.723 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:36:59.719+0000 7ff780782640 10 monclient(hunting): _renew_subs 2024-03-25T18:36:59.723 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:36:59.719+0000 7ff780782640 10 monclient(hunting): authenticate will time out at 8100.190631s 2024-03-25T18:36:59.736 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:36:59.735+0000 7ff77ed1f640 1 --1- >> v1:172.21.15.1:6789/0 conn(0x7ff7780a9a90 0x7ff778142c30 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.1:6789/0 says I am v1:172.21.15.1:45888/0 (socket says 172.21.15.1:45888) 2024-03-25T18:36:59.736 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:36:59.735+0000 7ff77ed1f640 1 -- 172.21.15.1:0/4238269413 learned_addr learned my addr 172.21.15.1:0/4238269413 (peer_addr_for_me v1:172.21.15.1:0/0) 2024-03-25T18:36:59.755 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:36:59.753+0000 7ff77d51c640 1 -- 172.21.15.1:0/4238269413 <== mon.0 v1:172.21.15.1:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1429309707 0 0) 0x7ff7780aa070 con 0x7ff7780a9a90 2024-03-25T18:36:59.755 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:36:59.753+0000 7ff77d51c640 10 monclient(hunting): _negotiate 2024-03-25T18:36:59.755 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:36:59.753+0000 7ff77d51c640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:36:59.755 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:36:59.753+0000 7ff77d51c640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:36:59.755 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:36:59.753+0000 7ff77d51c640 10 monclient(hunting): my global_id is 4659 2024-03-25T18:36:59.755 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:36:59.753+0000 7ff77d51c640 1 -- 172.21.15.1:0/4238269413 --> v1:172.21.15.1:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ff764003520 con 0x7ff7780a9a90 2024-03-25T18:36:59.760 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:36:59.758+0000 7ff77d51c640 1 -- 172.21.15.1:0/4238269413 <== mon.0 v1:172.21.15.1:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3229403676 0 0) 0x7ff764003520 con 0x7ff7780a9a90 2024-03-25T18:36:59.760 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:36:59.758+0000 7ff77d51c640 1 -- 172.21.15.1:0/4238269413 >> v1:172.21.15.1:6790/0 conn(0x7ff7780b49f0 legacy=0x7ff77813bd00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:36:59.760 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:36:59.758+0000 7ff77d51c640 1 -- 172.21.15.1:0/4238269413 >> v1:172.21.15.115:6789/0 conn(0x7ff7780b4dd0 legacy=0x7ff778142520 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:36:59.760 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:36:59.758+0000 7ff77d51c640 10 monclient: _finish_hunting 0 2024-03-25T18:36:59.760 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:36:59.758+0000 7ff77d51c640 1 monclient: found mon.noname-a 2024-03-25T18:36:59.760 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:36:59.758+0000 7ff77d51c640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:36:59.760 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:36:59.758+0000 7ff77d51c640 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.1:6789/0 2024-03-25T18:36:59.760 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:36:59.758+0000 7ff77d51c640 1 -- 172.21.15.1:0/4238269413 --> v1:172.21.15.1:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff77813c410 con 0x7ff7780a9a90 2024-03-25T18:36:59.760 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:36:59.758+0000 7ff77d51c640 10 monclient: _finish_auth 0 2024-03-25T18:36:59.760 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:36:59.758+0000 7ff77d51c640 10 monclient: _check_auth_tickets 2024-03-25T18:36:59.761 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:36:59.758+0000 7ff77d51c640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:36:59.761 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:36:59.758+0000 7ff780782640 5 monclient: authenticate success, global_id 4659 2024-03-25T18:36:59.761 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:36:59.758+0000 7ff77d51c640 1 -- 172.21.15.1:0/4238269413 <== mon.0 v1:172.21.15.1:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1586965025 0 0) 0x7ff770001a10 con 0x7ff7780a9a90 2024-03-25T18:36:59.761 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:36:59.758+0000 7ff780782640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T18:36:59.761 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:36:59.758+0000 7ff77d51c640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:36:59.761 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:36:59.758+0000 7ff77d51c640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-03-25T18:36:59.761 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:36:59.758+0000 7ff77d51c640 10 monclient: dump: 2024-03-25T18:36:59.761 INFO:teuthology.misc.health.smithi001.stderr:epoch 1 2024-03-25T18:36:59.761 INFO:teuthology.misc.health.smithi001.stderr:fsid c8ea72cf-9e99-4383-9b52-6c23d365fdfc 2024-03-25T18:36:59.761 INFO:teuthology.misc.health.smithi001.stderr:last_changed 2024-03-25T18:25:50.510519+0000 2024-03-25T18:36:59.761 INFO:teuthology.misc.health.smithi001.stderr:created 2024-03-25T18:25:50.510519+0000 2024-03-25T18:36:59.761 INFO:teuthology.misc.health.smithi001.stderr:min_mon_release 19 (squid) 2024-03-25T18:36:59.761 INFO:teuthology.misc.health.smithi001.stderr:election_strategy: 3 2024-03-25T18:36:59.761 INFO:teuthology.misc.health.smithi001.stderr:0: v1:172.21.15.1:6789/0 mon.a 2024-03-25T18:36:59.761 INFO:teuthology.misc.health.smithi001.stderr:1: v1:172.21.15.115:6789/0 mon.b 2024-03-25T18:36:59.762 INFO:teuthology.misc.health.smithi001.stderr:2: v1:172.21.15.1:6790/0 mon.c 2024-03-25T18:36:59.762 INFO:teuthology.misc.health.smithi001.stderr: 2024-03-25T18:36:59.762 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:36:59.758+0000 7ff780782640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T18:36:59.773 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:36:59.771+0000 7ff77d51c640 1 -- 172.21.15.1:0/4238269413 <== mon.0 v1:172.21.15.1:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7ff770001b70 con 0x7ff7780a9a90 2024-03-25T18:36:59.773 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:36:59.771+0000 7ff77d51c640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:36:59.773 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:36:59.771+0000 7ff77d51c640 1 -- 172.21.15.1:0/4238269413 <== mon.0 v1:172.21.15.1:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1586965025 0 0) 0x7ff770001f80 con 0x7ff7780a9a90 2024-03-25T18:36:59.773 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:36:59.771+0000 7ff780782640 10 monclient: get_monmap_and_config success 2024-03-25T18:36:59.773 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:36:59.771+0000 7ff780782640 4 set_mon_vals no callback set 2024-03-25T18:36:59.773 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:36:59.771+0000 7ff77d51c640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:36:59.773 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:36:59.771+0000 7ff77d51c640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-25T18:36:59.773 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:36:59.771+0000 7ff77d51c640 10 monclient: dump: 2024-03-25T18:36:59.773 INFO:teuthology.misc.health.smithi001.stderr:epoch 1 2024-03-25T18:36:59.773 INFO:teuthology.misc.health.smithi001.stderr:fsid c8ea72cf-9e99-4383-9b52-6c23d365fdfc 2024-03-25T18:36:59.773 INFO:teuthology.misc.health.smithi001.stderr:last_changed 2024-03-25T18:25:50.510519+0000 2024-03-25T18:36:59.774 INFO:teuthology.misc.health.smithi001.stderr:created 2024-03-25T18:25:50.510519+0000 2024-03-25T18:36:59.774 INFO:teuthology.misc.health.smithi001.stderr:min_mon_release 19 (squid) 2024-03-25T18:36:59.774 INFO:teuthology.misc.health.smithi001.stderr:election_strategy: 3 2024-03-25T18:36:59.774 INFO:teuthology.misc.health.smithi001.stderr:0: v1:172.21.15.1:6789/0 mon.a 2024-03-25T18:36:59.774 INFO:teuthology.misc.health.smithi001.stderr:1: v1:172.21.15.115:6789/0 mon.b 2024-03-25T18:36:59.774 INFO:teuthology.misc.health.smithi001.stderr:2: v1:172.21.15.1:6790/0 mon.c 2024-03-25T18:36:59.774 INFO:teuthology.misc.health.smithi001.stderr: 2024-03-25T18:36:59.775 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:36:59.772+0000 7ff780782640 10 monclient: shutdown 2024-03-25T18:36:59.775 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:36:59.772+0000 7ff780782640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:36:59.775 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:36:59.772+0000 7ff780782640 1 -- 172.21.15.1:0/4238269413 >> v1:172.21.15.1:6789/0 conn(0x7ff7780a9a90 legacy=0x7ff778142c30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:36:59.775 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:36:59.772+0000 7ff780782640 1 -- 172.21.15.1:0/4238269413 shutdown_connections 2024-03-25T18:36:59.775 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:36:59.772+0000 7ff780782640 1 -- 172.21.15.1:0/4238269413 >> 172.21.15.1:0/4238269413 conn(0x7ff7780615c0 msgr2=0x7ff7780619a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:36:59.775 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:36:59.772+0000 7ff780782640 1 -- 172.21.15.1:0/4238269413 shutdown_connections 2024-03-25T18:36:59.775 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:36:59.772+0000 7ff780782640 1 -- 172.21.15.1:0/4238269413 wait complete. 2024-03-25T18:36:59.775 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:36:59.773+0000 7ff780782640 10 monclient: build_initial_monmap 2024-03-25T18:36:59.775 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:36:59.773+0000 7ff780782640 10 monclient: monmap: 2024-03-25T18:36:59.775 INFO:teuthology.misc.health.smithi001.stderr:epoch 0 2024-03-25T18:36:59.776 INFO:teuthology.misc.health.smithi001.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T18:36:59.776 INFO:teuthology.misc.health.smithi001.stderr:last_changed 0.000000 2024-03-25T18:36:59.776 INFO:teuthology.misc.health.smithi001.stderr:created 0.000000 2024-03-25T18:36:59.776 INFO:teuthology.misc.health.smithi001.stderr:min_mon_release 0 (unknown) 2024-03-25T18:36:59.776 INFO:teuthology.misc.health.smithi001.stderr:election_strategy: 1 2024-03-25T18:36:59.776 INFO:teuthology.misc.health.smithi001.stderr:0: v1:172.21.15.1:6789/0 mon.noname-a 2024-03-25T18:36:59.776 INFO:teuthology.misc.health.smithi001.stderr:1: v1:172.21.15.115:6789/0 mon.noname-b 2024-03-25T18:36:59.776 INFO:teuthology.misc.health.smithi001.stderr:2: v1:172.21.15.1:6790/0 mon.noname-c 2024-03-25T18:36:59.776 INFO:teuthology.misc.health.smithi001.stderr: 2024-03-25T18:36:59.777 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:36:59.774+0000 7ff780782640 1 librados: starting msgr at 2024-03-25T18:36:59.778 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:36:59.774+0000 7ff780782640 1 librados: starting objecter 2024-03-25T18:36:59.778 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:36:59.774+0000 7ff780782640 1 Processor -- start 2024-03-25T18:36:59.778 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:36:59.774+0000 7ff780782640 1 -- start start 2024-03-25T18:36:59.778 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:36:59.774+0000 7ff780782640 1 librados: setting wanted keys 2024-03-25T18:36:59.778 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:36:59.774+0000 7ff780782640 1 librados: calling monclient init 2024-03-25T18:36:59.778 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:36:59.774+0000 7ff780782640 10 monclient: init 2024-03-25T18:36:59.778 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:36:59.775+0000 7ff780782640 10 monclient: _reopen_session rank -1 2024-03-25T18:36:59.778 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:36:59.775+0000 7ff780782640 10 monclient: _add_conns ranks=[1,2,0] 2024-03-25T18:36:59.778 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:36:59.775+0000 7ff780782640 10 monclient(hunting): picked mon.noname-b con 0x7ff7780a9a90 addr v1:172.21.15.115:6789/0 2024-03-25T18:36:59.778 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:36:59.775+0000 7ff780782640 10 monclient(hunting): picked mon.noname-c con 0x7ff7780b49f0 addr v1:172.21.15.1:6790/0 2024-03-25T18:36:59.778 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:36:59.775+0000 7ff780782640 10 monclient(hunting): picked mon.noname-a con 0x7ff7780b4dd0 addr v1:172.21.15.1:6789/0 2024-03-25T18:36:59.778 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:36:59.775+0000 7ff780782640 1 -- --> v1:172.21.15.1:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff77813c7a0 con 0x7ff7780b4dd0 2024-03-25T18:36:59.778 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:36:59.775+0000 7ff780782640 1 -- --> v1:172.21.15.115:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff778139100 con 0x7ff7780a9a90 2024-03-25T18:36:59.778 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:36:59.775+0000 7ff780782640 1 -- --> v1:172.21.15.1:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff77813ad50 con 0x7ff7780b49f0 2024-03-25T18:36:59.779 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:36:59.775+0000 7ff780782640 10 monclient(hunting): _renew_subs 2024-03-25T18:36:59.779 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:36:59.775+0000 7ff780782640 10 monclient(hunting): authenticate will time out at 8100.246066s 2024-03-25T18:36:59.789 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:36:59.787+0000 7ff77dd1d640 1 --1- >> v1:172.21.15.1:6790/0 conn(0x7ff7780b49f0 0x7ff77813b730 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.1:6790/0 says I am v1:172.21.15.1:48654/0 (socket says 172.21.15.1:48654) 2024-03-25T18:36:59.789 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:36:59.787+0000 7ff77dd1d640 1 -- 172.21.15.1:0/578820271 learned_addr learned my addr 172.21.15.1:0/578820271 (peer_addr_for_me v1:172.21.15.1:0/0) 2024-03-25T18:36:59.789 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:36:59.788+0000 7ff76effd640 10 client.?.objecter ms_handle_connect 0x7ff7780b49f0 2024-03-25T18:36:59.789 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:36:59.788+0000 7ff76effd640 10 client.?.objecter resend_mon_ops 2024-03-25T18:36:59.791 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:36:59.789+0000 7ff76effd640 10 client.?.objecter ms_handle_connect 0x7ff7780a9a90 2024-03-25T18:36:59.791 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:36:59.789+0000 7ff76effd640 10 client.?.objecter resend_mon_ops 2024-03-25T18:36:59.801 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:36:59.799+0000 7ff76effd640 1 -- 172.21.15.1:0/578820271 <== mon.2 v1:172.21.15.1:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2920702733 0 0) 0x7ff77813ad50 con 0x7ff7780b49f0 2024-03-25T18:36:59.801 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:36:59.799+0000 7ff76effd640 10 monclient(hunting): _negotiate 2024-03-25T18:36:59.801 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:36:59.799+0000 7ff76effd640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:36:59.801 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:36:59.799+0000 7ff76effd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:36:59.801 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:36:59.799+0000 7ff76effd640 10 monclient(hunting): my global_id is 4670 2024-03-25T18:36:59.801 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:36:59.799+0000 7ff76effd640 1 -- 172.21.15.1:0/578820271 --> v1:172.21.15.1:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ff75c004eb0 con 0x7ff7780b49f0 2024-03-25T18:36:59.803 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:36:59.801+0000 7ff76effd640 1 -- 172.21.15.1:0/578820271 <== mon.1 v1:172.21.15.115:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1554822286 0 0) 0x7ff778139100 con 0x7ff7780a9a90 2024-03-25T18:36:59.803 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:36:59.801+0000 7ff76effd640 10 monclient(hunting): _negotiate 2024-03-25T18:36:59.803 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:36:59.801+0000 7ff76effd640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:36:59.803 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:36:59.801+0000 7ff76effd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:36:59.803 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:36:59.801+0000 7ff76effd640 10 monclient(hunting): my global_id is 4675 2024-03-25T18:36:59.803 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:36:59.801+0000 7ff76effd640 1 -- 172.21.15.1:0/578820271 --> v1:172.21.15.115:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ff77813ad50 con 0x7ff7780a9a90 2024-03-25T18:36:59.807 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:36:59.806+0000 7ff76effd640 10 client.?.objecter ms_handle_connect 0x7ff7780b4dd0 2024-03-25T18:36:59.807 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:36:59.806+0000 7ff76effd640 10 client.?.objecter resend_mon_ops 2024-03-25T18:36:59.837 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:36:59.835+0000 7ff76effd640 1 -- 172.21.15.1:0/578820271 <== mon.0 v1:172.21.15.1:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2190868729 0 0) 0x7ff77813c7a0 con 0x7ff7780b4dd0 2024-03-25T18:36:59.837 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:36:59.835+0000 7ff76effd640 10 monclient(hunting): _negotiate 2024-03-25T18:36:59.837 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:36:59.835+0000 7ff76effd640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:36:59.837 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:36:59.835+0000 7ff76effd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:36:59.837 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:36:59.835+0000 7ff76effd640 10 monclient(hunting): my global_id is 4662 2024-03-25T18:36:59.837 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:36:59.835+0000 7ff76effd640 1 -- 172.21.15.1:0/578820271 --> v1:172.21.15.1:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ff778139100 con 0x7ff7780b4dd0 2024-03-25T18:36:59.905 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:36:59.903+0000 7ff76effd640 1 -- 172.21.15.1:0/578820271 <== mon.1 v1:172.21.15.115:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1365842187 0 0) 0x7ff77813ad50 con 0x7ff7780a9a90 2024-03-25T18:36:59.906 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:36:59.903+0000 7ff76effd640 1 -- 172.21.15.1:0/578820271 --> v1:172.21.15.115:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7ff77813c7a0 con 0x7ff7780a9a90 2024-03-25T18:36:59.906 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:36:59.903+0000 7ff76effd640 1 -- 172.21.15.1:0/578820271 <== mon.0 v1:172.21.15.1:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 888583649 0 0) 0x7ff778139100 con 0x7ff7780b4dd0 2024-03-25T18:36:59.906 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:36:59.903+0000 7ff76effd640 1 -- 172.21.15.1:0/578820271 --> v1:172.21.15.1:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7ff77813ad50 con 0x7ff7780b4dd0 2024-03-25T18:36:59.906 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:36:59.903+0000 7ff76effd640 1 -- 172.21.15.1:0/578820271 <== mon.1 v1:172.21.15.115:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1586965025 0 0) 0x7ff7740018a0 con 0x7ff7780a9a90 2024-03-25T18:36:59.906 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:36:59.903+0000 7ff76effd640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:36:59.906 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:36:59.903+0000 7ff76effd640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-03-25T18:36:59.906 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:36:59.903+0000 7ff76effd640 10 monclient(hunting): dump: 2024-03-25T18:36:59.906 INFO:teuthology.misc.health.smithi001.stderr:epoch 1 2024-03-25T18:36:59.907 INFO:teuthology.misc.health.smithi001.stderr:fsid c8ea72cf-9e99-4383-9b52-6c23d365fdfc 2024-03-25T18:36:59.907 INFO:teuthology.misc.health.smithi001.stderr:last_changed 2024-03-25T18:25:50.510519+0000 2024-03-25T18:36:59.907 INFO:teuthology.misc.health.smithi001.stderr:created 2024-03-25T18:25:50.510519+0000 2024-03-25T18:36:59.907 INFO:teuthology.misc.health.smithi001.stderr:min_mon_release 19 (squid) 2024-03-25T18:36:59.907 INFO:teuthology.misc.health.smithi001.stderr:election_strategy: 3 2024-03-25T18:36:59.907 INFO:teuthology.misc.health.smithi001.stderr:0: v1:172.21.15.1:6789/0 mon.a 2024-03-25T18:36:59.907 INFO:teuthology.misc.health.smithi001.stderr:1: v1:172.21.15.115:6789/0 mon.b 2024-03-25T18:36:59.907 INFO:teuthology.misc.health.smithi001.stderr:2: v1:172.21.15.1:6790/0 mon.c 2024-03-25T18:36:59.907 INFO:teuthology.misc.health.smithi001.stderr: 2024-03-25T18:36:59.907 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:36:59.903+0000 7ff76effd640 10 monclient(hunting): _finish_auth 0 2024-03-25T18:36:59.907 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:36:59.903+0000 7ff76effd640 1 -- 172.21.15.1:0/578820271 <== mon.0 v1:172.21.15.1:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1586965025 0 0) 0x7ff770001e80 con 0x7ff7780b4dd0 2024-03-25T18:36:59.908 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:36:59.903+0000 7ff76effd640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:36:59.908 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:36:59.903+0000 7ff76effd640 10 monclient(hunting): got monmap 1 from mon.a (according to old e1) 2024-03-25T18:36:59.908 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:36:59.903+0000 7ff76effd640 10 monclient(hunting): dump: 2024-03-25T18:36:59.908 INFO:teuthology.misc.health.smithi001.stderr:epoch 1 2024-03-25T18:36:59.908 INFO:teuthology.misc.health.smithi001.stderr:fsid c8ea72cf-9e99-4383-9b52-6c23d365fdfc 2024-03-25T18:36:59.908 INFO:teuthology.misc.health.smithi001.stderr:last_changed 2024-03-25T18:25:50.510519+0000 2024-03-25T18:36:59.908 INFO:teuthology.misc.health.smithi001.stderr:created 2024-03-25T18:25:50.510519+0000 2024-03-25T18:36:59.908 INFO:teuthology.misc.health.smithi001.stderr:min_mon_release 19 (squid) 2024-03-25T18:36:59.908 INFO:teuthology.misc.health.smithi001.stderr:election_strategy: 3 2024-03-25T18:36:59.908 INFO:teuthology.misc.health.smithi001.stderr:0: v1:172.21.15.1:6789/0 mon.a 2024-03-25T18:36:59.908 INFO:teuthology.misc.health.smithi001.stderr:1: v1:172.21.15.115:6789/0 mon.b 2024-03-25T18:36:59.908 INFO:teuthology.misc.health.smithi001.stderr:2: v1:172.21.15.1:6790/0 mon.c 2024-03-25T18:36:59.909 INFO:teuthology.misc.health.smithi001.stderr: 2024-03-25T18:36:59.918 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:36:59.916+0000 7ff76effd640 1 -- 172.21.15.1:0/578820271 <== mon.1 v1:172.21.15.115:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 3122662415 0 0) 0x7ff77813c7a0 con 0x7ff7780a9a90 2024-03-25T18:36:59.918 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:36:59.916+0000 7ff76effd640 1 -- 172.21.15.1:0/578820271 >> v1:172.21.15.1:6790/0 conn(0x7ff7780b49f0 legacy=0x7ff77813b730 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:36:59.918 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:36:59.916+0000 7ff76effd640 1 -- 172.21.15.1:0/578820271 >> v1:172.21.15.1:6789/0 conn(0x7ff7780b4dd0 legacy=0x7ff7781e2480 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:36:59.918 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:36:59.916+0000 7ff76effd640 10 monclient: _finish_hunting 0 2024-03-25T18:36:59.918 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:36:59.916+0000 7ff76effd640 1 monclient: found mon.b 2024-03-25T18:36:59.918 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:36:59.916+0000 7ff76effd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:36:59.918 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:36:59.916+0000 7ff76effd640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.115:6789/0 2024-03-25T18:36:59.919 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:36:59.916+0000 7ff76effd640 1 -- 172.21.15.1:0/578820271 --> v1:172.21.15.115:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff77813b030 con 0x7ff7780a9a90 2024-03-25T18:36:59.919 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:36:59.916+0000 7ff76effd640 10 monclient: _finish_auth 0 2024-03-25T18:36:59.919 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:36:59.916+0000 7ff76effd640 10 monclient: _check_auth_tickets 2024-03-25T18:36:59.919 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:36:59.916+0000 7ff76effd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:36:59.919 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:36:59.916+0000 7ff780782640 5 monclient: authenticate success, global_id 4675 2024-03-25T18:36:59.920 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:36:59.916+0000 7ff780782640 10 monclient: _renew_subs 2024-03-25T18:36:59.920 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:36:59.916+0000 7ff780782640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.115:6789/0 2024-03-25T18:36:59.920 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:36:59.916+0000 7ff780782640 1 -- 172.21.15.1:0/578820271 --> v1:172.21.15.115:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff77813b310 con 0x7ff7780a9a90 2024-03-25T18:36:59.920 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:36:59.916+0000 7ff780782640 10 client.4675.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T18:36:59.920 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:36:59.916+0000 7ff780782640 10 monclient: _renew_subs 2024-03-25T18:36:59.920 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:36:59.916+0000 7ff780782640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.115:6789/0 2024-03-25T18:36:59.920 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:36:59.916+0000 7ff780782640 1 -- 172.21.15.1:0/578820271 --> v1:172.21.15.115:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7ff778139750 con 0x7ff7780a9a90 2024-03-25T18:36:59.920 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:36:59.916+0000 7ff780782640 1 librados: init done 2024-03-25T18:36:59.921 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:36:59.917+0000 7ff780782640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T18:36:59.921 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:36:59.917+0000 7ff780782640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:36:59.921 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:36:59.917+0000 7ff780782640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.115:6789/0 2024-03-25T18:36:59.921 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:36:59.917+0000 7ff780782640 1 -- 172.21.15.1:0/578820271 --> v1:172.21.15.115:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff77804f730 con 0x7ff7780a9a90 2024-03-25T18:37:00.211 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:37:00.209+0000 7ff76effd640 1 -- 172.21.15.1:0/578820271 <== mon.1 v1:172.21.15.115:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7ff774001a00 con 0x7ff7780a9a90 2024-03-25T18:37:00.211 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:37:00.209+0000 7ff76effd640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:37:00.211 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:37:00.209+0000 7ff77cd1b640 4 set_mon_vals no callback set 2024-03-25T18:37:00.211 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:37:00.209+0000 7ff76effd640 1 -- 172.21.15.1:0/578820271 <== mon.1 v1:172.21.15.115:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1586965025 0 0) 0x7ff7740037d0 con 0x7ff7780a9a90 2024-03-25T18:37:00.211 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:37:00.209+0000 7ff76effd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:37:00.212 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:37:00.209+0000 7ff76effd640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-25T18:37:00.212 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:37:00.209+0000 7ff76effd640 10 monclient: dump: 2024-03-25T18:37:00.212 INFO:teuthology.misc.health.smithi001.stderr:epoch 1 2024-03-25T18:37:00.212 INFO:teuthology.misc.health.smithi001.stderr:fsid c8ea72cf-9e99-4383-9b52-6c23d365fdfc 2024-03-25T18:37:00.212 INFO:teuthology.misc.health.smithi001.stderr:last_changed 2024-03-25T18:25:50.510519+0000 2024-03-25T18:37:00.212 INFO:teuthology.misc.health.smithi001.stderr:created 2024-03-25T18:25:50.510519+0000 2024-03-25T18:37:00.212 INFO:teuthology.misc.health.smithi001.stderr:min_mon_release 19 (squid) 2024-03-25T18:37:00.212 INFO:teuthology.misc.health.smithi001.stderr:election_strategy: 3 2024-03-25T18:37:00.212 INFO:teuthology.misc.health.smithi001.stderr:0: v1:172.21.15.1:6789/0 mon.a 2024-03-25T18:37:00.212 INFO:teuthology.misc.health.smithi001.stderr:1: v1:172.21.15.115:6789/0 mon.b 2024-03-25T18:37:00.212 INFO:teuthology.misc.health.smithi001.stderr:2: v1:172.21.15.1:6790/0 mon.c 2024-03-25T18:37:00.212 INFO:teuthology.misc.health.smithi001.stderr: 2024-03-25T18:37:00.213 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:37:00.210+0000 7ff76effd640 1 -- 172.21.15.1:0/578820271 <== mon.1 v1:172.21.15.115:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 21787975 0 0) 0x7ff774019860 con 0x7ff7780a9a90 2024-03-25T18:37:00.213 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:37:00.211+0000 7ff77dd1d640 10 monclient: get_auth_request con 0x7ff75c06e020 auth_method 0 2024-03-25T18:37:00.213 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:37:00.211+0000 7ff76effd640 10 client.4675.objecter ms_handle_connect 0x7ff75c06e020 2024-03-25T18:37:00.217 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:37:00.215+0000 7ff76effd640 1 -- 172.21.15.1:0/578820271 <== mon.1 v1:172.21.15.115:6789/0 8 ==== osd_map(11..11 src has 1..11) v4 ==== 1645+0+0 (unknown 2504898309 0 0) 0x7ff774053560 con 0x7ff7780a9a90 2024-03-25T18:37:00.217 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:37:00.215+0000 7ff76effd640 10 client.4675.objecter ms_dispatch 0x7ff778000b90 osd_map(11..11 src has 1..11) v4 2024-03-25T18:37:00.217 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:37:00.215+0000 7ff76effd640 3 client.4675.objecter handle_osd_map got epochs [11,11] > 0 2024-03-25T18:37:00.218 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:37:00.215+0000 7ff76effd640 3 client.4675.objecter handle_osd_map decoding full epoch 11 2024-03-25T18:37:00.218 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:37:00.215+0000 7ff76effd640 20 client.4675.objecter dump_active .. 0 homeless 2024-03-25T18:37:00.218 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:37:00.216+0000 7ff76effd640 1 -- 172.21.15.1:0/578820271 <== mon.1 v1:172.21.15.115:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7ff77404eea0 con 0x7ff7780a9a90 2024-03-25T18:37:00.218 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:37:00.216+0000 7ff76effd640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:37:00.218 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:37:00.216+0000 7ff76effd640 10 monclient: _finish_command 1 = system:0 2024-03-25T18:37:00.474 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:37:00.472+0000 7ff780782640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:37:00.474 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:37:00.472+0000 7ff780782640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:37:00.474 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:37:00.472+0000 7ff780782640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.115:6789/0 2024-03-25T18:37:00.474 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:37:00.472+0000 7ff780782640 1 -- 172.21.15.1:0/578820271 --> v1:172.21.15.115:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7ff7780b6b90 con 0x7ff7780a9a90 2024-03-25T18:37:00.516 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:37:00.515+0000 7ff76effd640 1 -- 172.21.15.1:0/578820271 <== mon.1 v1:172.21.15.115:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v11) v1 ==== 74+0+7100 (unknown 238102980 0 2126609903) 0x7ff774050fa0 con 0x7ff7780a9a90 2024-03-25T18:37:00.517 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:37:00.515+0000 7ff76effd640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:37:00.517 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:37:00.515+0000 7ff76effd640 10 monclient: _finish_command 2 = system:0 2024-03-25T18:37:00.517 INFO:teuthology.misc.health.smithi001.stdout: 2024-03-25T18:37:00.517 INFO:teuthology.misc.health.smithi001.stdout:{"epoch":11,"fsid":"c8ea72cf-9e99-4383-9b52-6c23d365fdfc","created":"2024-03-25T18:26:38.856638+0000","modified":"2024-03-25T18:36:43.033551+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T18:36:43.033551+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"b9f34922-3a84-49be-8426-807c1eceaefc","up":0,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["autoout","exists","new"]},{"osd":1,"uuid":"42bf306f-55e3-4634-9874-8b4b73ac6644","up":0,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["autoout","exists","new"]},{"osd":2,"uuid":"292cb23a-18a9-4207-a7cc-4d5727feaec0","up":0,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["autoout","exists","new"]},{"osd":3,"uuid":"45e0e9ab-898d-4673-9590-b4117bc5e84a","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"831a5a09-73ab-4a61-9872-088e76bdbc66","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"88cd903d-3b55-4840-9396-6e3cfe005282","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"89db7355-bc9d-46f4-8ab0-e8d9e744e026","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"cac47bee-0914-4f08-896f-348838882866","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":65536,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":65536,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":65536,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T18:37:00.520 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:37:00.518+0000 7ff780782640 10 librados: watch_flush enter 2024-03-25T18:37:00.520 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:37:00.518+0000 7ff780782640 10 librados: watch_flush exit 2024-03-25T18:37:00.520 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:37:00.518+0000 7ff780782640 20 client.4675.objecter shutdown clearing up homeless session... 2024-03-25T18:37:00.520 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:37:00.518+0000 7ff780782640 10 client.4675.objecter successfully canceled tick 2024-03-25T18:37:00.520 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:37:00.518+0000 7ff780782640 1 -- 172.21.15.1:0/578820271 >> v1:172.21.15.115:6800/4237379176 conn(0x7ff75c06e020 legacy=0x7ff75c0704e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:37:00.520 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:37:00.518+0000 7ff780782640 10 monclient: shutdown 2024-03-25T18:37:00.520 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:37:00.518+0000 7ff780782640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:37:00.520 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:37:00.518+0000 7ff780782640 1 -- 172.21.15.1:0/578820271 >> v1:172.21.15.115:6789/0 conn(0x7ff7780a9a90 legacy=0x7ff778139c20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:37:00.520 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:37:00.518+0000 7ff780782640 1 -- 172.21.15.1:0/578820271 shutdown_connections 2024-03-25T18:37:00.520 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:37:00.518+0000 7ff780782640 1 -- 172.21.15.1:0/578820271 >> 172.21.15.1:0/578820271 conn(0x7ff7780615c0 msgr2=0x7ff778142be0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:37:00.521 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:37:00.518+0000 7ff780782640 1 -- 172.21.15.1:0/578820271 shutdown_connections 2024-03-25T18:37:00.521 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:37:00.518+0000 7ff780782640 1 -- 172.21.15.1:0/578820271 wait complete. 2024-03-25T18:37:00.521 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:37:00.518+0000 7ff780782640 1 librados: shutdown 2024-03-25T18:37:00.532 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T18:37:07.339 DEBUG:teuthology.orchestra.run.smithi001:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T18:37:07.455 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:37:07.454+0000 7f751d9a8640 10 monclient: get_monmap_and_config 2024-03-25T18:37:07.456 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:37:07.454+0000 7f751d9a8640 10 monclient: build_initial_monmap 2024-03-25T18:37:07.456 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:37:07.454+0000 7f751d9a8640 10 monclient: monmap: 2024-03-25T18:37:07.456 INFO:teuthology.misc.health.smithi001.stderr:epoch 0 2024-03-25T18:37:07.456 INFO:teuthology.misc.health.smithi001.stderr:fsid c8ea72cf-9e99-4383-9b52-6c23d365fdfc 2024-03-25T18:37:07.456 INFO:teuthology.misc.health.smithi001.stderr:last_changed 2024-03-25T18:37:07.455998+0000 2024-03-25T18:37:07.456 INFO:teuthology.misc.health.smithi001.stderr:created 2024-03-25T18:37:07.455998+0000 2024-03-25T18:37:07.456 INFO:teuthology.misc.health.smithi001.stderr:min_mon_release 0 (unknown) 2024-03-25T18:37:07.456 INFO:teuthology.misc.health.smithi001.stderr:election_strategy: 3 2024-03-25T18:37:07.456 INFO:teuthology.misc.health.smithi001.stderr:0: v1:172.21.15.1:6789/0 mon.noname-a 2024-03-25T18:37:07.456 INFO:teuthology.misc.health.smithi001.stderr:1: v1:172.21.15.115:6789/0 mon.noname-c 2024-03-25T18:37:07.457 INFO:teuthology.misc.health.smithi001.stderr:2: v1:172.21.15.1:6790/0 mon.noname-b 2024-03-25T18:37:07.457 INFO:teuthology.misc.health.smithi001.stderr: 2024-03-25T18:37:07.458 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:37:07.455+0000 7f751d9a8640 1 Processor -- start 2024-03-25T18:37:07.458 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:37:07.455+0000 7f751d9a8640 1 -- start start 2024-03-25T18:37:07.458 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:37:07.455+0000 7f751d9a8640 10 monclient: init 2024-03-25T18:37:07.458 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:37:07.455+0000 7f751d9a8640 10 monclient: _reopen_session rank -1 2024-03-25T18:37:07.458 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:37:07.455+0000 7f751d9a8640 10 monclient: _add_conns ranks=[2,0,1] 2024-03-25T18:37:07.458 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:37:07.455+0000 7f751d9a8640 10 monclient(hunting): picked mon.noname-b con 0x7f75180b4ff0 addr v1:172.21.15.1:6790/0 2024-03-25T18:37:07.459 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:37:07.456+0000 7f751d9a8640 10 monclient(hunting): picked mon.noname-a con 0x7f75180b4670 addr v1:172.21.15.1:6789/0 2024-03-25T18:37:07.459 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:37:07.456+0000 7f751d9a8640 10 monclient(hunting): picked mon.noname-c con 0x7f751813c480 addr v1:172.21.15.115:6789/0 2024-03-25T18:37:07.459 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:37:07.456+0000 7f751d9a8640 1 -- --> v1:172.21.15.1:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f75180ab0e0 con 0x7f75180b4670 2024-03-25T18:37:07.459 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:37:07.456+0000 7f751d9a8640 1 -- --> v1:172.21.15.115:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f75180ab3c0 con 0x7f751813c480 2024-03-25T18:37:07.459 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:37:07.456+0000 7f751d9a8640 1 -- --> v1:172.21.15.1:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f75180a9f30 con 0x7f75180b4ff0 2024-03-25T18:37:07.459 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:37:07.456+0000 7f751d9a8640 10 monclient(hunting): _renew_subs 2024-03-25T18:37:07.459 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:37:07.456+0000 7f751d9a8640 10 monclient(hunting): authenticate will time out at 8107.926884s 2024-03-25T18:37:07.462 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:37:07.461+0000 7f75177fe640 1 --1- >> v1:172.21.15.115:6789/0 conn(0x7f751813c480 0x7f751814dc00 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.115:6789/0 says I am v1:172.21.15.1:38808/0 (socket says 172.21.15.1:38808) 2024-03-25T18:37:07.463 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:37:07.461+0000 7f75177fe640 1 -- 172.21.15.1:0/748236894 learned_addr learned my addr 172.21.15.1:0/748236894 (peer_addr_for_me v1:172.21.15.1:0/0) 2024-03-25T18:37:07.469 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:37:07.467+0000 7f7515ffb640 1 -- 172.21.15.1:0/748236894 <== mon.1 v1:172.21.15.115:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1791515786 0 0) 0x7f75180ab3c0 con 0x7f751813c480 2024-03-25T18:37:07.469 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:37:07.467+0000 7f7515ffb640 10 monclient(hunting): _negotiate 2024-03-25T18:37:07.469 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:37:07.467+0000 7f7515ffb640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:37:07.469 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:37:07.467+0000 7f7515ffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:37:07.469 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:37:07.467+0000 7f7515ffb640 10 monclient(hunting): my global_id is 4678 2024-03-25T18:37:07.469 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:37:07.467+0000 7f7515ffb640 1 -- 172.21.15.1:0/748236894 --> v1:172.21.15.115:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f7500003520 con 0x7f751813c480 2024-03-25T18:37:07.475 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:37:07.473+0000 7f7515ffb640 1 -- 172.21.15.1:0/748236894 <== mon.1 v1:172.21.15.115:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1719876425 0 0) 0x7f7500003520 con 0x7f751813c480 2024-03-25T18:37:07.475 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:37:07.473+0000 7f7515ffb640 1 -- 172.21.15.1:0/748236894 >> v1:172.21.15.1:6790/0 conn(0x7f75180b4ff0 legacy=0x7f751813bd70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-25T18:37:07.475 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:37:07.473+0000 7f7515ffb640 1 -- 172.21.15.1:0/748236894 >> v1:172.21.15.1:6789/0 conn(0x7f75180b4670 legacy=0x7f75181425c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:37:07.476 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:37:07.474+0000 7f7515ffb640 10 monclient: _finish_hunting 0 2024-03-25T18:37:07.476 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:37:07.474+0000 7f7515ffb640 1 monclient: found mon.noname-c 2024-03-25T18:37:07.476 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:37:07.474+0000 7f7515ffb640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:37:07.476 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:37:07.474+0000 7f7515ffb640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.115:6789/0 2024-03-25T18:37:07.476 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:37:07.474+0000 7f7515ffb640 1 -- 172.21.15.1:0/748236894 --> v1:172.21.15.115:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f75180a9b70 con 0x7f751813c480 2024-03-25T18:37:07.476 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:37:07.474+0000 7f7515ffb640 10 monclient: _finish_auth 0 2024-03-25T18:37:07.476 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:37:07.474+0000 7f7515ffb640 10 monclient: _check_auth_tickets 2024-03-25T18:37:07.476 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:37:07.474+0000 7f7515ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:37:07.476 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:37:07.474+0000 7f7515ffb640 1 -- 172.21.15.1:0/748236894 <== mon.1 v1:172.21.15.115:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1586965025 0 0) 0x7f7510001a10 con 0x7f751813c480 2024-03-25T18:37:07.476 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:37:07.474+0000 7f751d9a8640 5 monclient: authenticate success, global_id 4678 2024-03-25T18:37:07.476 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:37:07.474+0000 7f751d9a8640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T18:37:07.476 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:37:07.474+0000 7f7515ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:37:07.476 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:37:07.474+0000 7f7515ffb640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-03-25T18:37:07.476 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:37:07.474+0000 7f7515ffb640 10 monclient: dump: 2024-03-25T18:37:07.477 INFO:teuthology.misc.health.smithi001.stderr:epoch 1 2024-03-25T18:37:07.477 INFO:teuthology.misc.health.smithi001.stderr:fsid c8ea72cf-9e99-4383-9b52-6c23d365fdfc 2024-03-25T18:37:07.477 INFO:teuthology.misc.health.smithi001.stderr:last_changed 2024-03-25T18:25:50.510519+0000 2024-03-25T18:37:07.477 INFO:teuthology.misc.health.smithi001.stderr:created 2024-03-25T18:25:50.510519+0000 2024-03-25T18:37:07.477 INFO:teuthology.misc.health.smithi001.stderr:min_mon_release 19 (squid) 2024-03-25T18:37:07.477 INFO:teuthology.misc.health.smithi001.stderr:election_strategy: 3 2024-03-25T18:37:07.477 INFO:teuthology.misc.health.smithi001.stderr:0: v1:172.21.15.1:6789/0 mon.a 2024-03-25T18:37:07.477 INFO:teuthology.misc.health.smithi001.stderr:1: v1:172.21.15.115:6789/0 mon.b 2024-03-25T18:37:07.477 INFO:teuthology.misc.health.smithi001.stderr:2: v1:172.21.15.1:6790/0 mon.c 2024-03-25T18:37:07.477 INFO:teuthology.misc.health.smithi001.stderr: 2024-03-25T18:37:07.477 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:37:07.474+0000 7f751d9a8640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T18:37:07.484 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:37:07.483+0000 7f7515ffb640 1 -- 172.21.15.1:0/748236894 <== mon.1 v1:172.21.15.115:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f7510001ca0 con 0x7f751813c480 2024-03-25T18:37:07.485 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:37:07.483+0000 7f7515ffb640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:37:07.485 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:37:07.483+0000 7f7515ffb640 1 -- 172.21.15.1:0/748236894 <== mon.1 v1:172.21.15.115:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1586965025 0 0) 0x7f75100020b0 con 0x7f751813c480 2024-03-25T18:37:07.485 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:37:07.483+0000 7f7515ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:37:07.485 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:37:07.483+0000 7f7515ffb640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-25T18:37:07.485 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:37:07.483+0000 7f7515ffb640 10 monclient: dump: 2024-03-25T18:37:07.485 INFO:teuthology.misc.health.smithi001.stderr:epoch 1 2024-03-25T18:37:07.485 INFO:teuthology.misc.health.smithi001.stderr:fsid c8ea72cf-9e99-4383-9b52-6c23d365fdfc 2024-03-25T18:37:07.485 INFO:teuthology.misc.health.smithi001.stderr:last_changed 2024-03-25T18:25:50.510519+0000 2024-03-25T18:37:07.485 INFO:teuthology.misc.health.smithi001.stderr:created 2024-03-25T18:25:50.510519+0000 2024-03-25T18:37:07.485 INFO:teuthology.misc.health.smithi001.stderr:min_mon_release 19 (squid) 2024-03-25T18:37:07.485 INFO:teuthology.misc.health.smithi001.stderr:election_strategy: 3 2024-03-25T18:37:07.485 INFO:teuthology.misc.health.smithi001.stderr:0: v1:172.21.15.1:6789/0 mon.a 2024-03-25T18:37:07.485 INFO:teuthology.misc.health.smithi001.stderr:1: v1:172.21.15.115:6789/0 mon.b 2024-03-25T18:37:07.485 INFO:teuthology.misc.health.smithi001.stderr:2: v1:172.21.15.1:6790/0 mon.c 2024-03-25T18:37:07.486 INFO:teuthology.misc.health.smithi001.stderr: 2024-03-25T18:37:07.486 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:37:07.483+0000 7f751d9a8640 10 monclient: get_monmap_and_config success 2024-03-25T18:37:07.486 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:37:07.483+0000 7f751d9a8640 4 set_mon_vals no callback set 2024-03-25T18:37:07.488 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:37:07.484+0000 7f751d9a8640 10 monclient: shutdown 2024-03-25T18:37:07.488 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:37:07.484+0000 7f751d9a8640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:37:07.488 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:37:07.484+0000 7f751d9a8640 1 -- 172.21.15.1:0/748236894 >> v1:172.21.15.115:6789/0 conn(0x7f751813c480 legacy=0x7f751814dc00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:37:07.488 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:37:07.484+0000 7f751d9a8640 1 -- 172.21.15.1:0/748236894 shutdown_connections 2024-03-25T18:37:07.488 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:37:07.484+0000 7f751d9a8640 1 -- 172.21.15.1:0/748236894 >> 172.21.15.1:0/748236894 conn(0x7f75180617a0 msgr2=0x7f7518061bb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:37:07.488 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:37:07.484+0000 7f751d9a8640 1 -- 172.21.15.1:0/748236894 shutdown_connections 2024-03-25T18:37:07.488 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:37:07.484+0000 7f751d9a8640 1 -- 172.21.15.1:0/748236894 wait complete. 2024-03-25T18:37:07.488 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:37:07.485+0000 7f751d9a8640 10 monclient: build_initial_monmap 2024-03-25T18:37:07.488 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:37:07.485+0000 7f751d9a8640 10 monclient: monmap: 2024-03-25T18:37:07.488 INFO:teuthology.misc.health.smithi001.stderr:epoch 0 2024-03-25T18:37:07.489 INFO:teuthology.misc.health.smithi001.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T18:37:07.489 INFO:teuthology.misc.health.smithi001.stderr:last_changed 0.000000 2024-03-25T18:37:07.489 INFO:teuthology.misc.health.smithi001.stderr:created 0.000000 2024-03-25T18:37:07.489 INFO:teuthology.misc.health.smithi001.stderr:min_mon_release 0 (unknown) 2024-03-25T18:37:07.489 INFO:teuthology.misc.health.smithi001.stderr:election_strategy: 1 2024-03-25T18:37:07.489 INFO:teuthology.misc.health.smithi001.stderr:0: v1:172.21.15.1:6789/0 mon.noname-a 2024-03-25T18:37:07.489 INFO:teuthology.misc.health.smithi001.stderr:1: v1:172.21.15.115:6789/0 mon.noname-b 2024-03-25T18:37:07.489 INFO:teuthology.misc.health.smithi001.stderr:2: v1:172.21.15.1:6790/0 mon.noname-c 2024-03-25T18:37:07.489 INFO:teuthology.misc.health.smithi001.stderr: 2024-03-25T18:37:07.489 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:37:07.485+0000 7f751d9a8640 1 librados: starting msgr at 2024-03-25T18:37:07.489 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:37:07.485+0000 7f751d9a8640 1 librados: starting objecter 2024-03-25T18:37:07.489 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:37:07.485+0000 7f751d9a8640 1 Processor -- start 2024-03-25T18:37:07.489 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:37:07.485+0000 7f751d9a8640 1 -- start start 2024-03-25T18:37:07.489 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:37:07.485+0000 7f751d9a8640 1 librados: setting wanted keys 2024-03-25T18:37:07.489 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:37:07.485+0000 7f751d9a8640 1 librados: calling monclient init 2024-03-25T18:37:07.490 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:37:07.485+0000 7f751d9a8640 10 monclient: init 2024-03-25T18:37:07.490 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:37:07.486+0000 7f751d9a8640 10 monclient: _reopen_session rank -1 2024-03-25T18:37:07.490 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:37:07.486+0000 7f751d9a8640 10 monclient: _add_conns ranks=[2,0,1] 2024-03-25T18:37:07.490 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:37:07.486+0000 7f751d9a8640 10 monclient(hunting): picked mon.noname-c con 0x7f75180b4670 addr v1:172.21.15.1:6790/0 2024-03-25T18:37:07.490 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:37:07.486+0000 7f751d9a8640 10 monclient(hunting): picked mon.noname-a con 0x7f75180b4ff0 addr v1:172.21.15.1:6789/0 2024-03-25T18:37:07.490 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:37:07.486+0000 7f751d9a8640 10 monclient(hunting): picked mon.noname-b con 0x7f751813c480 addr v1:172.21.15.115:6789/0 2024-03-25T18:37:07.490 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:37:07.486+0000 7f751d9a8640 1 -- --> v1:172.21.15.1:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f75181465e0 con 0x7f75180b4ff0 2024-03-25T18:37:07.490 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:37:07.486+0000 7f751d9a8640 1 -- --> v1:172.21.15.115:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f751813f870 con 0x7f751813c480 2024-03-25T18:37:07.490 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:37:07.486+0000 7f751d9a8640 1 -- --> v1:172.21.15.1:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7518130cc0 con 0x7f75180b4670 2024-03-25T18:37:07.490 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:37:07.486+0000 7f751d9a8640 10 monclient(hunting): _renew_subs 2024-03-25T18:37:07.490 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:37:07.486+0000 7f751d9a8640 10 monclient(hunting): authenticate will time out at 8107.957283s 2024-03-25T18:37:07.501 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:37:07.500+0000 7f75167fc640 1 --1- >> v1:172.21.15.1:6789/0 conn(0x7f75180b4ff0 0x7f75181451f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.1:6789/0 says I am v1:172.21.15.1:41122/0 (socket says 172.21.15.1:41122) 2024-03-25T18:37:07.501 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:37:07.500+0000 7f75167fc640 1 -- 172.21.15.1:0/3941596709 learned_addr learned my addr 172.21.15.1:0/3941596709 (peer_addr_for_me v1:172.21.15.1:0/0) 2024-03-25T18:37:07.502 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:37:07.500+0000 7f74fffff640 10 client.?.objecter ms_handle_connect 0x7f75180b4ff0 2024-03-25T18:37:07.502 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:37:07.500+0000 7f74fffff640 10 client.?.objecter resend_mon_ops 2024-03-25T18:37:07.506 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:37:07.505+0000 7f74fffff640 10 client.?.objecter ms_handle_connect 0x7f75180b4670 2024-03-25T18:37:07.506 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:37:07.505+0000 7f74fffff640 10 client.?.objecter resend_mon_ops 2024-03-25T18:37:07.508 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:37:07.506+0000 7f74fffff640 10 client.?.objecter ms_handle_connect 0x7f751813c480 2024-03-25T18:37:07.508 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:37:07.506+0000 7f74fffff640 10 client.?.objecter resend_mon_ops 2024-03-25T18:37:07.509 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:37:07.507+0000 7f74fffff640 1 -- 172.21.15.1:0/3941596709 <== mon.0 v1:172.21.15.1:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1030041047 0 0) 0x7f75181465e0 con 0x7f75180b4ff0 2024-03-25T18:37:07.509 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:37:07.507+0000 7f74fffff640 10 monclient(hunting): _negotiate 2024-03-25T18:37:07.509 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:37:07.507+0000 7f74fffff640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:37:07.509 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:37:07.507+0000 7f74fffff640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:37:07.509 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:37:07.507+0000 7f74fffff640 10 monclient(hunting): my global_id is 4665 2024-03-25T18:37:07.509 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:37:07.507+0000 7f74fffff640 1 -- 172.21.15.1:0/3941596709 --> v1:172.21.15.1:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f7504004eb0 con 0x7f75180b4ff0 2024-03-25T18:37:07.515 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:37:07.514+0000 7f74fffff640 1 -- 172.21.15.1:0/3941596709 <== mon.0 v1:172.21.15.1:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 4261408818 0 0) 0x7f7504004eb0 con 0x7f75180b4ff0 2024-03-25T18:37:07.515 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:37:07.514+0000 7f74fffff640 1 -- 172.21.15.1:0/3941596709 --> v1:172.21.15.1:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f75181465e0 con 0x7f75180b4ff0 2024-03-25T18:37:07.515 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:37:07.514+0000 7f74fffff640 1 -- 172.21.15.1:0/3941596709 <== mon.0 v1:172.21.15.1:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1586965025 0 0) 0x7f750c001a30 con 0x7f75180b4ff0 2024-03-25T18:37:07.516 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:37:07.514+0000 7f74fffff640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:37:07.516 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:37:07.514+0000 7f74fffff640 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2024-03-25T18:37:07.516 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:37:07.514+0000 7f74fffff640 10 monclient(hunting): dump: 2024-03-25T18:37:07.516 INFO:teuthology.misc.health.smithi001.stderr:epoch 1 2024-03-25T18:37:07.516 INFO:teuthology.misc.health.smithi001.stderr:fsid c8ea72cf-9e99-4383-9b52-6c23d365fdfc 2024-03-25T18:37:07.516 INFO:teuthology.misc.health.smithi001.stderr:last_changed 2024-03-25T18:25:50.510519+0000 2024-03-25T18:37:07.516 INFO:teuthology.misc.health.smithi001.stderr:created 2024-03-25T18:25:50.510519+0000 2024-03-25T18:37:07.516 INFO:teuthology.misc.health.smithi001.stderr:min_mon_release 19 (squid) 2024-03-25T18:37:07.516 INFO:teuthology.misc.health.smithi001.stderr:election_strategy: 3 2024-03-25T18:37:07.516 INFO:teuthology.misc.health.smithi001.stderr:0: v1:172.21.15.1:6789/0 mon.a 2024-03-25T18:37:07.516 INFO:teuthology.misc.health.smithi001.stderr:1: v1:172.21.15.115:6789/0 mon.b 2024-03-25T18:37:07.516 INFO:teuthology.misc.health.smithi001.stderr:2: v1:172.21.15.1:6790/0 mon.c 2024-03-25T18:37:07.516 INFO:teuthology.misc.health.smithi001.stderr: 2024-03-25T18:37:07.516 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:37:07.514+0000 7f74fffff640 10 monclient(hunting): _finish_auth 0 2024-03-25T18:37:07.522 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:37:07.519+0000 7f74fffff640 1 -- 172.21.15.1:0/3941596709 <== mon.0 v1:172.21.15.1:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 1116819433 0 0) 0x7f75181465e0 con 0x7f75180b4ff0 2024-03-25T18:37:07.522 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:37:07.519+0000 7f74fffff640 1 -- 172.21.15.1:0/3941596709 >> v1:172.21.15.1:6790/0 conn(0x7f75180b4670 legacy=0x7f751813c910 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:37:07.522 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:37:07.519+0000 7f74fffff640 1 -- 172.21.15.1:0/3941596709 >> v1:172.21.15.115:6789/0 conn(0x7f751813c480 legacy=0x7f7518145900 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:37:07.522 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:37:07.519+0000 7f74fffff640 10 monclient: _finish_hunting 0 2024-03-25T18:37:07.522 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:37:07.519+0000 7f74fffff640 1 monclient: found mon.a 2024-03-25T18:37:07.522 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:37:07.519+0000 7f74fffff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:37:07.522 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:37:07.519+0000 7f74fffff640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.1:6789/0 2024-03-25T18:37:07.522 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:37:07.519+0000 7f74fffff640 1 -- 172.21.15.1:0/3941596709 --> v1:172.21.15.1:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7518144dd0 con 0x7f75180b4ff0 2024-03-25T18:37:07.522 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:37:07.519+0000 7f74fffff640 10 monclient: _finish_auth 0 2024-03-25T18:37:07.522 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:37:07.519+0000 7f74fffff640 10 monclient: _check_auth_tickets 2024-03-25T18:37:07.522 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:37:07.519+0000 7f74fffff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:37:07.523 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:37:07.519+0000 7f751d9a8640 5 monclient: authenticate success, global_id 4665 2024-03-25T18:37:07.523 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:37:07.519+0000 7f751d9a8640 10 monclient: _renew_subs 2024-03-25T18:37:07.523 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:37:07.519+0000 7f751d9a8640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.1:6789/0 2024-03-25T18:37:07.523 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:37:07.519+0000 7f751d9a8640 1 -- 172.21.15.1:0/3941596709 --> v1:172.21.15.1:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f751813eb60 con 0x7f75180b4ff0 2024-03-25T18:37:07.523 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:37:07.519+0000 7f751d9a8640 10 client.4665.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T18:37:07.523 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:37:07.519+0000 7f751d9a8640 10 monclient: _renew_subs 2024-03-25T18:37:07.523 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:37:07.519+0000 7f751d9a8640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.1:6789/0 2024-03-25T18:37:07.523 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:37:07.519+0000 7f751d9a8640 1 -- 172.21.15.1:0/3941596709 --> v1:172.21.15.1:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f7518146010 con 0x7f75180b4ff0 2024-03-25T18:37:07.523 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:37:07.519+0000 7f751d9a8640 1 librados: init done 2024-03-25T18:37:07.523 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:37:07.520+0000 7f751d9a8640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T18:37:07.523 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:37:07.520+0000 7f751d9a8640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:37:07.523 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:37:07.520+0000 7f751d9a8640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.1:6789/0 2024-03-25T18:37:07.523 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:37:07.520+0000 7f751d9a8640 1 -- 172.21.15.1:0/3941596709 --> v1:172.21.15.1:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f751806dbd0 con 0x7f75180b4ff0 2024-03-25T18:37:07.847 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:37:07.845+0000 7f74fffff640 1 -- 172.21.15.1:0/3941596709 <== mon.0 v1:172.21.15.1:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f750c001cb0 con 0x7f75180b4ff0 2024-03-25T18:37:07.847 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:37:07.845+0000 7f74fffff640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:37:07.847 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:37:07.845+0000 7f74fffff640 1 -- 172.21.15.1:0/3941596709 <== mon.0 v1:172.21.15.1:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1586965025 0 0) 0x7f750c003a60 con 0x7f75180b4ff0 2024-03-25T18:37:07.847 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:37:07.845+0000 7f74fffff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:37:07.847 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:37:07.845+0000 7f74fffff640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-25T18:37:07.847 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:37:07.845+0000 7f74fffff640 10 monclient: dump: 2024-03-25T18:37:07.847 INFO:teuthology.misc.health.smithi001.stderr:epoch 1 2024-03-25T18:37:07.847 INFO:teuthology.misc.health.smithi001.stderr:fsid c8ea72cf-9e99-4383-9b52-6c23d365fdfc 2024-03-25T18:37:07.847 INFO:teuthology.misc.health.smithi001.stderr:last_changed 2024-03-25T18:25:50.510519+0000 2024-03-25T18:37:07.847 INFO:teuthology.misc.health.smithi001.stderr:created 2024-03-25T18:25:50.510519+0000 2024-03-25T18:37:07.847 INFO:teuthology.misc.health.smithi001.stderr:min_mon_release 19 (squid) 2024-03-25T18:37:07.847 INFO:teuthology.misc.health.smithi001.stderr:election_strategy: 3 2024-03-25T18:37:07.847 INFO:teuthology.misc.health.smithi001.stderr:0: v1:172.21.15.1:6789/0 mon.a 2024-03-25T18:37:07.848 INFO:teuthology.misc.health.smithi001.stderr:1: v1:172.21.15.115:6789/0 mon.b 2024-03-25T18:37:07.848 INFO:teuthology.misc.health.smithi001.stderr:2: v1:172.21.15.1:6790/0 mon.c 2024-03-25T18:37:07.848 INFO:teuthology.misc.health.smithi001.stderr: 2024-03-25T18:37:07.848 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:37:07.845+0000 7f75157fa640 4 set_mon_vals no callback set 2024-03-25T18:37:07.848 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:37:07.846+0000 7f74fffff640 1 -- 172.21.15.1:0/3941596709 <== mon.0 v1:172.21.15.1:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 21787975 0 0) 0x7f750c003c40 con 0x7f75180b4ff0 2024-03-25T18:37:07.848 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:37:07.846+0000 7f7516ffd640 10 monclient: get_auth_request con 0x7f750406dd00 auth_method 0 2024-03-25T18:37:07.848 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:37:07.846+0000 7f74fffff640 10 client.4665.objecter ms_handle_connect 0x7f750406dd00 2024-03-25T18:37:07.855 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:37:07.854+0000 7f74fffff640 1 -- 172.21.15.1:0/3941596709 <== mon.0 v1:172.21.15.1:6789/0 8 ==== osd_map(11..11 src has 1..11) v4 ==== 1645+0+0 (unknown 2504898309 0 0) 0x7f750c054870 con 0x7f75180b4ff0 2024-03-25T18:37:07.856 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:37:07.854+0000 7f74fffff640 10 client.4665.objecter ms_dispatch 0x7f7518000b90 osd_map(11..11 src has 1..11) v4 2024-03-25T18:37:07.856 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:37:07.854+0000 7f74fffff640 3 client.4665.objecter handle_osd_map got epochs [11,11] > 0 2024-03-25T18:37:07.856 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:37:07.854+0000 7f74fffff640 3 client.4665.objecter handle_osd_map decoding full epoch 11 2024-03-25T18:37:07.856 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:37:07.854+0000 7f74fffff640 20 client.4665.objecter dump_active .. 0 homeless 2024-03-25T18:37:07.856 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:37:07.854+0000 7f74fffff640 1 -- 172.21.15.1:0/3941596709 <== mon.0 v1:172.21.15.1:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f750c050380 con 0x7f75180b4ff0 2024-03-25T18:37:07.856 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:37:07.854+0000 7f74fffff640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:37:07.856 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:37:07.854+0000 7f74fffff640 10 monclient: _finish_command 1 = system:0 2024-03-25T18:37:08.077 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:37:08.075+0000 7f751d9a8640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:37:08.077 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:37:08.075+0000 7f751d9a8640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:37:08.077 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:37:08.076+0000 7f751d9a8640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.1:6789/0 2024-03-25T18:37:08.077 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:37:08.076+0000 7f751d9a8640 1 -- 172.21.15.1:0/3941596709 --> v1:172.21.15.1:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f751813fec0 con 0x7f75180b4ff0 2024-03-25T18:37:08.121 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:37:08.119+0000 7f74fffff640 1 -- 172.21.15.1:0/3941596709 <== mon.0 v1:172.21.15.1:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v11) v1 ==== 74+0+7100 (unknown 238102980 0 2126609903) 0x7f750c0522b0 con 0x7f75180b4ff0 2024-03-25T18:37:08.121 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:37:08.119+0000 7f74fffff640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:37:08.121 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:37:08.119+0000 7f74fffff640 10 monclient: _finish_command 2 = system:0 2024-03-25T18:37:08.121 INFO:teuthology.misc.health.smithi001.stdout: 2024-03-25T18:37:08.122 INFO:teuthology.misc.health.smithi001.stdout:{"epoch":11,"fsid":"c8ea72cf-9e99-4383-9b52-6c23d365fdfc","created":"2024-03-25T18:26:38.856638+0000","modified":"2024-03-25T18:36:43.033551+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T18:36:43.033551+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"b9f34922-3a84-49be-8426-807c1eceaefc","up":0,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["autoout","exists","new"]},{"osd":1,"uuid":"42bf306f-55e3-4634-9874-8b4b73ac6644","up":0,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["autoout","exists","new"]},{"osd":2,"uuid":"292cb23a-18a9-4207-a7cc-4d5727feaec0","up":0,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["autoout","exists","new"]},{"osd":3,"uuid":"45e0e9ab-898d-4673-9590-b4117bc5e84a","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"831a5a09-73ab-4a61-9872-088e76bdbc66","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"88cd903d-3b55-4840-9396-6e3cfe005282","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"89db7355-bc9d-46f4-8ab0-e8d9e744e026","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"cac47bee-0914-4f08-896f-348838882866","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":65536,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":65536,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":65536,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T18:37:08.125 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:37:08.122+0000 7f751d9a8640 10 librados: watch_flush enter 2024-03-25T18:37:08.125 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:37:08.123+0000 7f751d9a8640 10 librados: watch_flush exit 2024-03-25T18:37:08.125 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:37:08.123+0000 7f751d9a8640 20 client.4665.objecter shutdown clearing up homeless session... 2024-03-25T18:37:08.125 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:37:08.123+0000 7f751d9a8640 10 client.4665.objecter successfully canceled tick 2024-03-25T18:37:08.125 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:37:08.123+0000 7f751d9a8640 1 -- 172.21.15.1:0/3941596709 >> v1:172.21.15.115:6800/4237379176 conn(0x7f750406dd00 legacy=0x7f75040701c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:37:08.125 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:37:08.123+0000 7f751d9a8640 10 monclient: shutdown 2024-03-25T18:37:08.125 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:37:08.123+0000 7f751d9a8640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:37:08.125 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:37:08.123+0000 7f751d9a8640 1 -- 172.21.15.1:0/3941596709 >> v1:172.21.15.1:6789/0 conn(0x7f75180b4ff0 legacy=0x7f75181451f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:37:08.125 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:37:08.123+0000 7f751d9a8640 1 -- 172.21.15.1:0/3941596709 shutdown_connections 2024-03-25T18:37:08.125 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:37:08.123+0000 7f751d9a8640 1 -- 172.21.15.1:0/3941596709 >> 172.21.15.1:0/3941596709 conn(0x7f75180617a0 msgr2=0x7f7518140de0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:37:08.125 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:37:08.123+0000 7f751d9a8640 1 -- 172.21.15.1:0/3941596709 shutdown_connections 2024-03-25T18:37:08.125 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:37:08.123+0000 7f751d9a8640 1 -- 172.21.15.1:0/3941596709 wait complete. 2024-03-25T18:37:08.126 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:37:08.123+0000 7f751d9a8640 1 librados: shutdown 2024-03-25T18:37:08.138 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T18:37:08.191 INFO:tasks.ceph.mon.a.smithi001.stderr:==00:00:10:46.749 34876== Warning: unimplemented fcntl command: 1036 2024-03-25T18:37:08.256 INFO:tasks.ceph.mon.a.smithi001.stderr:==00:00:10:46.815 34876== Warning: unimplemented fcntl command: 1036 2024-03-25T18:37:08.285 INFO:tasks.ceph.mon.a.smithi001.stderr:==00:00:10:46.844 34876== Warning: unimplemented fcntl command: 1036 2024-03-25T18:37:08.302 INFO:tasks.ceph.mon.c.smithi001.stderr:==00:00:10:46.861 34877== Warning: unimplemented fcntl command: 1036 2024-03-25T18:37:08.308 INFO:tasks.ceph.mon.b.smithi115.stderr:==00:00:10:46.792 34836== Warning: unimplemented fcntl command: 1036 2024-03-25T18:37:08.311 INFO:tasks.ceph.mon.c.smithi001.stderr:==00:00:10:46.870 34877== Warning: unimplemented fcntl command: 1036 2024-03-25T18:37:08.314 INFO:tasks.ceph.mon.b.smithi115.stderr:==00:00:10:46.798 34836== Warning: unimplemented fcntl command: 1036 2024-03-25T18:37:08.349 INFO:tasks.ceph.mon.b.smithi115.stderr:==00:00:10:46.833 34836== Warning: unimplemented fcntl command: 1036 2024-03-25T18:37:08.355 INFO:tasks.ceph.mon.c.smithi001.stderr:==00:00:10:46.914 34877== Warning: unimplemented fcntl command: 1036 2024-03-25T18:37:14.944 DEBUG:teuthology.orchestra.run.smithi001:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T18:37:15.054 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:37:15.053+0000 7fa70fbed640 10 monclient: get_monmap_and_config 2024-03-25T18:37:15.054 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:37:15.053+0000 7fa70fbed640 10 monclient: build_initial_monmap 2024-03-25T18:37:15.054 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:37:15.053+0000 7fa70fbed640 10 monclient: monmap: 2024-03-25T18:37:15.055 INFO:teuthology.misc.health.smithi001.stderr:epoch 0 2024-03-25T18:37:15.055 INFO:teuthology.misc.health.smithi001.stderr:fsid c8ea72cf-9e99-4383-9b52-6c23d365fdfc 2024-03-25T18:37:15.055 INFO:teuthology.misc.health.smithi001.stderr:last_changed 2024-03-25T18:37:15.054655+0000 2024-03-25T18:37:15.055 INFO:teuthology.misc.health.smithi001.stderr:created 2024-03-25T18:37:15.054655+0000 2024-03-25T18:37:15.055 INFO:teuthology.misc.health.smithi001.stderr:min_mon_release 0 (unknown) 2024-03-25T18:37:15.055 INFO:teuthology.misc.health.smithi001.stderr:election_strategy: 3 2024-03-25T18:37:15.055 INFO:teuthology.misc.health.smithi001.stderr:0: v1:172.21.15.1:6789/0 mon.noname-a 2024-03-25T18:37:15.055 INFO:teuthology.misc.health.smithi001.stderr:1: v1:172.21.15.115:6789/0 mon.noname-c 2024-03-25T18:37:15.055 INFO:teuthology.misc.health.smithi001.stderr:2: v1:172.21.15.1:6790/0 mon.noname-b 2024-03-25T18:37:15.055 INFO:teuthology.misc.health.smithi001.stderr: 2024-03-25T18:37:15.056 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:37:15.053+0000 7fa70fbed640 1 Processor -- start 2024-03-25T18:37:15.056 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:37:15.054+0000 7fa70fbed640 1 -- start start 2024-03-25T18:37:15.056 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:37:15.054+0000 7fa70fbed640 10 monclient: init 2024-03-25T18:37:15.056 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:37:15.054+0000 7fa70fbed640 10 monclient: _reopen_session rank -1 2024-03-25T18:37:15.056 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:37:15.054+0000 7fa70fbed640 10 monclient: _add_conns ranks=[0,2,1] 2024-03-25T18:37:15.057 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:37:15.054+0000 7fa70fbed640 10 monclient(hunting): picked mon.noname-a con 0x7fa7080b4ff0 addr v1:172.21.15.1:6789/0 2024-03-25T18:37:15.057 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:37:15.054+0000 7fa70fbed640 10 monclient(hunting): picked mon.noname-b con 0x7fa7080b4670 addr v1:172.21.15.1:6790/0 2024-03-25T18:37:15.057 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:37:15.054+0000 7fa70fbed640 10 monclient(hunting): picked mon.noname-c con 0x7fa70813c480 addr v1:172.21.15.115:6789/0 2024-03-25T18:37:15.057 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:37:15.054+0000 7fa70fbed640 1 -- --> v1:172.21.15.1:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa7080ab0e0 con 0x7fa7080b4ff0 2024-03-25T18:37:15.057 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:37:15.054+0000 7fa70fbed640 1 -- --> v1:172.21.15.115:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa7080ab3c0 con 0x7fa70813c480 2024-03-25T18:37:15.057 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:37:15.054+0000 7fa70fbed640 1 -- --> v1:172.21.15.1:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa7080a9f30 con 0x7fa7080b4670 2024-03-25T18:37:15.057 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:37:15.054+0000 7fa70fbed640 10 monclient(hunting): _renew_subs 2024-03-25T18:37:15.057 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:37:15.054+0000 7fa70fbed640 10 monclient(hunting): authenticate will time out at 8115.525497s 2024-03-25T18:37:15.064 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:37:15.063+0000 7fa70d989640 1 --1- >> v1:172.21.15.1:6789/0 conn(0x7fa7080b4ff0 0x7fa70813bd70 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.1:6789/0 says I am v1:172.21.15.1:43758/0 (socket says 172.21.15.1:43758) 2024-03-25T18:37:15.064 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:37:15.063+0000 7fa70d989640 1 -- 172.21.15.1:0/2014425186 learned_addr learned my addr 172.21.15.1:0/2014425186 (peer_addr_for_me v1:172.21.15.1:0/0) 2024-03-25T18:37:15.079 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:37:15.077+0000 7fa70c987640 1 -- 172.21.15.1:0/2014425186 <== mon.0 v1:172.21.15.1:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4248553165 0 0) 0x7fa7080ab0e0 con 0x7fa7080b4ff0 2024-03-25T18:37:15.079 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:37:15.077+0000 7fa70c987640 10 monclient(hunting): _negotiate 2024-03-25T18:37:15.079 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:37:15.077+0000 7fa70c987640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:37:15.079 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:37:15.077+0000 7fa70c987640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:37:15.079 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:37:15.077+0000 7fa70c987640 10 monclient(hunting): my global_id is 4668 2024-03-25T18:37:15.079 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:37:15.077+0000 7fa70c987640 1 -- 172.21.15.1:0/2014425186 --> v1:172.21.15.1:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fa6f4003520 con 0x7fa7080b4ff0 2024-03-25T18:37:15.083 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:37:15.082+0000 7fa70c987640 1 -- 172.21.15.1:0/2014425186 <== mon.1 v1:172.21.15.115:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2990947344 0 0) 0x7fa7080ab3c0 con 0x7fa70813c480 2024-03-25T18:37:15.083 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:37:15.082+0000 7fa70c987640 10 monclient(hunting): _negotiate 2024-03-25T18:37:15.083 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:37:15.082+0000 7fa70c987640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:37:15.083 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:37:15.082+0000 7fa70c987640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:37:15.083 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:37:15.082+0000 7fa70c987640 10 monclient(hunting): my global_id is 4684 2024-03-25T18:37:15.083 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:37:15.082+0000 7fa70c987640 1 -- 172.21.15.1:0/2014425186 --> v1:172.21.15.115:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fa7080ab0e0 con 0x7fa70813c480 2024-03-25T18:37:15.088 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:37:15.086+0000 7fa70c987640 1 -- 172.21.15.1:0/2014425186 <== mon.1 v1:172.21.15.115:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1759766809 0 0) 0x7fa7080ab0e0 con 0x7fa70813c480 2024-03-25T18:37:15.088 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:37:15.086+0000 7fa70c987640 1 -- 172.21.15.1:0/2014425186 >> v1:172.21.15.1:6790/0 conn(0x7fa7080b4670 legacy=0x7fa7081425c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:37:15.088 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:37:15.086+0000 7fa70c987640 1 -- 172.21.15.1:0/2014425186 >> v1:172.21.15.1:6789/0 conn(0x7fa7080b4ff0 legacy=0x7fa70813bd70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:37:15.088 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:37:15.086+0000 7fa70c987640 10 monclient: _finish_hunting 0 2024-03-25T18:37:15.088 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:37:15.086+0000 7fa70c987640 1 monclient: found mon.noname-c 2024-03-25T18:37:15.089 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:37:15.086+0000 7fa70c987640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:37:15.089 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:37:15.086+0000 7fa70c987640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.115:6789/0 2024-03-25T18:37:15.089 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:37:15.086+0000 7fa70c987640 1 -- 172.21.15.1:0/2014425186 --> v1:172.21.15.115:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa7080a9b70 con 0x7fa70813c480 2024-03-25T18:37:15.089 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:37:15.086+0000 7fa70c987640 10 monclient: _finish_auth 0 2024-03-25T18:37:15.089 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:37:15.086+0000 7fa70c987640 10 monclient: _check_auth_tickets 2024-03-25T18:37:15.089 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:37:15.086+0000 7fa70c987640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:37:15.089 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:37:15.086+0000 7fa70c987640 1 -- 172.21.15.1:0/2014425186 <== mon.1 v1:172.21.15.115:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1586965025 0 0) 0x7fa7000019f0 con 0x7fa70813c480 2024-03-25T18:37:15.089 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:37:15.086+0000 7fa70c987640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:37:15.089 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:37:15.086+0000 7fa70c987640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-03-25T18:37:15.089 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:37:15.086+0000 7fa70c987640 10 monclient: dump: 2024-03-25T18:37:15.089 INFO:teuthology.misc.health.smithi001.stderr:epoch 1 2024-03-25T18:37:15.089 INFO:teuthology.misc.health.smithi001.stderr:fsid c8ea72cf-9e99-4383-9b52-6c23d365fdfc 2024-03-25T18:37:15.089 INFO:teuthology.misc.health.smithi001.stderr:last_changed 2024-03-25T18:25:50.510519+0000 2024-03-25T18:37:15.089 INFO:teuthology.misc.health.smithi001.stderr:created 2024-03-25T18:25:50.510519+0000 2024-03-25T18:37:15.090 INFO:teuthology.misc.health.smithi001.stderr:min_mon_release 19 (squid) 2024-03-25T18:37:15.090 INFO:teuthology.misc.health.smithi001.stderr:election_strategy: 3 2024-03-25T18:37:15.090 INFO:teuthology.misc.health.smithi001.stderr:0: v1:172.21.15.1:6789/0 mon.a 2024-03-25T18:37:15.090 INFO:teuthology.misc.health.smithi001.stderr:1: v1:172.21.15.115:6789/0 mon.b 2024-03-25T18:37:15.090 INFO:teuthology.misc.health.smithi001.stderr:2: v1:172.21.15.1:6790/0 mon.c 2024-03-25T18:37:15.090 INFO:teuthology.misc.health.smithi001.stderr: 2024-03-25T18:37:15.090 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:37:15.086+0000 7fa70fbed640 5 monclient: authenticate success, global_id 4684 2024-03-25T18:37:15.090 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:37:15.086+0000 7fa70fbed640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T18:37:15.099 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:37:15.098+0000 7fa70c987640 1 -- 172.21.15.1:0/2014425186 <== mon.1 v1:172.21.15.115:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fa700001b50 con 0x7fa70813c480 2024-03-25T18:37:15.100 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:37:15.098+0000 7fa70c987640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:37:15.100 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:37:15.098+0000 7fa70c987640 1 -- 172.21.15.1:0/2014425186 <== mon.1 v1:172.21.15.115:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1586965025 0 0) 0x7fa700001f60 con 0x7fa70813c480 2024-03-25T18:37:15.100 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:37:15.098+0000 7fa70fbed640 10 monclient: get_monmap_and_config success 2024-03-25T18:37:15.100 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:37:15.098+0000 7fa70fbed640 4 set_mon_vals no callback set 2024-03-25T18:37:15.100 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:37:15.098+0000 7fa70c987640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:37:15.100 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:37:15.098+0000 7fa70c987640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-25T18:37:15.100 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:37:15.098+0000 7fa70c987640 10 monclient: dump: 2024-03-25T18:37:15.100 INFO:teuthology.misc.health.smithi001.stderr:epoch 1 2024-03-25T18:37:15.100 INFO:teuthology.misc.health.smithi001.stderr:fsid c8ea72cf-9e99-4383-9b52-6c23d365fdfc 2024-03-25T18:37:15.100 INFO:teuthology.misc.health.smithi001.stderr:last_changed 2024-03-25T18:25:50.510519+0000 2024-03-25T18:37:15.100 INFO:teuthology.misc.health.smithi001.stderr:created 2024-03-25T18:25:50.510519+0000 2024-03-25T18:37:15.100 INFO:teuthology.misc.health.smithi001.stderr:min_mon_release 19 (squid) 2024-03-25T18:37:15.100 INFO:teuthology.misc.health.smithi001.stderr:election_strategy: 3 2024-03-25T18:37:15.101 INFO:teuthology.misc.health.smithi001.stderr:0: v1:172.21.15.1:6789/0 mon.a 2024-03-25T18:37:15.101 INFO:teuthology.misc.health.smithi001.stderr:1: v1:172.21.15.115:6789/0 mon.b 2024-03-25T18:37:15.101 INFO:teuthology.misc.health.smithi001.stderr:2: v1:172.21.15.1:6790/0 mon.c 2024-03-25T18:37:15.101 INFO:teuthology.misc.health.smithi001.stderr: 2024-03-25T18:37:15.102 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:37:15.099+0000 7fa70fbed640 10 monclient: shutdown 2024-03-25T18:37:15.102 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:37:15.099+0000 7fa70fbed640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:37:15.102 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:37:15.099+0000 7fa70fbed640 1 -- 172.21.15.1:0/2014425186 >> v1:172.21.15.115:6789/0 conn(0x7fa70813c480 legacy=0x7fa70814dc00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:37:15.102 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:37:15.099+0000 7fa70fbed640 1 -- 172.21.15.1:0/2014425186 shutdown_connections 2024-03-25T18:37:15.103 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:37:15.099+0000 7fa70fbed640 1 -- 172.21.15.1:0/2014425186 >> 172.21.15.1:0/2014425186 conn(0x7fa7080617a0 msgr2=0x7fa708061bb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:37:15.103 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:37:15.099+0000 7fa70fbed640 1 -- 172.21.15.1:0/2014425186 shutdown_connections 2024-03-25T18:37:15.103 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:37:15.099+0000 7fa70fbed640 1 -- 172.21.15.1:0/2014425186 wait complete. 2024-03-25T18:37:15.103 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:37:15.100+0000 7fa70fbed640 10 monclient: build_initial_monmap 2024-03-25T18:37:15.103 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:37:15.100+0000 7fa70fbed640 10 monclient: monmap: 2024-03-25T18:37:15.103 INFO:teuthology.misc.health.smithi001.stderr:epoch 0 2024-03-25T18:37:15.103 INFO:teuthology.misc.health.smithi001.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T18:37:15.103 INFO:teuthology.misc.health.smithi001.stderr:last_changed 0.000000 2024-03-25T18:37:15.103 INFO:teuthology.misc.health.smithi001.stderr:created 0.000000 2024-03-25T18:37:15.103 INFO:teuthology.misc.health.smithi001.stderr:min_mon_release 0 (unknown) 2024-03-25T18:37:15.103 INFO:teuthology.misc.health.smithi001.stderr:election_strategy: 1 2024-03-25T18:37:15.103 INFO:teuthology.misc.health.smithi001.stderr:0: v1:172.21.15.1:6789/0 mon.noname-a 2024-03-25T18:37:15.103 INFO:teuthology.misc.health.smithi001.stderr:1: v1:172.21.15.115:6789/0 mon.noname-b 2024-03-25T18:37:15.103 INFO:teuthology.misc.health.smithi001.stderr:2: v1:172.21.15.1:6790/0 mon.noname-c 2024-03-25T18:37:15.104 INFO:teuthology.misc.health.smithi001.stderr: 2024-03-25T18:37:15.104 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:37:15.101+0000 7fa70fbed640 1 librados: starting msgr at 2024-03-25T18:37:15.104 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:37:15.101+0000 7fa70fbed640 1 librados: starting objecter 2024-03-25T18:37:15.104 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:37:15.101+0000 7fa70fbed640 1 Processor -- start 2024-03-25T18:37:15.104 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:37:15.101+0000 7fa70fbed640 1 -- start start 2024-03-25T18:37:15.104 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:37:15.101+0000 7fa70fbed640 1 librados: setting wanted keys 2024-03-25T18:37:15.104 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:37:15.101+0000 7fa70fbed640 1 librados: calling monclient init 2024-03-25T18:37:15.104 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:37:15.101+0000 7fa70fbed640 10 monclient: init 2024-03-25T18:37:15.105 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:37:15.101+0000 7fa70fbed640 10 monclient: _reopen_session rank -1 2024-03-25T18:37:15.105 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:37:15.101+0000 7fa70fbed640 10 monclient: _add_conns ranks=[1,0,2] 2024-03-25T18:37:15.105 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:37:15.101+0000 7fa70fbed640 10 monclient(hunting): picked mon.noname-b con 0x7fa7080b4670 addr v1:172.21.15.115:6789/0 2024-03-25T18:37:15.105 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:37:15.101+0000 7fa70fbed640 10 monclient(hunting): picked mon.noname-a con 0x7fa7080b4ff0 addr v1:172.21.15.1:6789/0 2024-03-25T18:37:15.105 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:37:15.102+0000 7fa70fbed640 10 monclient(hunting): picked mon.noname-c con 0x7fa70813c480 addr v1:172.21.15.1:6790/0 2024-03-25T18:37:15.105 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:37:15.102+0000 7fa70fbed640 1 -- --> v1:172.21.15.1:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa708144870 con 0x7fa7080b4ff0 2024-03-25T18:37:15.105 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:37:15.102+0000 7fa70fbed640 1 -- --> v1:172.21.15.115:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa708144ae0 con 0x7fa7080b4670 2024-03-25T18:37:15.105 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:37:15.102+0000 7fa70fbed640 1 -- --> v1:172.21.15.1:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa70813ec30 con 0x7fa70813c480 2024-03-25T18:37:15.106 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:37:15.102+0000 7fa70fbed640 10 monclient(hunting): _renew_subs 2024-03-25T18:37:15.106 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:37:15.102+0000 7fa70fbed640 10 monclient(hunting): authenticate will time out at 8115.572702s 2024-03-25T18:37:15.109 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:37:15.108+0000 7fa70d989640 1 --1- >> v1:172.21.15.115:6789/0 conn(0x7fa7080b4670 0x7fa70813cbf0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.115:6789/0 says I am v1:172.21.15.1:38196/0 (socket says 172.21.15.1:38196) 2024-03-25T18:37:15.109 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:37:15.108+0000 7fa70d989640 1 -- 172.21.15.1:0/2516180413 learned_addr learned my addr 172.21.15.1:0/2516180413 (peer_addr_for_me v1:172.21.15.1:0/0) 2024-03-25T18:37:15.111 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:37:15.109+0000 7fa6fe7fc640 10 client.?.objecter ms_handle_connect 0x7fa7080b4670 2024-03-25T18:37:15.111 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:37:15.110+0000 7fa6fe7fc640 10 client.?.objecter resend_mon_ops 2024-03-25T18:37:15.118 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:37:15.117+0000 7fa6fe7fc640 1 -- 172.21.15.1:0/2516180413 <== mon.1 v1:172.21.15.115:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2495572054 0 0) 0x7fa708144ae0 con 0x7fa7080b4670 2024-03-25T18:37:15.118 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:37:15.117+0000 7fa6fe7fc640 10 monclient(hunting): _negotiate 2024-03-25T18:37:15.119 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:37:15.117+0000 7fa6fe7fc640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:37:15.119 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:37:15.117+0000 7fa6fe7fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:37:15.119 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:37:15.117+0000 7fa6fe7fc640 10 monclient(hunting): my global_id is 4687 2024-03-25T18:37:15.119 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:37:15.117+0000 7fa6fe7fc640 1 -- 172.21.15.1:0/2516180413 --> v1:172.21.15.115:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fa6ec004eb0 con 0x7fa7080b4670 2024-03-25T18:37:15.119 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:37:15.118+0000 7fa6fe7fc640 10 client.?.objecter ms_handle_connect 0x7fa7080b4ff0 2024-03-25T18:37:15.119 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:37:15.118+0000 7fa6fe7fc640 10 client.?.objecter resend_mon_ops 2024-03-25T18:37:15.126 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:37:15.124+0000 7fa6fe7fc640 1 -- 172.21.15.1:0/2516180413 <== mon.1 v1:172.21.15.115:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1959712584 0 0) 0x7fa6ec004eb0 con 0x7fa7080b4670 2024-03-25T18:37:15.126 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:37:15.124+0000 7fa6fe7fc640 1 -- 172.21.15.1:0/2516180413 --> v1:172.21.15.115:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fa708144ae0 con 0x7fa7080b4670 2024-03-25T18:37:15.126 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:37:15.124+0000 7fa6fe7fc640 1 -- 172.21.15.1:0/2516180413 <== mon.1 v1:172.21.15.115:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1586965025 0 0) 0x7fa704001950 con 0x7fa7080b4670 2024-03-25T18:37:15.126 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:37:15.124+0000 7fa6fe7fc640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:37:15.126 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:37:15.124+0000 7fa6fe7fc640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-03-25T18:37:15.126 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:37:15.124+0000 7fa6fe7fc640 10 monclient(hunting): dump: 2024-03-25T18:37:15.126 INFO:teuthology.misc.health.smithi001.stderr:epoch 1 2024-03-25T18:37:15.127 INFO:teuthology.misc.health.smithi001.stderr:fsid c8ea72cf-9e99-4383-9b52-6c23d365fdfc 2024-03-25T18:37:15.127 INFO:teuthology.misc.health.smithi001.stderr:last_changed 2024-03-25T18:25:50.510519+0000 2024-03-25T18:37:15.127 INFO:teuthology.misc.health.smithi001.stderr:created 2024-03-25T18:25:50.510519+0000 2024-03-25T18:37:15.127 INFO:teuthology.misc.health.smithi001.stderr:min_mon_release 19 (squid) 2024-03-25T18:37:15.127 INFO:teuthology.misc.health.smithi001.stderr:election_strategy: 3 2024-03-25T18:37:15.127 INFO:teuthology.misc.health.smithi001.stderr:0: v1:172.21.15.1:6789/0 mon.a 2024-03-25T18:37:15.127 INFO:teuthology.misc.health.smithi001.stderr:1: v1:172.21.15.115:6789/0 mon.b 2024-03-25T18:37:15.127 INFO:teuthology.misc.health.smithi001.stderr:2: v1:172.21.15.1:6790/0 mon.c 2024-03-25T18:37:15.127 INFO:teuthology.misc.health.smithi001.stderr: 2024-03-25T18:37:15.127 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:37:15.125+0000 7fa6fe7fc640 10 monclient(hunting): _finish_auth 0 2024-03-25T18:37:15.127 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:37:15.126+0000 7fa6fe7fc640 10 client.?.objecter ms_handle_connect 0x7fa70813c480 2024-03-25T18:37:15.128 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:37:15.126+0000 7fa6fe7fc640 10 client.?.objecter resend_mon_ops 2024-03-25T18:37:15.134 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:37:15.132+0000 7fa6fe7fc640 1 -- 172.21.15.1:0/2516180413 <== mon.1 v1:172.21.15.115:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 1272432468 0 0) 0x7fa708144ae0 con 0x7fa7080b4670 2024-03-25T18:37:15.134 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:37:15.132+0000 7fa6fe7fc640 1 -- 172.21.15.1:0/2516180413 >> v1:172.21.15.1:6790/0 conn(0x7fa70813c480 legacy=0x7fa7081e2380 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:37:15.134 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:37:15.132+0000 7fa6fe7fc640 1 -- 172.21.15.1:0/2516180413 >> v1:172.21.15.1:6789/0 conn(0x7fa7080b4ff0 legacy=0x7fa70813d300 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:37:15.134 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:37:15.132+0000 7fa6fe7fc640 10 monclient: _finish_hunting 0 2024-03-25T18:37:15.134 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:37:15.132+0000 7fa6fe7fc640 1 monclient: found mon.b 2024-03-25T18:37:15.134 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:37:15.132+0000 7fa6fe7fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:37:15.134 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:37:15.132+0000 7fa6fe7fc640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.115:6789/0 2024-03-25T18:37:15.134 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:37:15.132+0000 7fa6fe7fc640 1 -- 172.21.15.1:0/2516180413 --> v1:172.21.15.115:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa708143680 con 0x7fa7080b4670 2024-03-25T18:37:15.134 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:37:15.132+0000 7fa6fe7fc640 10 monclient: _finish_auth 0 2024-03-25T18:37:15.134 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:37:15.132+0000 7fa6fe7fc640 10 monclient: _check_auth_tickets 2024-03-25T18:37:15.135 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:37:15.132+0000 7fa6fe7fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:37:15.135 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:37:15.132+0000 7fa70fbed640 5 monclient: authenticate success, global_id 4687 2024-03-25T18:37:15.135 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:37:15.132+0000 7fa70fbed640 10 monclient: _renew_subs 2024-03-25T18:37:15.135 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:37:15.132+0000 7fa70fbed640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.115:6789/0 2024-03-25T18:37:15.135 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:37:15.132+0000 7fa70fbed640 1 -- 172.21.15.1:0/2516180413 --> v1:172.21.15.115:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa708143930 con 0x7fa7080b4670 2024-03-25T18:37:15.135 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:37:15.132+0000 7fa70fbed640 10 client.4687.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T18:37:15.135 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:37:15.132+0000 7fa70fbed640 10 monclient: _renew_subs 2024-03-25T18:37:15.135 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:37:15.132+0000 7fa70fbed640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.115:6789/0 2024-03-25T18:37:15.135 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:37:15.132+0000 7fa70fbed640 1 -- 172.21.15.1:0/2516180413 --> v1:172.21.15.115:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7fa7081460f0 con 0x7fa7080b4670 2024-03-25T18:37:15.135 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:37:15.132+0000 7fa70fbed640 1 librados: init done 2024-03-25T18:37:15.136 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:37:15.133+0000 7fa70fbed640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T18:37:15.136 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:37:15.133+0000 7fa70fbed640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:37:15.136 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:37:15.133+0000 7fa70fbed640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.115:6789/0 2024-03-25T18:37:15.136 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:37:15.133+0000 7fa70fbed640 1 -- 172.21.15.1:0/2516180413 --> v1:172.21.15.115:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa708146440 con 0x7fa7080b4670 2024-03-25T18:37:15.409 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:37:15.408+0000 7fa6fe7fc640 1 -- 172.21.15.1:0/2516180413 <== mon.1 v1:172.21.15.115:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fa7040047e0 con 0x7fa7080b4670 2024-03-25T18:37:15.409 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:37:15.408+0000 7fa6fe7fc640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:37:15.410 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:37:15.408+0000 7fa6fe7fc640 1 -- 172.21.15.1:0/2516180413 <== mon.1 v1:172.21.15.115:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1586965025 0 0) 0x7fa7040049d0 con 0x7fa7080b4670 2024-03-25T18:37:15.410 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:37:15.408+0000 7fa6fe7fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:37:15.410 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:37:15.408+0000 7fa6fe7fc640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-25T18:37:15.410 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:37:15.408+0000 7fa6fe7fc640 10 monclient: dump: 2024-03-25T18:37:15.410 INFO:teuthology.misc.health.smithi001.stderr:epoch 1 2024-03-25T18:37:15.410 INFO:teuthology.misc.health.smithi001.stderr:fsid c8ea72cf-9e99-4383-9b52-6c23d365fdfc 2024-03-25T18:37:15.410 INFO:teuthology.misc.health.smithi001.stderr:last_changed 2024-03-25T18:25:50.510519+0000 2024-03-25T18:37:15.410 INFO:teuthology.misc.health.smithi001.stderr:created 2024-03-25T18:25:50.510519+0000 2024-03-25T18:37:15.410 INFO:teuthology.misc.health.smithi001.stderr:min_mon_release 19 (squid) 2024-03-25T18:37:15.410 INFO:teuthology.misc.health.smithi001.stderr:election_strategy: 3 2024-03-25T18:37:15.410 INFO:teuthology.misc.health.smithi001.stderr:0: v1:172.21.15.1:6789/0 mon.a 2024-03-25T18:37:15.410 INFO:teuthology.misc.health.smithi001.stderr:1: v1:172.21.15.115:6789/0 mon.b 2024-03-25T18:37:15.410 INFO:teuthology.misc.health.smithi001.stderr:2: v1:172.21.15.1:6790/0 mon.c 2024-03-25T18:37:15.411 INFO:teuthology.misc.health.smithi001.stderr: 2024-03-25T18:37:15.411 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:37:15.408+0000 7fa6fffff640 4 set_mon_vals no callback set 2024-03-25T18:37:15.411 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:37:15.409+0000 7fa6fe7fc640 1 -- 172.21.15.1:0/2516180413 <== mon.1 v1:172.21.15.115:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 21787975 0 0) 0x7fa704004bf0 con 0x7fa7080b4670 2024-03-25T18:37:15.411 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:37:15.410+0000 7fa70d188640 10 monclient: get_auth_request con 0x7fa6ec06dc70 auth_method 0 2024-03-25T18:37:15.411 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:37:15.410+0000 7fa6fe7fc640 10 client.4687.objecter ms_handle_connect 0x7fa6ec06dc70 2024-03-25T18:37:15.415 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:37:15.414+0000 7fa6fe7fc640 1 -- 172.21.15.1:0/2516180413 <== mon.1 v1:172.21.15.115:6789/0 8 ==== osd_map(11..11 src has 1..11) v4 ==== 1645+0+0 (unknown 2504898309 0 0) 0x7fa704054700 con 0x7fa7080b4670 2024-03-25T18:37:15.415 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:37:15.414+0000 7fa6fe7fc640 10 client.4687.objecter ms_dispatch 0x7fa708000b90 osd_map(11..11 src has 1..11) v4 2024-03-25T18:37:15.415 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:37:15.414+0000 7fa6fe7fc640 3 client.4687.objecter handle_osd_map got epochs [11,11] > 0 2024-03-25T18:37:15.415 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:37:15.414+0000 7fa6fe7fc640 3 client.4687.objecter handle_osd_map decoding full epoch 11 2024-03-25T18:37:15.416 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:37:15.414+0000 7fa6fe7fc640 20 client.4687.objecter dump_active .. 0 homeless 2024-03-25T18:37:15.416 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:37:15.414+0000 7fa6fe7fc640 1 -- 172.21.15.1:0/2516180413 <== mon.1 v1:172.21.15.115:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7fa704050210 con 0x7fa7080b4670 2024-03-25T18:37:15.416 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:37:15.414+0000 7fa6fe7fc640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:37:15.416 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:37:15.414+0000 7fa6fe7fc640 10 monclient: _finish_command 1 = system:0 2024-03-25T18:37:15.674 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:37:15.672+0000 7fa70fbed640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:37:15.674 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:37:15.672+0000 7fa70fbed640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:37:15.674 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:37:15.672+0000 7fa70fbed640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.115:6789/0 2024-03-25T18:37:15.674 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:37:15.672+0000 7fa70fbed640 1 -- 172.21.15.1:0/2516180413 --> v1:172.21.15.115:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fa70806dbd0 con 0x7fa7080b4670 2024-03-25T18:37:15.699 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:37:15.697+0000 7fa6fe7fc640 1 -- 172.21.15.1:0/2516180413 <== mon.1 v1:172.21.15.115:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v11) v1 ==== 74+0+7100 (unknown 238102980 0 2126609903) 0x7fa704052140 con 0x7fa7080b4670 2024-03-25T18:37:15.699 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:37:15.697+0000 7fa6fe7fc640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:37:15.699 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:37:15.697+0000 7fa6fe7fc640 10 monclient: _finish_command 2 = system:0 2024-03-25T18:37:15.699 INFO:teuthology.misc.health.smithi001.stdout: 2024-03-25T18:37:15.699 INFO:teuthology.misc.health.smithi001.stdout:{"epoch":11,"fsid":"c8ea72cf-9e99-4383-9b52-6c23d365fdfc","created":"2024-03-25T18:26:38.856638+0000","modified":"2024-03-25T18:36:43.033551+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T18:36:43.033551+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"b9f34922-3a84-49be-8426-807c1eceaefc","up":0,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["autoout","exists","new"]},{"osd":1,"uuid":"42bf306f-55e3-4634-9874-8b4b73ac6644","up":0,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["autoout","exists","new"]},{"osd":2,"uuid":"292cb23a-18a9-4207-a7cc-4d5727feaec0","up":0,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["autoout","exists","new"]},{"osd":3,"uuid":"45e0e9ab-898d-4673-9590-b4117bc5e84a","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"831a5a09-73ab-4a61-9872-088e76bdbc66","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"88cd903d-3b55-4840-9396-6e3cfe005282","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"89db7355-bc9d-46f4-8ab0-e8d9e744e026","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"cac47bee-0914-4f08-896f-348838882866","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":65536,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":65536,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":65536,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T18:37:15.701 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:37:15.699+0000 7fa70fbed640 10 librados: watch_flush enter 2024-03-25T18:37:15.701 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:37:15.700+0000 7fa70fbed640 10 librados: watch_flush exit 2024-03-25T18:37:15.701 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:37:15.700+0000 7fa70fbed640 20 client.4687.objecter shutdown clearing up homeless session... 2024-03-25T18:37:15.701 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:37:15.700+0000 7fa70fbed640 10 client.4687.objecter successfully canceled tick 2024-03-25T18:37:15.702 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:37:15.700+0000 7fa70fbed640 1 -- 172.21.15.1:0/2516180413 >> v1:172.21.15.115:6800/4237379176 conn(0x7fa6ec06dc70 legacy=0x7fa6ec070130 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:37:15.702 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:37:15.700+0000 7fa70fbed640 10 monclient: shutdown 2024-03-25T18:37:15.702 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:37:15.700+0000 7fa70fbed640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:37:15.703 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:37:15.700+0000 7fa70fbed640 1 -- 172.21.15.1:0/2516180413 >> v1:172.21.15.115:6789/0 conn(0x7fa7080b4670 legacy=0x7fa70813cbf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:37:15.703 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:37:15.700+0000 7fa70fbed640 1 -- 172.21.15.1:0/2516180413 shutdown_connections 2024-03-25T18:37:15.703 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:37:15.700+0000 7fa70fbed640 1 -- 172.21.15.1:0/2516180413 >> 172.21.15.1:0/2516180413 conn(0x7fa7080617a0 msgr2=0x7fa7081405c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:37:15.703 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:37:15.700+0000 7fa70fbed640 1 -- 172.21.15.1:0/2516180413 shutdown_connections 2024-03-25T18:37:15.703 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:37:15.700+0000 7fa70fbed640 1 -- 172.21.15.1:0/2516180413 wait complete. 2024-03-25T18:37:15.703 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:37:15.701+0000 7fa70fbed640 1 librados: shutdown 2024-03-25T18:37:15.714 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T18:37:22.520 DEBUG:teuthology.orchestra.run.smithi001:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T18:37:22.635 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:37:22.633+0000 7f484fc0f640 10 monclient: get_monmap_and_config 2024-03-25T18:37:22.635 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:37:22.633+0000 7f484fc0f640 10 monclient: build_initial_monmap 2024-03-25T18:37:22.635 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:37:22.633+0000 7f484fc0f640 10 monclient: monmap: 2024-03-25T18:37:22.635 INFO:teuthology.misc.health.smithi001.stderr:epoch 0 2024-03-25T18:37:22.635 INFO:teuthology.misc.health.smithi001.stderr:fsid c8ea72cf-9e99-4383-9b52-6c23d365fdfc 2024-03-25T18:37:22.635 INFO:teuthology.misc.health.smithi001.stderr:last_changed 2024-03-25T18:37:22.635325+0000 2024-03-25T18:37:22.635 INFO:teuthology.misc.health.smithi001.stderr:created 2024-03-25T18:37:22.635325+0000 2024-03-25T18:37:22.635 INFO:teuthology.misc.health.smithi001.stderr:min_mon_release 0 (unknown) 2024-03-25T18:37:22.635 INFO:teuthology.misc.health.smithi001.stderr:election_strategy: 3 2024-03-25T18:37:22.635 INFO:teuthology.misc.health.smithi001.stderr:0: v1:172.21.15.1:6789/0 mon.noname-a 2024-03-25T18:37:22.636 INFO:teuthology.misc.health.smithi001.stderr:1: v1:172.21.15.115:6789/0 mon.noname-c 2024-03-25T18:37:22.636 INFO:teuthology.misc.health.smithi001.stderr:2: v1:172.21.15.1:6790/0 mon.noname-b 2024-03-25T18:37:22.636 INFO:teuthology.misc.health.smithi001.stderr: 2024-03-25T18:37:22.637 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:37:22.634+0000 7f484fc0f640 1 Processor -- start 2024-03-25T18:37:22.637 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:37:22.634+0000 7f484fc0f640 1 -- start start 2024-03-25T18:37:22.637 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:37:22.634+0000 7f484fc0f640 10 monclient: init 2024-03-25T18:37:22.637 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:37:22.635+0000 7f484fc0f640 10 monclient: _reopen_session rank -1 2024-03-25T18:37:22.637 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:37:22.635+0000 7f484fc0f640 10 monclient: _add_conns ranks=[1,2,0] 2024-03-25T18:37:22.637 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:37:22.635+0000 7f484fc0f640 10 monclient(hunting): picked mon.noname-c con 0x7f48480b4e70 addr v1:172.21.15.115:6789/0 2024-03-25T18:37:22.637 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:37:22.635+0000 7f484fc0f640 10 monclient(hunting): picked mon.noname-b con 0x7f48480b44b0 addr v1:172.21.15.1:6790/0 2024-03-25T18:37:22.637 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:37:22.635+0000 7f484fc0f640 10 monclient(hunting): picked mon.noname-a con 0x7f48480b1b00 addr v1:172.21.15.1:6789/0 2024-03-25T18:37:22.637 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:37:22.635+0000 7f484fc0f640 1 -- --> v1:172.21.15.1:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f48480a96f0 con 0x7f48480b1b00 2024-03-25T18:37:22.637 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:37:22.635+0000 7f484fc0f640 1 -- --> v1:172.21.15.115:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f48480ab0e0 con 0x7f48480b4e70 2024-03-25T18:37:22.638 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:37:22.635+0000 7f484fc0f640 1 -- --> v1:172.21.15.1:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4848145f70 con 0x7f48480b44b0 2024-03-25T18:37:22.638 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:37:22.635+0000 7f484fc0f640 10 monclient(hunting): _renew_subs 2024-03-25T18:37:22.638 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:37:22.635+0000 7f484fc0f640 10 monclient(hunting): authenticate will time out at 8123.106399s 2024-03-25T18:37:22.650 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:37:22.648+0000 7f484d1aa640 1 --1- >> v1:172.21.15.1:6790/0 conn(0x7f48480b44b0 0x7f4848145860 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.1:6790/0 says I am v1:172.21.15.1:37176/0 (socket says 172.21.15.1:37176) 2024-03-25T18:37:22.650 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:37:22.648+0000 7f484d1aa640 1 -- 172.21.15.1:0/4221382803 learned_addr learned my addr 172.21.15.1:0/4221382803 (peer_addr_for_me v1:172.21.15.1:0/0) 2024-03-25T18:37:22.663 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:37:22.661+0000 7f484c9a9640 1 -- 172.21.15.1:0/4221382803 <== mon.2 v1:172.21.15.1:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 286317434 0 0) 0x7f4848145f70 con 0x7f48480b44b0 2024-03-25T18:37:22.663 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:37:22.661+0000 7f484c9a9640 10 monclient(hunting): _negotiate 2024-03-25T18:37:22.663 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:37:22.661+0000 7f484c9a9640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:37:22.663 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:37:22.661+0000 7f484c9a9640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:37:22.663 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:37:22.661+0000 7f484c9a9640 10 monclient(hunting): my global_id is 4682 2024-03-25T18:37:22.663 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:37:22.661+0000 7f484c9a9640 1 -- 172.21.15.1:0/4221382803 --> v1:172.21.15.1:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f4834003520 con 0x7f48480b44b0 2024-03-25T18:37:22.666 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:37:22.665+0000 7f484c9a9640 1 -- 172.21.15.1:0/4221382803 <== mon.1 v1:172.21.15.115:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2807447023 0 0) 0x7f48480ab0e0 con 0x7f48480b4e70 2024-03-25T18:37:22.666 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:37:22.665+0000 7f484c9a9640 10 monclient(hunting): _negotiate 2024-03-25T18:37:22.667 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:37:22.665+0000 7f484c9a9640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:37:22.667 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:37:22.665+0000 7f484c9a9640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:37:22.667 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:37:22.665+0000 7f484c9a9640 10 monclient(hunting): my global_id is 4690 2024-03-25T18:37:22.667 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:37:22.665+0000 7f484c9a9640 1 -- 172.21.15.1:0/4221382803 --> v1:172.21.15.115:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f4848145f70 con 0x7f48480b4e70 2024-03-25T18:37:22.679 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:37:22.677+0000 7f484c9a9640 1 -- 172.21.15.1:0/4221382803 <== mon.2 v1:172.21.15.1:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1527433747 0 0) 0x7f4834003520 con 0x7f48480b44b0 2024-03-25T18:37:22.679 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:37:22.677+0000 7f484c9a9640 1 -- 172.21.15.1:0/4221382803 >> v1:172.21.15.115:6789/0 conn(0x7f48480b4e70 legacy=0x7f484813f940 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:37:22.679 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:37:22.677+0000 7f484c9a9640 1 -- 172.21.15.1:0/4221382803 >> v1:172.21.15.1:6789/0 conn(0x7f48480b1b00 legacy=0x7f484814dc00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:37:22.679 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:37:22.677+0000 7f484c9a9640 10 monclient: _finish_hunting 0 2024-03-25T18:37:22.679 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:37:22.677+0000 7f484c9a9640 1 monclient: found mon.noname-b 2024-03-25T18:37:22.679 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:37:22.677+0000 7f484c9a9640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:37:22.680 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:37:22.677+0000 7f484c9a9640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.1:6790/0 2024-03-25T18:37:22.680 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:37:22.677+0000 7f484c9a9640 1 -- 172.21.15.1:0/4221382803 --> v1:172.21.15.1:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f48480a9b80 con 0x7f48480b44b0 2024-03-25T18:37:22.680 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:37:22.677+0000 7f484c9a9640 10 monclient: _finish_auth 0 2024-03-25T18:37:22.680 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:37:22.677+0000 7f484c9a9640 10 monclient: _check_auth_tickets 2024-03-25T18:37:22.680 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:37:22.677+0000 7f484c9a9640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:37:22.680 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:37:22.677+0000 7f484c9a9640 1 -- 172.21.15.1:0/4221382803 <== mon.2 v1:172.21.15.1:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1586965025 0 0) 0x7f4838001690 con 0x7f48480b44b0 2024-03-25T18:37:22.680 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:37:22.677+0000 7f484fc0f640 5 monclient: authenticate success, global_id 4682 2024-03-25T18:37:22.680 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:37:22.677+0000 7f484fc0f640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T18:37:22.680 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:37:22.677+0000 7f484c9a9640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:37:22.680 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:37:22.677+0000 7f484c9a9640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-03-25T18:37:22.680 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:37:22.677+0000 7f484c9a9640 10 monclient: dump: 2024-03-25T18:37:22.680 INFO:teuthology.misc.health.smithi001.stderr:epoch 1 2024-03-25T18:37:22.680 INFO:teuthology.misc.health.smithi001.stderr:fsid c8ea72cf-9e99-4383-9b52-6c23d365fdfc 2024-03-25T18:37:22.680 INFO:teuthology.misc.health.smithi001.stderr:last_changed 2024-03-25T18:25:50.510519+0000 2024-03-25T18:37:22.680 INFO:teuthology.misc.health.smithi001.stderr:created 2024-03-25T18:25:50.510519+0000 2024-03-25T18:37:22.681 INFO:teuthology.misc.health.smithi001.stderr:min_mon_release 19 (squid) 2024-03-25T18:37:22.681 INFO:teuthology.misc.health.smithi001.stderr:election_strategy: 3 2024-03-25T18:37:22.681 INFO:teuthology.misc.health.smithi001.stderr:0: v1:172.21.15.1:6789/0 mon.a 2024-03-25T18:37:22.681 INFO:teuthology.misc.health.smithi001.stderr:1: v1:172.21.15.115:6789/0 mon.b 2024-03-25T18:37:22.681 INFO:teuthology.misc.health.smithi001.stderr:2: v1:172.21.15.1:6790/0 mon.c 2024-03-25T18:37:22.681 INFO:teuthology.misc.health.smithi001.stderr: 2024-03-25T18:37:22.681 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:37:22.678+0000 7f484fc0f640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T18:37:22.686 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:37:22.685+0000 7f484c9a9640 1 -- 172.21.15.1:0/4221382803 <== mon.2 v1:172.21.15.1:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f48380017f0 con 0x7f48480b44b0 2024-03-25T18:37:22.686 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:37:22.685+0000 7f484c9a9640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:37:22.686 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:37:22.685+0000 7f484c9a9640 1 -- 172.21.15.1:0/4221382803 <== mon.2 v1:172.21.15.1:6790/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1586965025 0 0) 0x7f4838001c00 con 0x7f48480b44b0 2024-03-25T18:37:22.686 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:37:22.685+0000 7f484fc0f640 10 monclient: get_monmap_and_config success 2024-03-25T18:37:22.686 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:37:22.685+0000 7f484fc0f640 4 set_mon_vals no callback set 2024-03-25T18:37:22.686 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:37:22.685+0000 7f484c9a9640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:37:22.687 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:37:22.685+0000 7f484c9a9640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-25T18:37:22.687 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:37:22.685+0000 7f484c9a9640 10 monclient: dump: 2024-03-25T18:37:22.687 INFO:teuthology.misc.health.smithi001.stderr:epoch 1 2024-03-25T18:37:22.687 INFO:teuthology.misc.health.smithi001.stderr:fsid c8ea72cf-9e99-4383-9b52-6c23d365fdfc 2024-03-25T18:37:22.687 INFO:teuthology.misc.health.smithi001.stderr:last_changed 2024-03-25T18:25:50.510519+0000 2024-03-25T18:37:22.687 INFO:teuthology.misc.health.smithi001.stderr:created 2024-03-25T18:25:50.510519+0000 2024-03-25T18:37:22.687 INFO:teuthology.misc.health.smithi001.stderr:min_mon_release 19 (squid) 2024-03-25T18:37:22.687 INFO:teuthology.misc.health.smithi001.stderr:election_strategy: 3 2024-03-25T18:37:22.687 INFO:teuthology.misc.health.smithi001.stderr:0: v1:172.21.15.1:6789/0 mon.a 2024-03-25T18:37:22.687 INFO:teuthology.misc.health.smithi001.stderr:1: v1:172.21.15.115:6789/0 mon.b 2024-03-25T18:37:22.687 INFO:teuthology.misc.health.smithi001.stderr:2: v1:172.21.15.1:6790/0 mon.c 2024-03-25T18:37:22.687 INFO:teuthology.misc.health.smithi001.stderr: 2024-03-25T18:37:22.688 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:37:22.686+0000 7f484fc0f640 10 monclient: shutdown 2024-03-25T18:37:22.688 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:37:22.686+0000 7f484fc0f640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:37:22.688 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:37:22.686+0000 7f484fc0f640 1 -- 172.21.15.1:0/4221382803 >> v1:172.21.15.1:6790/0 conn(0x7f48480b44b0 legacy=0x7f4848145860 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:37:22.688 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:37:22.686+0000 7f484fc0f640 1 -- 172.21.15.1:0/4221382803 shutdown_connections 2024-03-25T18:37:22.688 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:37:22.686+0000 7f484fc0f640 1 -- 172.21.15.1:0/4221382803 >> 172.21.15.1:0/4221382803 conn(0x7f48480617a0 msgr2=0x7f4848061bb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:37:22.689 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:37:22.686+0000 7f484fc0f640 1 -- 172.21.15.1:0/4221382803 shutdown_connections 2024-03-25T18:37:22.689 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:37:22.686+0000 7f484fc0f640 1 -- 172.21.15.1:0/4221382803 wait complete. 2024-03-25T18:37:22.689 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:37:22.687+0000 7f484fc0f640 10 monclient: build_initial_monmap 2024-03-25T18:37:22.689 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:37:22.687+0000 7f484fc0f640 10 monclient: monmap: 2024-03-25T18:37:22.689 INFO:teuthology.misc.health.smithi001.stderr:epoch 0 2024-03-25T18:37:22.689 INFO:teuthology.misc.health.smithi001.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T18:37:22.689 INFO:teuthology.misc.health.smithi001.stderr:last_changed 0.000000 2024-03-25T18:37:22.689 INFO:teuthology.misc.health.smithi001.stderr:created 0.000000 2024-03-25T18:37:22.689 INFO:teuthology.misc.health.smithi001.stderr:min_mon_release 0 (unknown) 2024-03-25T18:37:22.689 INFO:teuthology.misc.health.smithi001.stderr:election_strategy: 1 2024-03-25T18:37:22.689 INFO:teuthology.misc.health.smithi001.stderr:0: v1:172.21.15.1:6789/0 mon.noname-a 2024-03-25T18:37:22.689 INFO:teuthology.misc.health.smithi001.stderr:1: v1:172.21.15.115:6789/0 mon.noname-b 2024-03-25T18:37:22.689 INFO:teuthology.misc.health.smithi001.stderr:2: v1:172.21.15.1:6790/0 mon.noname-c 2024-03-25T18:37:22.689 INFO:teuthology.misc.health.smithi001.stderr: 2024-03-25T18:37:22.691 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:37:22.687+0000 7f484fc0f640 1 librados: starting msgr at 2024-03-25T18:37:22.691 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:37:22.687+0000 7f484fc0f640 1 librados: starting objecter 2024-03-25T18:37:22.691 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:37:22.687+0000 7f484fc0f640 1 Processor -- start 2024-03-25T18:37:22.691 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:37:22.687+0000 7f484fc0f640 1 -- start start 2024-03-25T18:37:22.691 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:37:22.687+0000 7f484fc0f640 1 librados: setting wanted keys 2024-03-25T18:37:22.691 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:37:22.687+0000 7f484fc0f640 1 librados: calling monclient init 2024-03-25T18:37:22.691 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:37:22.687+0000 7f484fc0f640 10 monclient: init 2024-03-25T18:37:22.691 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:37:22.688+0000 7f484fc0f640 10 monclient: _reopen_session rank -1 2024-03-25T18:37:22.691 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:37:22.688+0000 7f484fc0f640 10 monclient: _add_conns ranks=[1,0,2] 2024-03-25T18:37:22.691 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:37:22.688+0000 7f484fc0f640 10 monclient(hunting): picked mon.noname-b con 0x7f48480b1b00 addr v1:172.21.15.115:6789/0 2024-03-25T18:37:22.692 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:37:22.688+0000 7f484fc0f640 10 monclient(hunting): picked mon.noname-a con 0x7f48480b44b0 addr v1:172.21.15.1:6789/0 2024-03-25T18:37:22.692 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:37:22.688+0000 7f484fc0f640 10 monclient(hunting): picked mon.noname-c con 0x7f48480b4e70 addr v1:172.21.15.1:6790/0 2024-03-25T18:37:22.692 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:37:22.688+0000 7f484fc0f640 1 -- --> v1:172.21.15.1:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f48481369f0 con 0x7f48480b44b0 2024-03-25T18:37:22.692 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:37:22.688+0000 7f484fc0f640 1 -- --> v1:172.21.15.115:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4848137990 con 0x7f48480b1b00 2024-03-25T18:37:22.692 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:37:22.688+0000 7f484fc0f640 1 -- --> v1:172.21.15.1:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4848137c70 con 0x7f48480b4e70 2024-03-25T18:37:22.692 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:37:22.688+0000 7f484fc0f640 10 monclient(hunting): _renew_subs 2024-03-25T18:37:22.692 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:37:22.688+0000 7f484fc0f640 10 monclient(hunting): authenticate will time out at 8123.159448s 2024-03-25T18:37:22.707 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:37:22.706+0000 7f484d1aa640 1 --1- >> v1:172.21.15.1:6789/0 conn(0x7f48480b44b0 0x7f4848133e80 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.1:6789/0 says I am v1:172.21.15.1:36936/0 (socket says 172.21.15.1:36936) 2024-03-25T18:37:22.707 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:37:22.706+0000 7f484d1aa640 1 -- 172.21.15.1:0/3559945778 learned_addr learned my addr 172.21.15.1:0/3559945778 (peer_addr_for_me v1:172.21.15.1:0/0) 2024-03-25T18:37:22.711 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:37:22.709+0000 7f483e7fc640 10 client.?.objecter ms_handle_connect 0x7f48480b44b0 2024-03-25T18:37:22.711 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:37:22.709+0000 7f483e7fc640 10 client.?.objecter resend_mon_ops 2024-03-25T18:37:22.725 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:37:22.724+0000 7f483e7fc640 10 client.?.objecter ms_handle_connect 0x7f48480b1b00 2024-03-25T18:37:22.725 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:37:22.724+0000 7f483e7fc640 10 client.?.objecter resend_mon_ops 2024-03-25T18:37:22.732 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:37:22.730+0000 7f483e7fc640 10 client.?.objecter ms_handle_connect 0x7f48480b4e70 2024-03-25T18:37:22.732 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:37:22.730+0000 7f483e7fc640 10 client.?.objecter resend_mon_ops 2024-03-25T18:37:22.732 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:37:22.731+0000 7f483e7fc640 1 -- 172.21.15.1:0/3559945778 <== mon.0 v1:172.21.15.1:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1547951161 0 0) 0x7f48481369f0 con 0x7f48480b44b0 2024-03-25T18:37:22.732 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:37:22.731+0000 7f483e7fc640 10 monclient(hunting): _negotiate 2024-03-25T18:37:22.732 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:37:22.731+0000 7f483e7fc640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:37:22.732 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:37:22.731+0000 7f483e7fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:37:22.733 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:37:22.731+0000 7f483e7fc640 10 monclient(hunting): my global_id is 4677 2024-03-25T18:37:22.733 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:37:22.731+0000 7f483e7fc640 1 -- 172.21.15.1:0/3559945778 --> v1:172.21.15.1:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f482c004eb0 con 0x7f48480b44b0 2024-03-25T18:37:22.741 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:37:22.739+0000 7f483e7fc640 1 -- 172.21.15.1:0/3559945778 <== mon.1 v1:172.21.15.115:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1416347312 0 0) 0x7f4848137990 con 0x7f48480b1b00 2024-03-25T18:37:22.741 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:37:22.739+0000 7f483e7fc640 10 monclient(hunting): _negotiate 2024-03-25T18:37:22.741 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:37:22.739+0000 7f483e7fc640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:37:22.741 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:37:22.739+0000 7f483e7fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:37:22.741 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:37:22.739+0000 7f483e7fc640 10 monclient(hunting): my global_id is 4693 2024-03-25T18:37:22.741 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:37:22.739+0000 7f483e7fc640 1 -- 172.21.15.1:0/3559945778 --> v1:172.21.15.115:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f48481369f0 con 0x7f48480b1b00 2024-03-25T18:37:22.743 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:37:22.741+0000 7f483e7fc640 1 -- 172.21.15.1:0/3559945778 <== mon.2 v1:172.21.15.1:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1116024228 0 0) 0x7f4848137c70 con 0x7f48480b4e70 2024-03-25T18:37:22.743 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:37:22.741+0000 7f483e7fc640 10 monclient(hunting): _negotiate 2024-03-25T18:37:22.743 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:37:22.741+0000 7f483e7fc640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:37:22.743 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:37:22.741+0000 7f483e7fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:37:22.743 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:37:22.741+0000 7f483e7fc640 10 monclient(hunting): my global_id is 4685 2024-03-25T18:37:22.743 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:37:22.741+0000 7f483e7fc640 1 -- 172.21.15.1:0/3559945778 --> v1:172.21.15.1:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f4848137990 con 0x7f48480b4e70 2024-03-25T18:37:22.771 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:37:22.769+0000 7f483e7fc640 1 -- 172.21.15.1:0/3559945778 <== mon.0 v1:172.21.15.1:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 468770297 0 0) 0x7f482c004eb0 con 0x7f48480b44b0 2024-03-25T18:37:22.771 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:37:22.769+0000 7f483e7fc640 1 -- 172.21.15.1:0/3559945778 --> v1:172.21.15.1:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f4848137c70 con 0x7f48480b44b0 2024-03-25T18:37:22.771 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:37:22.769+0000 7f483e7fc640 1 -- 172.21.15.1:0/3559945778 <== mon.0 v1:172.21.15.1:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1586965025 0 0) 0x7f4838001b40 con 0x7f48480b44b0 2024-03-25T18:37:22.771 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:37:22.769+0000 7f483e7fc640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:37:22.771 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:37:22.769+0000 7f483e7fc640 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2024-03-25T18:37:22.771 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:37:22.769+0000 7f483e7fc640 10 monclient(hunting): dump: 2024-03-25T18:37:22.771 INFO:teuthology.misc.health.smithi001.stderr:epoch 1 2024-03-25T18:37:22.771 INFO:teuthology.misc.health.smithi001.stderr:fsid c8ea72cf-9e99-4383-9b52-6c23d365fdfc 2024-03-25T18:37:22.771 INFO:teuthology.misc.health.smithi001.stderr:last_changed 2024-03-25T18:25:50.510519+0000 2024-03-25T18:37:22.772 INFO:teuthology.misc.health.smithi001.stderr:created 2024-03-25T18:25:50.510519+0000 2024-03-25T18:37:22.772 INFO:teuthology.misc.health.smithi001.stderr:min_mon_release 19 (squid) 2024-03-25T18:37:22.772 INFO:teuthology.misc.health.smithi001.stderr:election_strategy: 3 2024-03-25T18:37:22.772 INFO:teuthology.misc.health.smithi001.stderr:0: v1:172.21.15.1:6789/0 mon.a 2024-03-25T18:37:22.772 INFO:teuthology.misc.health.smithi001.stderr:1: v1:172.21.15.115:6789/0 mon.b 2024-03-25T18:37:22.772 INFO:teuthology.misc.health.smithi001.stderr:2: v1:172.21.15.1:6790/0 mon.c 2024-03-25T18:37:22.772 INFO:teuthology.misc.health.smithi001.stderr: 2024-03-25T18:37:22.772 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:37:22.769+0000 7f483e7fc640 10 monclient(hunting): _finish_auth 0 2024-03-25T18:37:22.777 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:37:22.775+0000 7f483e7fc640 1 -- 172.21.15.1:0/3559945778 <== mon.1 v1:172.21.15.115:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1033158787 0 0) 0x7f48481369f0 con 0x7f48480b1b00 2024-03-25T18:37:22.777 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:37:22.775+0000 7f483e7fc640 1 -- 172.21.15.1:0/3559945778 --> v1:172.21.15.115:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f482c004eb0 con 0x7f48480b1b00 2024-03-25T18:37:22.777 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:37:22.775+0000 7f483e7fc640 1 -- 172.21.15.1:0/3559945778 <== mon.1 v1:172.21.15.115:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1586965025 0 0) 0x7f4844001850 con 0x7f48480b1b00 2024-03-25T18:37:22.777 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:37:22.775+0000 7f483e7fc640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:37:22.777 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:37:22.775+0000 7f483e7fc640 10 monclient(hunting): got monmap 1 from mon.b (according to old e1) 2024-03-25T18:37:22.777 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:37:22.775+0000 7f483e7fc640 10 monclient(hunting): dump: 2024-03-25T18:37:22.777 INFO:teuthology.misc.health.smithi001.stderr:epoch 1 2024-03-25T18:37:22.777 INFO:teuthology.misc.health.smithi001.stderr:fsid c8ea72cf-9e99-4383-9b52-6c23d365fdfc 2024-03-25T18:37:22.777 INFO:teuthology.misc.health.smithi001.stderr:last_changed 2024-03-25T18:25:50.510519+0000 2024-03-25T18:37:22.778 INFO:teuthology.misc.health.smithi001.stderr:created 2024-03-25T18:25:50.510519+0000 2024-03-25T18:37:22.778 INFO:teuthology.misc.health.smithi001.stderr:min_mon_release 19 (squid) 2024-03-25T18:37:22.778 INFO:teuthology.misc.health.smithi001.stderr:election_strategy: 3 2024-03-25T18:37:22.778 INFO:teuthology.misc.health.smithi001.stderr:0: v1:172.21.15.1:6789/0 mon.a 2024-03-25T18:37:22.778 INFO:teuthology.misc.health.smithi001.stderr:1: v1:172.21.15.115:6789/0 mon.b 2024-03-25T18:37:22.778 INFO:teuthology.misc.health.smithi001.stderr:2: v1:172.21.15.1:6790/0 mon.c 2024-03-25T18:37:22.778 INFO:teuthology.misc.health.smithi001.stderr: 2024-03-25T18:37:22.785 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:37:22.784+0000 7f483e7fc640 1 -- 172.21.15.1:0/3559945778 <== mon.2 v1:172.21.15.1:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 184073823 0 0) 0x7f4848137990 con 0x7f48480b4e70 2024-03-25T18:37:22.785 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:37:22.784+0000 7f483e7fc640 1 -- 172.21.15.1:0/3559945778 --> v1:172.21.15.1:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f48481369f0 con 0x7f48480b4e70 2024-03-25T18:37:22.785 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:37:22.784+0000 7f483e7fc640 1 -- 172.21.15.1:0/3559945778 <== mon.2 v1:172.21.15.1:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1586965025 0 0) 0x7f4840001b70 con 0x7f48480b4e70 2024-03-25T18:37:22.785 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:37:22.784+0000 7f483e7fc640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:37:22.785 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:37:22.784+0000 7f483e7fc640 10 monclient(hunting): got monmap 1 from mon.c (according to old e1) 2024-03-25T18:37:22.786 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:37:22.784+0000 7f483e7fc640 10 monclient(hunting): dump: 2024-03-25T18:37:22.786 INFO:teuthology.misc.health.smithi001.stderr:epoch 1 2024-03-25T18:37:22.786 INFO:teuthology.misc.health.smithi001.stderr:fsid c8ea72cf-9e99-4383-9b52-6c23d365fdfc 2024-03-25T18:37:22.786 INFO:teuthology.misc.health.smithi001.stderr:last_changed 2024-03-25T18:25:50.510519+0000 2024-03-25T18:37:22.786 INFO:teuthology.misc.health.smithi001.stderr:created 2024-03-25T18:25:50.510519+0000 2024-03-25T18:37:22.786 INFO:teuthology.misc.health.smithi001.stderr:min_mon_release 19 (squid) 2024-03-25T18:37:22.786 INFO:teuthology.misc.health.smithi001.stderr:election_strategy: 3 2024-03-25T18:37:22.786 INFO:teuthology.misc.health.smithi001.stderr:0: v1:172.21.15.1:6789/0 mon.a 2024-03-25T18:37:22.786 INFO:teuthology.misc.health.smithi001.stderr:1: v1:172.21.15.115:6789/0 mon.b 2024-03-25T18:37:22.786 INFO:teuthology.misc.health.smithi001.stderr:2: v1:172.21.15.1:6790/0 mon.c 2024-03-25T18:37:22.786 INFO:teuthology.misc.health.smithi001.stderr: 2024-03-25T18:37:22.790 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:37:22.788+0000 7f483e7fc640 1 -- 172.21.15.1:0/3559945778 <== mon.1 v1:172.21.15.115:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 899276950 0 0) 0x7f482c004eb0 con 0x7f48480b1b00 2024-03-25T18:37:22.790 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:37:22.788+0000 7f483e7fc640 1 -- 172.21.15.1:0/3559945778 >> v1:172.21.15.1:6790/0 conn(0x7f48480b4e70 legacy=0x7f4848138090 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:37:22.790 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:37:22.788+0000 7f483e7fc640 1 -- 172.21.15.1:0/3559945778 >> v1:172.21.15.1:6789/0 conn(0x7f48480b44b0 legacy=0x7f4848133e80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:37:22.790 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:37:22.788+0000 7f483e7fc640 10 monclient: _finish_hunting 0 2024-03-25T18:37:22.790 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:37:22.788+0000 7f483e7fc640 1 monclient: found mon.b 2024-03-25T18:37:22.790 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:37:22.788+0000 7f483e7fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:37:22.790 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:37:22.788+0000 7f483e7fc640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.115:6789/0 2024-03-25T18:37:22.790 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:37:22.788+0000 7f483e7fc640 1 -- 172.21.15.1:0/3559945778 --> v1:172.21.15.115:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f48481364d0 con 0x7f48480b1b00 2024-03-25T18:37:22.790 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:37:22.788+0000 7f483e7fc640 10 monclient: _finish_auth 0 2024-03-25T18:37:22.790 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:37:22.788+0000 7f483e7fc640 10 monclient: _check_auth_tickets 2024-03-25T18:37:22.790 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:37:22.788+0000 7f483e7fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:37:22.790 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:37:22.788+0000 7f484fc0f640 5 monclient: authenticate success, global_id 4693 2024-03-25T18:37:22.790 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:37:22.788+0000 7f484fc0f640 10 monclient: _renew_subs 2024-03-25T18:37:22.791 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:37:22.788+0000 7f484fc0f640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.115:6789/0 2024-03-25T18:37:22.791 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:37:22.788+0000 7f484fc0f640 1 -- 172.21.15.1:0/3559945778 --> v1:172.21.15.115:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4848136040 con 0x7f48480b1b00 2024-03-25T18:37:22.791 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:37:22.788+0000 7f484fc0f640 10 client.4693.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T18:37:22.791 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:37:22.788+0000 7f484fc0f640 10 monclient: _renew_subs 2024-03-25T18:37:22.791 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:37:22.788+0000 7f484fc0f640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.115:6789/0 2024-03-25T18:37:22.791 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:37:22.788+0000 7f484fc0f640 1 -- 172.21.15.1:0/3559945778 --> v1:172.21.15.115:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f4848134770 con 0x7f48480b1b00 2024-03-25T18:37:22.791 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:37:22.788+0000 7f484fc0f640 1 librados: init done 2024-03-25T18:37:22.791 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:37:22.789+0000 7f484fc0f640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T18:37:22.791 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:37:22.789+0000 7f484fc0f640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:37:22.791 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:37:22.789+0000 7f484fc0f640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.115:6789/0 2024-03-25T18:37:22.792 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:37:22.789+0000 7f484fc0f640 1 -- 172.21.15.1:0/3559945778 --> v1:172.21.15.115:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f484804f730 con 0x7f48480b1b00 2024-03-25T18:37:23.095 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:37:23.093+0000 7f483e7fc640 1 -- 172.21.15.1:0/3559945778 <== mon.1 v1:172.21.15.115:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f4844001d00 con 0x7f48480b1b00 2024-03-25T18:37:23.095 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:37:23.093+0000 7f483e7fc640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:37:23.095 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:37:23.093+0000 7f483e7fc640 1 -- 172.21.15.1:0/3559945778 <== mon.1 v1:172.21.15.115:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1586965025 0 0) 0x7f4844003790 con 0x7f48480b1b00 2024-03-25T18:37:23.095 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:37:23.093+0000 7f483e7fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:37:23.095 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:37:23.093+0000 7f483e7fc640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-25T18:37:23.095 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:37:23.093+0000 7f483e7fc640 10 monclient: dump: 2024-03-25T18:37:23.095 INFO:teuthology.misc.health.smithi001.stderr:epoch 1 2024-03-25T18:37:23.095 INFO:teuthology.misc.health.smithi001.stderr:fsid c8ea72cf-9e99-4383-9b52-6c23d365fdfc 2024-03-25T18:37:23.095 INFO:teuthology.misc.health.smithi001.stderr:last_changed 2024-03-25T18:25:50.510519+0000 2024-03-25T18:37:23.095 INFO:teuthology.misc.health.smithi001.stderr:created 2024-03-25T18:25:50.510519+0000 2024-03-25T18:37:23.095 INFO:teuthology.misc.health.smithi001.stderr:min_mon_release 19 (squid) 2024-03-25T18:37:23.095 INFO:teuthology.misc.health.smithi001.stderr:election_strategy: 3 2024-03-25T18:37:23.096 INFO:teuthology.misc.health.smithi001.stderr:0: v1:172.21.15.1:6789/0 mon.a 2024-03-25T18:37:23.096 INFO:teuthology.misc.health.smithi001.stderr:1: v1:172.21.15.115:6789/0 mon.b 2024-03-25T18:37:23.096 INFO:teuthology.misc.health.smithi001.stderr:2: v1:172.21.15.1:6790/0 mon.c 2024-03-25T18:37:23.096 INFO:teuthology.misc.health.smithi001.stderr: 2024-03-25T18:37:23.096 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:37:23.093+0000 7f483ffff640 4 set_mon_vals no callback set 2024-03-25T18:37:23.096 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:37:23.094+0000 7f483e7fc640 1 -- 172.21.15.1:0/3559945778 <== mon.1 v1:172.21.15.115:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 21787975 0 0) 0x7f4844019820 con 0x7f48480b1b00 2024-03-25T18:37:23.096 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:37:23.094+0000 7f484d1aa640 10 monclient: get_auth_request con 0x7f482c06e160 auth_method 0 2024-03-25T18:37:23.096 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:37:23.095+0000 7f483e7fc640 10 client.4693.objecter ms_handle_connect 0x7f482c06e160 2024-03-25T18:37:23.100 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:37:23.099+0000 7f483e7fc640 1 -- 172.21.15.1:0/3559945778 <== mon.1 v1:172.21.15.115:6789/0 8 ==== osd_map(11..11 src has 1..11) v4 ==== 1645+0+0 (unknown 2504898309 0 0) 0x7f4844053440 con 0x7f48480b1b00 2024-03-25T18:37:23.100 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:37:23.099+0000 7f483e7fc640 10 client.4693.objecter ms_dispatch 0x7f4848000b90 osd_map(11..11 src has 1..11) v4 2024-03-25T18:37:23.100 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:37:23.099+0000 7f483e7fc640 3 client.4693.objecter handle_osd_map got epochs [11,11] > 0 2024-03-25T18:37:23.100 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:37:23.099+0000 7f483e7fc640 3 client.4693.objecter handle_osd_map decoding full epoch 11 2024-03-25T18:37:23.100 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:37:23.099+0000 7f483e7fc640 20 client.4693.objecter dump_active .. 0 homeless 2024-03-25T18:37:23.101 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:37:23.099+0000 7f483e7fc640 1 -- 172.21.15.1:0/3559945778 <== mon.1 v1:172.21.15.115:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f484404ed80 con 0x7f48480b1b00 2024-03-25T18:37:23.101 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:37:23.099+0000 7f483e7fc640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:37:23.101 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:37:23.099+0000 7f483e7fc640 10 monclient: _finish_command 1 = system:0 2024-03-25T18:37:23.342 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:37:23.340+0000 7f484fc0f640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:37:23.342 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:37:23.340+0000 7f484fc0f640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:37:23.342 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:37:23.340+0000 7f484fc0f640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.115:6789/0 2024-03-25T18:37:23.342 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:37:23.340+0000 7f484fc0f640 1 -- 172.21.15.1:0/3559945778 --> v1:172.21.15.115:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f4848136710 con 0x7f48480b1b00 2024-03-25T18:37:23.394 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:37:23.393+0000 7f483e7fc640 1 -- 172.21.15.1:0/3559945778 <== mon.1 v1:172.21.15.115:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v11) v1 ==== 74+0+7100 (unknown 238102980 0 2126609903) 0x7f4848136710 con 0x7f48480b1b00 2024-03-25T18:37:23.394 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:37:23.393+0000 7f483e7fc640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:37:23.394 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:37:23.393+0000 7f483e7fc640 10 monclient: _finish_command 2 = system:0 2024-03-25T18:37:23.395 INFO:teuthology.misc.health.smithi001.stdout: 2024-03-25T18:37:23.395 INFO:teuthology.misc.health.smithi001.stdout:{"epoch":11,"fsid":"c8ea72cf-9e99-4383-9b52-6c23d365fdfc","created":"2024-03-25T18:26:38.856638+0000","modified":"2024-03-25T18:36:43.033551+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T18:36:43.033551+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"b9f34922-3a84-49be-8426-807c1eceaefc","up":0,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["autoout","exists","new"]},{"osd":1,"uuid":"42bf306f-55e3-4634-9874-8b4b73ac6644","up":0,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["autoout","exists","new"]},{"osd":2,"uuid":"292cb23a-18a9-4207-a7cc-4d5727feaec0","up":0,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["autoout","exists","new"]},{"osd":3,"uuid":"45e0e9ab-898d-4673-9590-b4117bc5e84a","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"831a5a09-73ab-4a61-9872-088e76bdbc66","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"88cd903d-3b55-4840-9396-6e3cfe005282","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"89db7355-bc9d-46f4-8ab0-e8d9e744e026","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"cac47bee-0914-4f08-896f-348838882866","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":65536,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":65536,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":65536,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T18:37:23.397 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:37:23.395+0000 7f484fc0f640 10 librados: watch_flush enter 2024-03-25T18:37:23.397 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:37:23.395+0000 7f484fc0f640 10 librados: watch_flush exit 2024-03-25T18:37:23.397 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:37:23.395+0000 7f484fc0f640 20 client.4693.objecter shutdown clearing up homeless session... 2024-03-25T18:37:23.397 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:37:23.395+0000 7f484fc0f640 10 client.4693.objecter successfully canceled tick 2024-03-25T18:37:23.397 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:37:23.395+0000 7f484fc0f640 1 -- 172.21.15.1:0/3559945778 >> v1:172.21.15.115:6800/4237379176 conn(0x7f482c06e160 legacy=0x7f482c070620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:37:23.398 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:37:23.395+0000 7f484fc0f640 10 monclient: shutdown 2024-03-25T18:37:23.398 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:37:23.395+0000 7f484fc0f640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:37:23.398 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:37:23.395+0000 7f484fc0f640 1 -- 172.21.15.1:0/3559945778 >> v1:172.21.15.115:6789/0 conn(0x7f48480b1b00 legacy=0x7f4848135880 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:37:23.398 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:37:23.395+0000 7f484fc0f640 1 -- 172.21.15.1:0/3559945778 shutdown_connections 2024-03-25T18:37:23.398 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:37:23.395+0000 7f484fc0f640 1 -- 172.21.15.1:0/3559945778 >> 172.21.15.1:0/3559945778 conn(0x7f48480617a0 msgr2=0x7f48480b30a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:37:23.398 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:37:23.396+0000 7f484fc0f640 1 -- 172.21.15.1:0/3559945778 shutdown_connections 2024-03-25T18:37:23.398 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:37:23.396+0000 7f484fc0f640 1 -- 172.21.15.1:0/3559945778 wait complete. 2024-03-25T18:37:23.398 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:37:23.396+0000 7f484fc0f640 1 librados: shutdown 2024-03-25T18:37:23.409 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T18:37:28.265 INFO:tasks.ceph.mon.a.smithi001.stderr:==00:00:11:06.824 34876== Warning: unimplemented fcntl command: 1036 2024-03-25T18:37:28.316 INFO:tasks.ceph.mon.a.smithi001.stderr:==00:00:11:06.875 34876== Warning: unimplemented fcntl command: 1036 2024-03-25T18:37:28.358 INFO:tasks.ceph.mon.a.smithi001.stderr:==00:00:11:06.917 34876== Warning: unimplemented fcntl command: 1036 2024-03-25T18:37:28.365 INFO:tasks.ceph.mon.b.smithi115.stderr:==00:00:11:06.849 34836== Warning: unimplemented fcntl command: 1036 2024-03-25T18:37:28.373 INFO:tasks.ceph.mon.b.smithi115.stderr:==00:00:11:06.857 34836== Warning: unimplemented fcntl command: 1036 2024-03-25T18:37:28.377 INFO:tasks.ceph.mon.c.smithi001.stderr:==00:00:11:06.936 34877== Warning: unimplemented fcntl command: 1036 2024-03-25T18:37:28.390 INFO:tasks.ceph.mon.c.smithi001.stderr:==00:00:11:06.948 34877== Warning: unimplemented fcntl command: 1036 2024-03-25T18:37:28.407 INFO:tasks.ceph.mon.b.smithi115.stderr:==00:00:11:06.891 34836== Warning: unimplemented fcntl command: 1036 2024-03-25T18:37:28.419 INFO:tasks.ceph.mon.c.smithi001.stderr:==00:00:11:06.978 34877== Warning: unimplemented fcntl command: 1036 2024-03-25T18:37:30.216 DEBUG:teuthology.orchestra.run.smithi001:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T18:37:30.332 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:37:30.330+0000 7f270653e640 10 monclient: get_monmap_and_config 2024-03-25T18:37:30.332 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:37:30.330+0000 7f270653e640 10 monclient: build_initial_monmap 2024-03-25T18:37:30.332 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:37:30.330+0000 7f270653e640 10 monclient: monmap: 2024-03-25T18:37:30.332 INFO:teuthology.misc.health.smithi001.stderr:epoch 0 2024-03-25T18:37:30.332 INFO:teuthology.misc.health.smithi001.stderr:fsid c8ea72cf-9e99-4383-9b52-6c23d365fdfc 2024-03-25T18:37:30.332 INFO:teuthology.misc.health.smithi001.stderr:last_changed 2024-03-25T18:37:30.332430+0000 2024-03-25T18:37:30.333 INFO:teuthology.misc.health.smithi001.stderr:created 2024-03-25T18:37:30.332430+0000 2024-03-25T18:37:30.333 INFO:teuthology.misc.health.smithi001.stderr:min_mon_release 0 (unknown) 2024-03-25T18:37:30.333 INFO:teuthology.misc.health.smithi001.stderr:election_strategy: 3 2024-03-25T18:37:30.333 INFO:teuthology.misc.health.smithi001.stderr:0: v1:172.21.15.1:6789/0 mon.noname-a 2024-03-25T18:37:30.333 INFO:teuthology.misc.health.smithi001.stderr:1: v1:172.21.15.115:6789/0 mon.noname-c 2024-03-25T18:37:30.333 INFO:teuthology.misc.health.smithi001.stderr:2: v1:172.21.15.1:6790/0 mon.noname-b 2024-03-25T18:37:30.333 INFO:teuthology.misc.health.smithi001.stderr: 2024-03-25T18:37:30.334 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:37:30.331+0000 7f270653e640 1 Processor -- start 2024-03-25T18:37:30.334 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:37:30.331+0000 7f270653e640 1 -- start start 2024-03-25T18:37:30.334 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:37:30.331+0000 7f270653e640 10 monclient: init 2024-03-25T18:37:30.334 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:37:30.332+0000 7f270653e640 10 monclient: _reopen_session rank -1 2024-03-25T18:37:30.334 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:37:30.332+0000 7f270653e640 10 monclient: _add_conns ranks=[1,2,0] 2024-03-25T18:37:30.334 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:37:30.332+0000 7f270653e640 10 monclient(hunting): picked mon.noname-c con 0x7f27000b4ff0 addr v1:172.21.15.115:6789/0 2024-03-25T18:37:30.335 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:37:30.332+0000 7f270653e640 10 monclient(hunting): picked mon.noname-b con 0x7f27000b4670 addr v1:172.21.15.1:6790/0 2024-03-25T18:37:30.335 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:37:30.332+0000 7f270653e640 10 monclient(hunting): picked mon.noname-a con 0x7f270013c480 addr v1:172.21.15.1:6789/0 2024-03-25T18:37:30.335 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:37:30.332+0000 7f270653e640 1 -- --> v1:172.21.15.1:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f27000ab0e0 con 0x7f270013c480 2024-03-25T18:37:30.335 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:37:30.332+0000 7f270653e640 1 -- --> v1:172.21.15.115:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f27000ab3c0 con 0x7f27000b4ff0 2024-03-25T18:37:30.335 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:37:30.332+0000 7f270653e640 1 -- --> v1:172.21.15.1:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f27000a9f30 con 0x7f27000b4670 2024-03-25T18:37:30.335 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:37:30.332+0000 7f270653e640 10 monclient(hunting): _renew_subs 2024-03-25T18:37:30.335 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:37:30.332+0000 7f270653e640 10 monclient(hunting): authenticate will time out at 8130.803344s 2024-03-25T18:37:30.347 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:37:30.345+0000 7f26ff7fe640 1 --1- >> v1:172.21.15.1:6790/0 conn(0x7f27000b4670 0x7f27001425c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.1:6790/0 says I am v1:172.21.15.1:42108/0 (socket says 172.21.15.1:42108) 2024-03-25T18:37:30.347 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:37:30.345+0000 7f26ff7fe640 1 -- 172.21.15.1:0/2119856594 learned_addr learned my addr 172.21.15.1:0/2119856594 (peer_addr_for_me v1:172.21.15.1:0/0) 2024-03-25T18:37:30.365 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:37:30.363+0000 7f26feffd640 1 -- 172.21.15.1:0/2119856594 <== mon.1 v1:172.21.15.115:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1529894790 0 0) 0x7f27000ab3c0 con 0x7f27000b4ff0 2024-03-25T18:37:30.365 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:37:30.363+0000 7f26feffd640 10 monclient(hunting): _negotiate 2024-03-25T18:37:30.365 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:37:30.363+0000 7f26feffd640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:37:30.365 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:37:30.363+0000 7f26feffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:37:30.365 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:37:30.363+0000 7f26feffd640 10 monclient(hunting): my global_id is 4696 2024-03-25T18:37:30.365 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:37:30.363+0000 7f26feffd640 1 -- 172.21.15.1:0/2119856594 --> v1:172.21.15.115:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f26e8003520 con 0x7f27000b4ff0 2024-03-25T18:37:30.370 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:37:30.369+0000 7f26feffd640 1 -- 172.21.15.1:0/2119856594 <== mon.2 v1:172.21.15.1:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1108031188 0 0) 0x7f27000a9f30 con 0x7f27000b4670 2024-03-25T18:37:30.370 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:37:30.369+0000 7f26feffd640 10 monclient(hunting): _negotiate 2024-03-25T18:37:30.370 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:37:30.369+0000 7f26feffd640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:37:30.370 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:37:30.369+0000 7f26feffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:37:30.370 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:37:30.369+0000 7f26feffd640 10 monclient(hunting): my global_id is 4688 2024-03-25T18:37:30.370 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:37:30.369+0000 7f26feffd640 1 -- 172.21.15.1:0/2119856594 --> v1:172.21.15.1:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f27000ab3c0 con 0x7f27000b4670 2024-03-25T18:37:30.376 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:37:30.374+0000 7f26feffd640 1 -- 172.21.15.1:0/2119856594 <== mon.0 v1:172.21.15.1:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2165335250 0 0) 0x7f27000ab0e0 con 0x7f270013c480 2024-03-25T18:37:30.376 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:37:30.374+0000 7f26feffd640 10 monclient(hunting): _negotiate 2024-03-25T18:37:30.376 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:37:30.374+0000 7f26feffd640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:37:30.376 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:37:30.374+0000 7f26feffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:37:30.376 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:37:30.374+0000 7f26feffd640 10 monclient(hunting): my global_id is 4680 2024-03-25T18:37:30.376 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:37:30.374+0000 7f26feffd640 1 -- 172.21.15.1:0/2119856594 --> v1:172.21.15.1:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f27000a9f30 con 0x7f270013c480 2024-03-25T18:37:30.394 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:37:30.392+0000 7f26feffd640 1 -- 172.21.15.1:0/2119856594 <== mon.1 v1:172.21.15.115:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3449528194 0 0) 0x7f26e8003520 con 0x7f27000b4ff0 2024-03-25T18:37:30.394 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:37:30.392+0000 7f26feffd640 1 -- 172.21.15.1:0/2119856594 >> v1:172.21.15.1:6790/0 conn(0x7f27000b4670 legacy=0x7f27001425c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:37:30.394 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:37:30.392+0000 7f26feffd640 1 -- 172.21.15.1:0/2119856594 >> v1:172.21.15.1:6789/0 conn(0x7f270013c480 legacy=0x7f270014dc00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:37:30.395 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:37:30.392+0000 7f26feffd640 10 monclient: _finish_hunting 0 2024-03-25T18:37:30.395 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:37:30.392+0000 7f26feffd640 1 monclient: found mon.noname-c 2024-03-25T18:37:30.395 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:37:30.392+0000 7f26feffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:37:30.395 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:37:30.393+0000 7f26feffd640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.115:6789/0 2024-03-25T18:37:30.395 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:37:30.393+0000 7f26feffd640 1 -- 172.21.15.1:0/2119856594 --> v1:172.21.15.115:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f27000a9b70 con 0x7f27000b4ff0 2024-03-25T18:37:30.395 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:37:30.393+0000 7f26feffd640 10 monclient: _finish_auth 0 2024-03-25T18:37:30.395 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:37:30.393+0000 7f26feffd640 10 monclient: _check_auth_tickets 2024-03-25T18:37:30.395 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:37:30.393+0000 7f26feffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:37:30.395 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:37:30.393+0000 7f26feffd640 1 -- 172.21.15.1:0/2119856594 <== mon.1 v1:172.21.15.115:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1586965025 0 0) 0x7f26f0001690 con 0x7f27000b4ff0 2024-03-25T18:37:30.395 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:37:30.393+0000 7f26feffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:37:30.395 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:37:30.393+0000 7f26feffd640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-03-25T18:37:30.395 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:37:30.393+0000 7f26feffd640 10 monclient: dump: 2024-03-25T18:37:30.395 INFO:teuthology.misc.health.smithi001.stderr:epoch 1 2024-03-25T18:37:30.395 INFO:teuthology.misc.health.smithi001.stderr:fsid c8ea72cf-9e99-4383-9b52-6c23d365fdfc 2024-03-25T18:37:30.395 INFO:teuthology.misc.health.smithi001.stderr:last_changed 2024-03-25T18:25:50.510519+0000 2024-03-25T18:37:30.396 INFO:teuthology.misc.health.smithi001.stderr:created 2024-03-25T18:25:50.510519+0000 2024-03-25T18:37:30.396 INFO:teuthology.misc.health.smithi001.stderr:min_mon_release 19 (squid) 2024-03-25T18:37:30.396 INFO:teuthology.misc.health.smithi001.stderr:election_strategy: 3 2024-03-25T18:37:30.396 INFO:teuthology.misc.health.smithi001.stderr:0: v1:172.21.15.1:6789/0 mon.a 2024-03-25T18:37:30.396 INFO:teuthology.misc.health.smithi001.stderr:1: v1:172.21.15.115:6789/0 mon.b 2024-03-25T18:37:30.396 INFO:teuthology.misc.health.smithi001.stderr:2: v1:172.21.15.1:6790/0 mon.c 2024-03-25T18:37:30.396 INFO:teuthology.misc.health.smithi001.stderr: 2024-03-25T18:37:30.396 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:37:30.393+0000 7f270653e640 5 monclient: authenticate success, global_id 4696 2024-03-25T18:37:30.396 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:37:30.393+0000 7f270653e640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T18:37:30.424 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:37:30.423+0000 7f26feffd640 1 -- 172.21.15.1:0/2119856594 <== mon.1 v1:172.21.15.115:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f26f00017f0 con 0x7f27000b4ff0 2024-03-25T18:37:30.424 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:37:30.423+0000 7f26feffd640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:37:30.424 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:37:30.423+0000 7f26feffd640 1 -- 172.21.15.1:0/2119856594 <== mon.1 v1:172.21.15.115:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1586965025 0 0) 0x7f26f0001c00 con 0x7f27000b4ff0 2024-03-25T18:37:30.424 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:37:30.423+0000 7f26feffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:37:30.424 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:37:30.423+0000 7f26feffd640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-25T18:37:30.424 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:37:30.423+0000 7f26feffd640 10 monclient: dump: 2024-03-25T18:37:30.425 INFO:teuthology.misc.health.smithi001.stderr:epoch 1 2024-03-25T18:37:30.425 INFO:teuthology.misc.health.smithi001.stderr:fsid c8ea72cf-9e99-4383-9b52-6c23d365fdfc 2024-03-25T18:37:30.425 INFO:teuthology.misc.health.smithi001.stderr:last_changed 2024-03-25T18:25:50.510519+0000 2024-03-25T18:37:30.425 INFO:teuthology.misc.health.smithi001.stderr:created 2024-03-25T18:25:50.510519+0000 2024-03-25T18:37:30.425 INFO:teuthology.misc.health.smithi001.stderr:min_mon_release 19 (squid) 2024-03-25T18:37:30.425 INFO:teuthology.misc.health.smithi001.stderr:election_strategy: 3 2024-03-25T18:37:30.425 INFO:teuthology.misc.health.smithi001.stderr:0: v1:172.21.15.1:6789/0 mon.a 2024-03-25T18:37:30.425 INFO:teuthology.misc.health.smithi001.stderr:1: v1:172.21.15.115:6789/0 mon.b 2024-03-25T18:37:30.425 INFO:teuthology.misc.health.smithi001.stderr:2: v1:172.21.15.1:6790/0 mon.c 2024-03-25T18:37:30.425 INFO:teuthology.misc.health.smithi001.stderr: 2024-03-25T18:37:30.425 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:37:30.423+0000 7f270653e640 10 monclient: get_monmap_and_config success 2024-03-25T18:37:30.425 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:37:30.423+0000 7f270653e640 4 set_mon_vals no callback set 2024-03-25T18:37:30.427 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:37:30.423+0000 7f270653e640 10 monclient: shutdown 2024-03-25T18:37:30.427 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:37:30.423+0000 7f270653e640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:37:30.427 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:37:30.423+0000 7f270653e640 1 -- 172.21.15.1:0/2119856594 >> v1:172.21.15.115:6789/0 conn(0x7f27000b4ff0 legacy=0x7f270013bd70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:37:30.427 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:37:30.423+0000 7f270653e640 1 -- 172.21.15.1:0/2119856594 shutdown_connections 2024-03-25T18:37:30.427 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:37:30.423+0000 7f270653e640 1 -- 172.21.15.1:0/2119856594 >> 172.21.15.1:0/2119856594 conn(0x7f27000617a0 msgr2=0x7f2700061bb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:37:30.427 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:37:30.424+0000 7f270653e640 1 -- 172.21.15.1:0/2119856594 shutdown_connections 2024-03-25T18:37:30.427 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:37:30.424+0000 7f270653e640 1 -- 172.21.15.1:0/2119856594 wait complete. 2024-03-25T18:37:30.428 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:37:30.424+0000 7f270653e640 10 monclient: build_initial_monmap 2024-03-25T18:37:30.428 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:37:30.424+0000 7f270653e640 10 monclient: monmap: 2024-03-25T18:37:30.428 INFO:teuthology.misc.health.smithi001.stderr:epoch 0 2024-03-25T18:37:30.428 INFO:teuthology.misc.health.smithi001.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T18:37:30.428 INFO:teuthology.misc.health.smithi001.stderr:last_changed 0.000000 2024-03-25T18:37:30.428 INFO:teuthology.misc.health.smithi001.stderr:created 0.000000 2024-03-25T18:37:30.428 INFO:teuthology.misc.health.smithi001.stderr:min_mon_release 0 (unknown) 2024-03-25T18:37:30.428 INFO:teuthology.misc.health.smithi001.stderr:election_strategy: 1 2024-03-25T18:37:30.428 INFO:teuthology.misc.health.smithi001.stderr:0: v1:172.21.15.1:6789/0 mon.noname-a 2024-03-25T18:37:30.428 INFO:teuthology.misc.health.smithi001.stderr:1: v1:172.21.15.115:6789/0 mon.noname-b 2024-03-25T18:37:30.428 INFO:teuthology.misc.health.smithi001.stderr:2: v1:172.21.15.1:6790/0 mon.noname-c 2024-03-25T18:37:30.428 INFO:teuthology.misc.health.smithi001.stderr: 2024-03-25T18:37:30.428 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:37:30.424+0000 7f270653e640 1 librados: starting msgr at 2024-03-25T18:37:30.428 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:37:30.424+0000 7f270653e640 1 librados: starting objecter 2024-03-25T18:37:30.429 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:37:30.425+0000 7f270653e640 1 Processor -- start 2024-03-25T18:37:30.429 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:37:30.425+0000 7f270653e640 1 -- start start 2024-03-25T18:37:30.429 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:37:30.425+0000 7f270653e640 1 librados: setting wanted keys 2024-03-25T18:37:30.429 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:37:30.425+0000 7f270653e640 1 librados: calling monclient init 2024-03-25T18:37:30.429 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:37:30.425+0000 7f270653e640 10 monclient: init 2024-03-25T18:37:30.429 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:37:30.425+0000 7f270653e640 10 monclient: _reopen_session rank -1 2024-03-25T18:37:30.429 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:37:30.425+0000 7f270653e640 10 monclient: _add_conns ranks=[0,2,1] 2024-03-25T18:37:30.429 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:37:30.425+0000 7f270653e640 10 monclient(hunting): picked mon.noname-a con 0x7f27000b4670 addr v1:172.21.15.1:6789/0 2024-03-25T18:37:30.429 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:37:30.425+0000 7f270653e640 10 monclient(hunting): picked mon.noname-c con 0x7f27000b4ff0 addr v1:172.21.15.1:6790/0 2024-03-25T18:37:30.429 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:37:30.425+0000 7f270653e640 10 monclient(hunting): picked mon.noname-b con 0x7f270013c480 addr v1:172.21.15.115:6789/0 2024-03-25T18:37:30.429 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:37:30.425+0000 7f270653e640 1 -- --> v1:172.21.15.1:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2700144a00 con 0x7f27000b4670 2024-03-25T18:37:30.429 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:37:30.425+0000 7f270653e640 1 -- --> v1:172.21.15.115:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f270013f120 con 0x7f270013c480 2024-03-25T18:37:30.429 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:37:30.425+0000 7f270653e640 1 -- --> v1:172.21.15.1:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2700144c30 con 0x7f27000b4ff0 2024-03-25T18:37:30.429 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:37:30.425+0000 7f270653e640 10 monclient(hunting): _renew_subs 2024-03-25T18:37:30.430 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:37:30.425+0000 7f270653e640 10 monclient(hunting): authenticate will time out at 8130.896516s 2024-03-25T18:37:30.435 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:37:30.434+0000 7f26fffff640 1 --1- >> v1:172.21.15.1:6789/0 conn(0x7f27000b4670 0x7f27001456a0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.1:6789/0 says I am v1:172.21.15.1:41830/0 (socket says 172.21.15.1:41830) 2024-03-25T18:37:30.436 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:37:30.434+0000 7f26fffff640 1 -- 172.21.15.1:0/3017165564 learned_addr learned my addr 172.21.15.1:0/3017165564 (peer_addr_for_me v1:172.21.15.1:0/0) 2024-03-25T18:37:30.438 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:37:30.436+0000 7f26fcff9640 10 client.?.objecter ms_handle_connect 0x7f27000b4670 2024-03-25T18:37:30.438 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:37:30.436+0000 7f26fcff9640 10 client.?.objecter resend_mon_ops 2024-03-25T18:37:30.443 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:37:30.441+0000 7f26fcff9640 10 client.?.objecter ms_handle_connect 0x7f27000b4ff0 2024-03-25T18:37:30.443 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:37:30.441+0000 7f26fcff9640 10 client.?.objecter resend_mon_ops 2024-03-25T18:37:30.444 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:37:30.443+0000 7f26fcff9640 10 client.?.objecter ms_handle_connect 0x7f270013c480 2024-03-25T18:37:30.444 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:37:30.443+0000 7f26fcff9640 10 client.?.objecter resend_mon_ops 2024-03-25T18:37:30.447 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:37:30.446+0000 7f26fcff9640 1 -- 172.21.15.1:0/3017165564 <== mon.2 v1:172.21.15.1:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3645085659 0 0) 0x7f2700144c30 con 0x7f27000b4ff0 2024-03-25T18:37:30.447 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:37:30.446+0000 7f26fcff9640 10 monclient(hunting): _negotiate 2024-03-25T18:37:30.447 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:37:30.446+0000 7f26fcff9640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:37:30.447 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:37:30.446+0000 7f26fcff9640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:37:30.447 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:37:30.446+0000 7f26fcff9640 10 monclient(hunting): my global_id is 4691 2024-03-25T18:37:30.448 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:37:30.446+0000 7f26fcff9640 1 -- 172.21.15.1:0/3017165564 --> v1:172.21.15.1:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f26ec004eb0 con 0x7f27000b4ff0 2024-03-25T18:37:30.448 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:37:30.446+0000 7f26fcff9640 1 -- 172.21.15.1:0/3017165564 <== mon.0 v1:172.21.15.1:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 510365707 0 0) 0x7f2700144a00 con 0x7f27000b4670 2024-03-25T18:37:30.448 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:37:30.446+0000 7f26fcff9640 10 monclient(hunting): _negotiate 2024-03-25T18:37:30.448 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:37:30.446+0000 7f26fcff9640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:37:30.448 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:37:30.447+0000 7f26fcff9640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:37:30.448 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:37:30.447+0000 7f26fcff9640 10 monclient(hunting): my global_id is 4683 2024-03-25T18:37:30.448 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:37:30.447+0000 7f26fcff9640 1 -- 172.21.15.1:0/3017165564 --> v1:172.21.15.1:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f2700144c30 con 0x7f27000b4670 2024-03-25T18:37:30.454 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:37:30.453+0000 7f26fcff9640 1 -- 172.21.15.1:0/3017165564 <== mon.1 v1:172.21.15.115:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2089842190 0 0) 0x7f270013f120 con 0x7f270013c480 2024-03-25T18:37:30.454 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:37:30.453+0000 7f26fcff9640 10 monclient(hunting): _negotiate 2024-03-25T18:37:30.454 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:37:30.453+0000 7f26fcff9640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:37:30.454 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:37:30.453+0000 7f26fcff9640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:37:30.455 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:37:30.453+0000 7f26fcff9640 10 monclient(hunting): my global_id is 4699 2024-03-25T18:37:30.455 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:37:30.453+0000 7f26fcff9640 1 -- 172.21.15.1:0/3017165564 --> v1:172.21.15.115:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f2700144a00 con 0x7f270013c480 2024-03-25T18:37:30.460 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:37:30.459+0000 7f26fcff9640 1 -- 172.21.15.1:0/3017165564 <== mon.2 v1:172.21.15.1:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3168825502 0 0) 0x7f26ec004eb0 con 0x7f27000b4ff0 2024-03-25T18:37:30.461 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:37:30.459+0000 7f26fcff9640 1 -- 172.21.15.1:0/3017165564 --> v1:172.21.15.1:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f270013f120 con 0x7f27000b4ff0 2024-03-25T18:37:30.461 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:37:30.459+0000 7f26fcff9640 1 -- 172.21.15.1:0/3017165564 <== mon.2 v1:172.21.15.1:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1586965025 0 0) 0x7f26f4001970 con 0x7f27000b4ff0 2024-03-25T18:37:30.461 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:37:30.459+0000 7f26fcff9640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:37:30.461 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:37:30.459+0000 7f26fcff9640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-03-25T18:37:30.461 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:37:30.459+0000 7f26fcff9640 10 monclient(hunting): dump: 2024-03-25T18:37:30.461 INFO:teuthology.misc.health.smithi001.stderr:epoch 1 2024-03-25T18:37:30.461 INFO:teuthology.misc.health.smithi001.stderr:fsid c8ea72cf-9e99-4383-9b52-6c23d365fdfc 2024-03-25T18:37:30.461 INFO:teuthology.misc.health.smithi001.stderr:last_changed 2024-03-25T18:25:50.510519+0000 2024-03-25T18:37:30.461 INFO:teuthology.misc.health.smithi001.stderr:created 2024-03-25T18:25:50.510519+0000 2024-03-25T18:37:30.461 INFO:teuthology.misc.health.smithi001.stderr:min_mon_release 19 (squid) 2024-03-25T18:37:30.461 INFO:teuthology.misc.health.smithi001.stderr:election_strategy: 3 2024-03-25T18:37:30.461 INFO:teuthology.misc.health.smithi001.stderr:0: v1:172.21.15.1:6789/0 mon.a 2024-03-25T18:37:30.461 INFO:teuthology.misc.health.smithi001.stderr:1: v1:172.21.15.115:6789/0 mon.b 2024-03-25T18:37:30.461 INFO:teuthology.misc.health.smithi001.stderr:2: v1:172.21.15.1:6790/0 mon.c 2024-03-25T18:37:30.461 INFO:teuthology.misc.health.smithi001.stderr: 2024-03-25T18:37:30.462 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:37:30.459+0000 7f26fcff9640 10 monclient(hunting): _finish_auth 0 2024-03-25T18:37:30.470 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:37:30.468+0000 7f26fcff9640 1 -- 172.21.15.1:0/3017165564 <== mon.0 v1:172.21.15.1:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3051572240 0 0) 0x7f2700144c30 con 0x7f27000b4670 2024-03-25T18:37:30.470 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:37:30.468+0000 7f26fcff9640 1 -- 172.21.15.1:0/3017165564 --> v1:172.21.15.1:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f26ec004eb0 con 0x7f27000b4670 2024-03-25T18:37:30.470 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:37:30.468+0000 7f26fcff9640 1 -- 172.21.15.1:0/3017165564 <== mon.0 v1:172.21.15.1:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1586965025 0 0) 0x7f26f0001b40 con 0x7f27000b4670 2024-03-25T18:37:30.470 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:37:30.468+0000 7f26fcff9640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:37:30.470 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:37:30.468+0000 7f26fcff9640 10 monclient(hunting): got monmap 1 from mon.a (according to old e1) 2024-03-25T18:37:30.470 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:37:30.468+0000 7f26fcff9640 10 monclient(hunting): dump: 2024-03-25T18:37:30.470 INFO:teuthology.misc.health.smithi001.stderr:epoch 1 2024-03-25T18:37:30.470 INFO:teuthology.misc.health.smithi001.stderr:fsid c8ea72cf-9e99-4383-9b52-6c23d365fdfc 2024-03-25T18:37:30.470 INFO:teuthology.misc.health.smithi001.stderr:last_changed 2024-03-25T18:25:50.510519+0000 2024-03-25T18:37:30.471 INFO:teuthology.misc.health.smithi001.stderr:created 2024-03-25T18:25:50.510519+0000 2024-03-25T18:37:30.471 INFO:teuthology.misc.health.smithi001.stderr:min_mon_release 19 (squid) 2024-03-25T18:37:30.471 INFO:teuthology.misc.health.smithi001.stderr:election_strategy: 3 2024-03-25T18:37:30.471 INFO:teuthology.misc.health.smithi001.stderr:0: v1:172.21.15.1:6789/0 mon.a 2024-03-25T18:37:30.471 INFO:teuthology.misc.health.smithi001.stderr:1: v1:172.21.15.115:6789/0 mon.b 2024-03-25T18:37:30.471 INFO:teuthology.misc.health.smithi001.stderr:2: v1:172.21.15.1:6790/0 mon.c 2024-03-25T18:37:30.471 INFO:teuthology.misc.health.smithi001.stderr: 2024-03-25T18:37:30.473 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:37:30.471+0000 7f26fcff9640 1 -- 172.21.15.1:0/3017165564 <== mon.2 v1:172.21.15.1:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 2887223484 0 0) 0x7f270013f120 con 0x7f27000b4ff0 2024-03-25T18:37:30.473 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:37:30.471+0000 7f26fcff9640 1 -- 172.21.15.1:0/3017165564 >> v1:172.21.15.115:6789/0 conn(0x7f270013c480 legacy=0x7f27001303c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:37:30.473 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:37:30.471+0000 7f26fcff9640 1 -- 172.21.15.1:0/3017165564 >> v1:172.21.15.1:6789/0 conn(0x7f27000b4670 legacy=0x7f27001456a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:37:30.473 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:37:30.471+0000 7f26fcff9640 10 monclient: _finish_hunting 0 2024-03-25T18:37:30.473 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:37:30.471+0000 7f26fcff9640 1 monclient: found mon.c 2024-03-25T18:37:30.473 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:37:30.471+0000 7f26fcff9640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:37:30.473 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:37:30.471+0000 7f26fcff9640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.1:6790/0 2024-03-25T18:37:30.473 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:37:30.471+0000 7f26fcff9640 1 -- 172.21.15.1:0/3017165564 --> v1:172.21.15.1:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2700144f10 con 0x7f27000b4ff0 2024-03-25T18:37:30.473 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:37:30.471+0000 7f26fcff9640 10 monclient: _finish_auth 0 2024-03-25T18:37:30.473 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:37:30.471+0000 7f26fcff9640 10 monclient: _check_auth_tickets 2024-03-25T18:37:30.473 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:37:30.471+0000 7f26fcff9640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:37:30.473 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:37:30.471+0000 7f270653e640 5 monclient: authenticate success, global_id 4691 2024-03-25T18:37:30.474 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:37:30.471+0000 7f270653e640 10 monclient: _renew_subs 2024-03-25T18:37:30.474 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:37:30.471+0000 7f270653e640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.1:6790/0 2024-03-25T18:37:30.474 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:37:30.471+0000 7f270653e640 1 -- 172.21.15.1:0/3017165564 --> v1:172.21.15.1:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2700130ad0 con 0x7f27000b4ff0 2024-03-25T18:37:30.474 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:37:30.471+0000 7f270653e640 10 client.4691.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T18:37:30.474 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:37:30.471+0000 7f270653e640 10 monclient: _renew_subs 2024-03-25T18:37:30.474 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:37:30.471+0000 7f270653e640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.1:6790/0 2024-03-25T18:37:30.474 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:37:30.471+0000 7f270653e640 1 -- 172.21.15.1:0/3017165564 --> v1:172.21.15.1:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f2700143b20 con 0x7f27000b4ff0 2024-03-25T18:37:30.474 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:37:30.471+0000 7f270653e640 1 librados: init done 2024-03-25T18:37:30.475 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:37:30.472+0000 7f270653e640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T18:37:30.475 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:37:30.472+0000 7f270653e640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:37:30.475 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:37:30.472+0000 7f270653e640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.1:6790/0 2024-03-25T18:37:30.475 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:37:30.472+0000 7f270653e640 1 -- 172.21.15.1:0/3017165564 --> v1:172.21.15.1:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f270006dbd0 con 0x7f27000b4ff0 2024-03-25T18:37:30.792 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:37:30.791+0000 7f26fcff9640 1 -- 172.21.15.1:0/3017165564 <== mon.2 v1:172.21.15.1:6790/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f26f40047e0 con 0x7f27000b4ff0 2024-03-25T18:37:30.792 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:37:30.791+0000 7f26fcff9640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:37:30.792 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:37:30.791+0000 7f26fcff9640 1 -- 172.21.15.1:0/3017165564 <== mon.2 v1:172.21.15.1:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1586965025 0 0) 0x7f26f4004af0 con 0x7f27000b4ff0 2024-03-25T18:37:30.793 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:37:30.791+0000 7f26fcff9640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:37:30.793 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:37:30.791+0000 7f26fcff9640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-25T18:37:30.793 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:37:30.791+0000 7f26fcff9640 10 monclient: dump: 2024-03-25T18:37:30.793 INFO:teuthology.misc.health.smithi001.stderr:epoch 1 2024-03-25T18:37:30.793 INFO:teuthology.misc.health.smithi001.stderr:fsid c8ea72cf-9e99-4383-9b52-6c23d365fdfc 2024-03-25T18:37:30.793 INFO:teuthology.misc.health.smithi001.stderr:last_changed 2024-03-25T18:25:50.510519+0000 2024-03-25T18:37:30.793 INFO:teuthology.misc.health.smithi001.stderr:created 2024-03-25T18:25:50.510519+0000 2024-03-25T18:37:30.793 INFO:teuthology.misc.health.smithi001.stderr:min_mon_release 19 (squid) 2024-03-25T18:37:30.793 INFO:teuthology.misc.health.smithi001.stderr:election_strategy: 3 2024-03-25T18:37:30.793 INFO:teuthology.misc.health.smithi001.stderr:0: v1:172.21.15.1:6789/0 mon.a 2024-03-25T18:37:30.793 INFO:teuthology.misc.health.smithi001.stderr:1: v1:172.21.15.115:6789/0 mon.b 2024-03-25T18:37:30.793 INFO:teuthology.misc.health.smithi001.stderr:2: v1:172.21.15.1:6790/0 mon.c 2024-03-25T18:37:30.793 INFO:teuthology.misc.health.smithi001.stderr: 2024-03-25T18:37:30.793 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:37:30.791+0000 7f26fe7fc640 4 set_mon_vals no callback set 2024-03-25T18:37:30.794 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:37:30.791+0000 7f26fcff9640 1 -- 172.21.15.1:0/3017165564 <== mon.2 v1:172.21.15.1:6790/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 21787975 0 0) 0x7f26f401aac0 con 0x7f27000b4ff0 2024-03-25T18:37:30.794 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:37:30.792+0000 7f26fffff640 10 monclient: get_auth_request con 0x7f26ec06ded0 auth_method 0 2024-03-25T18:37:30.794 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:37:30.792+0000 7f26fcff9640 10 client.4691.objecter ms_handle_connect 0x7f26ec06ded0 2024-03-25T18:37:30.799 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:37:30.797+0000 7f26fcff9640 1 -- 172.21.15.1:0/3017165564 <== mon.2 v1:172.21.15.1:6790/0 8 ==== osd_map(11..11 src has 1..11) v4 ==== 1645+0+0 (unknown 2504898309 0 0) 0x7f26f4054810 con 0x7f27000b4ff0 2024-03-25T18:37:30.800 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:37:30.798+0000 7f26fcff9640 10 client.4691.objecter ms_dispatch 0x7f2700000b90 osd_map(11..11 src has 1..11) v4 2024-03-25T18:37:30.800 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:37:30.798+0000 7f26fcff9640 3 client.4691.objecter handle_osd_map got epochs [11,11] > 0 2024-03-25T18:37:30.800 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:37:30.798+0000 7f26fcff9640 3 client.4691.objecter handle_osd_map decoding full epoch 11 2024-03-25T18:37:30.800 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:37:30.798+0000 7f26fcff9640 20 client.4691.objecter dump_active .. 0 homeless 2024-03-25T18:37:30.800 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:37:30.798+0000 7f26fcff9640 1 -- 172.21.15.1:0/3017165564 <== mon.2 v1:172.21.15.1:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f26f4050150 con 0x7f27000b4ff0 2024-03-25T18:37:30.800 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:37:30.798+0000 7f26fcff9640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:37:30.800 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:37:30.798+0000 7f26fcff9640 10 monclient: _finish_command 1 = system:0 2024-03-25T18:37:31.038 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:37:31.036+0000 7f270653e640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:37:31.038 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:37:31.036+0000 7f270653e640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:37:31.038 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:37:31.036+0000 7f270653e640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.1:6790/0 2024-03-25T18:37:31.038 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:37:31.036+0000 7f270653e640 1 -- 172.21.15.1:0/3017165564 --> v1:172.21.15.1:6790/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f27000688c0 con 0x7f27000b4ff0 2024-03-25T18:37:31.203 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:37:31.201+0000 7f26fcff9640 1 -- 172.21.15.1:0/3017165564 <== mon.2 v1:172.21.15.1:6790/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v11) v1 ==== 74+0+7100 (unknown 238102980 0 2126609903) 0x7f26f4052250 con 0x7f27000b4ff0 2024-03-25T18:37:31.203 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:37:31.201+0000 7f26fcff9640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:37:31.203 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:37:31.201+0000 7f26fcff9640 10 monclient: _finish_command 2 = system:0 2024-03-25T18:37:31.203 INFO:teuthology.misc.health.smithi001.stdout: 2024-03-25T18:37:31.204 INFO:teuthology.misc.health.smithi001.stdout:{"epoch":11,"fsid":"c8ea72cf-9e99-4383-9b52-6c23d365fdfc","created":"2024-03-25T18:26:38.856638+0000","modified":"2024-03-25T18:36:43.033551+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T18:36:43.033551+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"b9f34922-3a84-49be-8426-807c1eceaefc","up":0,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["autoout","exists","new"]},{"osd":1,"uuid":"42bf306f-55e3-4634-9874-8b4b73ac6644","up":0,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["autoout","exists","new"]},{"osd":2,"uuid":"292cb23a-18a9-4207-a7cc-4d5727feaec0","up":0,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["autoout","exists","new"]},{"osd":3,"uuid":"45e0e9ab-898d-4673-9590-b4117bc5e84a","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"831a5a09-73ab-4a61-9872-088e76bdbc66","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"88cd903d-3b55-4840-9396-6e3cfe005282","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"89db7355-bc9d-46f4-8ab0-e8d9e744e026","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"cac47bee-0914-4f08-896f-348838882866","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":65536,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":65536,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":65536,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T18:37:31.206 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:37:31.204+0000 7f270653e640 10 librados: watch_flush enter 2024-03-25T18:37:31.206 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:37:31.204+0000 7f270653e640 10 librados: watch_flush exit 2024-03-25T18:37:31.206 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:37:31.204+0000 7f270653e640 20 client.4691.objecter shutdown clearing up homeless session... 2024-03-25T18:37:31.206 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:37:31.204+0000 7f270653e640 10 client.4691.objecter successfully canceled tick 2024-03-25T18:37:31.206 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:37:31.204+0000 7f270653e640 1 -- 172.21.15.1:0/3017165564 >> v1:172.21.15.115:6800/4237379176 conn(0x7f26ec06ded0 legacy=0x7f26ec070390 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:37:31.206 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:37:31.204+0000 7f270653e640 10 monclient: shutdown 2024-03-25T18:37:31.206 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:37:31.204+0000 7f270653e640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:37:31.206 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:37:31.204+0000 7f270653e640 1 -- 172.21.15.1:0/3017165564 >> v1:172.21.15.1:6790/0 conn(0x7f27000b4ff0 legacy=0x7f270012fcb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:37:31.206 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:37:31.204+0000 7f270653e640 1 -- 172.21.15.1:0/3017165564 shutdown_connections 2024-03-25T18:37:31.206 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:37:31.204+0000 7f270653e640 1 -- 172.21.15.1:0/3017165564 >> 172.21.15.1:0/3017165564 conn(0x7f27000617a0 msgr2=0x7f27001407f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:37:31.207 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:37:31.204+0000 7f270653e640 1 -- 172.21.15.1:0/3017165564 shutdown_connections 2024-03-25T18:37:31.207 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:37:31.204+0000 7f270653e640 1 -- 172.21.15.1:0/3017165564 wait complete. 2024-03-25T18:37:31.207 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:37:31.204+0000 7f270653e640 1 librados: shutdown 2024-03-25T18:37:31.217 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T18:37:36.948 INFO:tasks.ceph.mon.a.smithi001.stderr:==00:00:11:15.507 34876== Warning: unimplemented fcntl command: 1036 2024-03-25T18:37:36.955 INFO:tasks.ceph.mon.a.smithi001.stderr:==00:00:11:15.514 34876== Warning: unimplemented fcntl command: 1036 2024-03-25T18:37:36.977 INFO:tasks.ceph.mon.c.smithi001.stderr:==00:00:11:15.535 34877== Warning: unimplemented fcntl command: 1036 2024-03-25T18:37:36.979 INFO:tasks.ceph.mon.c.smithi001.stderr:==00:00:11:15.538 34877== Warning: unimplemented fcntl command: 1036 2024-03-25T18:37:36.982 INFO:tasks.ceph.mon.a.smithi001.stderr:==00:00:11:15.541 34876== Warning: unimplemented fcntl command: 1036 2024-03-25T18:37:36.997 INFO:tasks.ceph.mon.c.smithi001.stderr:==00:00:11:15.556 34877== Warning: unimplemented fcntl command: 1036 2024-03-25T18:37:37.135 INFO:tasks.ceph.mon.b.smithi115.stderr:==00:00:11:15.619 34836== Warning: unimplemented fcntl command: 1036 2024-03-25T18:37:37.138 INFO:tasks.ceph.mon.b.smithi115.stderr:==00:00:11:15.621 34836== Warning: unimplemented fcntl command: 1036 2024-03-25T18:37:37.161 INFO:tasks.ceph.mon.b.smithi115.stderr:==00:00:11:15.645 34836== Warning: unimplemented fcntl command: 1036 2024-03-25T18:37:38.023 DEBUG:teuthology.orchestra.run.smithi001:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T18:37:38.139 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:37:38.139+0000 7fad8ff2b640 10 monclient: get_monmap_and_config 2024-03-25T18:37:38.139 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:37:38.139+0000 7fad8ff2b640 10 monclient: build_initial_monmap 2024-03-25T18:37:38.139 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:37:38.139+0000 7fad8ff2b640 10 monclient: monmap: 2024-03-25T18:37:38.139 INFO:teuthology.misc.health.smithi001.stderr:epoch 0 2024-03-25T18:37:38.139 INFO:teuthology.misc.health.smithi001.stderr:fsid c8ea72cf-9e99-4383-9b52-6c23d365fdfc 2024-03-25T18:37:38.140 INFO:teuthology.misc.health.smithi001.stderr:last_changed 2024-03-25T18:37:38.139752+0000 2024-03-25T18:37:38.140 INFO:teuthology.misc.health.smithi001.stderr:created 2024-03-25T18:37:38.139752+0000 2024-03-25T18:37:38.140 INFO:teuthology.misc.health.smithi001.stderr:min_mon_release 0 (unknown) 2024-03-25T18:37:38.140 INFO:teuthology.misc.health.smithi001.stderr:election_strategy: 3 2024-03-25T18:37:38.140 INFO:teuthology.misc.health.smithi001.stderr:0: v1:172.21.15.1:6789/0 mon.noname-a 2024-03-25T18:37:38.140 INFO:teuthology.misc.health.smithi001.stderr:1: v1:172.21.15.115:6789/0 mon.noname-c 2024-03-25T18:37:38.140 INFO:teuthology.misc.health.smithi001.stderr:2: v1:172.21.15.1:6790/0 mon.noname-b 2024-03-25T18:37:38.140 INFO:teuthology.misc.health.smithi001.stderr: 2024-03-25T18:37:38.141 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:37:38.139+0000 7fad8ff2b640 1 Processor -- start 2024-03-25T18:37:38.141 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:37:38.140+0000 7fad8ff2b640 1 -- start start 2024-03-25T18:37:38.141 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:37:38.140+0000 7fad8ff2b640 10 monclient: init 2024-03-25T18:37:38.141 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:37:38.140+0000 7fad8ff2b640 10 monclient: _reopen_session rank -1 2024-03-25T18:37:38.141 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:37:38.140+0000 7fad8ff2b640 10 monclient: _add_conns ranks=[0,1,2] 2024-03-25T18:37:38.141 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:37:38.140+0000 7fad8ff2b640 10 monclient(hunting): picked mon.noname-a con 0x7fad880b4e70 addr v1:172.21.15.1:6789/0 2024-03-25T18:37:38.141 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:37:38.140+0000 7fad8ff2b640 10 monclient(hunting): picked mon.noname-c con 0x7fad880b44b0 addr v1:172.21.15.115:6789/0 2024-03-25T18:37:38.141 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:37:38.140+0000 7fad8ff2b640 10 monclient(hunting): picked mon.noname-b con 0x7fad880b1b00 addr v1:172.21.15.1:6790/0 2024-03-25T18:37:38.141 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:37:38.140+0000 7fad8ff2b640 1 -- --> v1:172.21.15.1:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fad880a96f0 con 0x7fad880b4e70 2024-03-25T18:37:38.142 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:37:38.140+0000 7fad8ff2b640 1 -- --> v1:172.21.15.115:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fad880ab0e0 con 0x7fad880b44b0 2024-03-25T18:37:38.142 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:37:38.140+0000 7fad8ff2b640 1 -- --> v1:172.21.15.1:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fad88145f70 con 0x7fad880b1b00 2024-03-25T18:37:38.142 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:37:38.140+0000 7fad8ff2b640 10 monclient(hunting): _renew_subs 2024-03-25T18:37:38.142 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:37:38.140+0000 7fad8ff2b640 10 monclient(hunting): authenticate will time out at 8138.610626s 2024-03-25T18:37:38.151 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:37:38.150+0000 7fad8dcc7640 1 --1- >> v1:172.21.15.1:6789/0 conn(0x7fad880b4e70 0x7fad8813f940 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.1:6789/0 says I am v1:172.21.15.1:60282/0 (socket says 172.21.15.1:60282) 2024-03-25T18:37:38.151 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:37:38.150+0000 7fad8dcc7640 1 -- 172.21.15.1:0/1976397323 learned_addr learned my addr 172.21.15.1:0/1976397323 (peer_addr_for_me v1:172.21.15.1:0/0) 2024-03-25T18:37:38.169 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:37:38.169+0000 7fad8ccc5640 1 -- 172.21.15.1:0/1976397323 <== mon.0 v1:172.21.15.1:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1546606464 0 0) 0x7fad880a96f0 con 0x7fad880b4e70 2024-03-25T18:37:38.169 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:37:38.169+0000 7fad8ccc5640 10 monclient(hunting): _negotiate 2024-03-25T18:37:38.169 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:37:38.169+0000 7fad8ccc5640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:37:38.169 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:37:38.169+0000 7fad8ccc5640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:37:38.169 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:37:38.169+0000 7fad8ccc5640 10 monclient(hunting): my global_id is 4686 2024-03-25T18:37:38.169 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:37:38.169+0000 7fad8ccc5640 1 -- 172.21.15.1:0/1976397323 --> v1:172.21.15.1:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fad74003520 con 0x7fad880b4e70 2024-03-25T18:37:38.170 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:37:38.169+0000 7fad8ccc5640 1 -- 172.21.15.1:0/1976397323 <== mon.2 v1:172.21.15.1:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1267933774 0 0) 0x7fad88145f70 con 0x7fad880b1b00 2024-03-25T18:37:38.170 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:37:38.169+0000 7fad8ccc5640 10 monclient(hunting): _negotiate 2024-03-25T18:37:38.170 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:37:38.169+0000 7fad8ccc5640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:37:38.170 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:37:38.169+0000 7fad8ccc5640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:37:38.170 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:37:38.169+0000 7fad8ccc5640 10 monclient(hunting): my global_id is 4694 2024-03-25T18:37:38.170 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:37:38.169+0000 7fad8ccc5640 1 -- 172.21.15.1:0/1976397323 --> v1:172.21.15.1:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fad880a96f0 con 0x7fad880b1b00 2024-03-25T18:37:38.173 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:37:38.172+0000 7fad8ccc5640 1 -- 172.21.15.1:0/1976397323 <== mon.1 v1:172.21.15.115:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 541372226 0 0) 0x7fad880ab0e0 con 0x7fad880b44b0 2024-03-25T18:37:38.173 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:37:38.172+0000 7fad8ccc5640 10 monclient(hunting): _negotiate 2024-03-25T18:37:38.173 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:37:38.172+0000 7fad8ccc5640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:37:38.173 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:37:38.172+0000 7fad8ccc5640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:37:38.173 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:37:38.172+0000 7fad8ccc5640 10 monclient(hunting): my global_id is 4702 2024-03-25T18:37:38.173 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:37:38.172+0000 7fad8ccc5640 1 -- 172.21.15.1:0/1976397323 --> v1:172.21.15.115:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fad88145f70 con 0x7fad880b44b0 2024-03-25T18:37:38.183 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:37:38.182+0000 7fad8ccc5640 1 -- 172.21.15.1:0/1976397323 <== mon.2 v1:172.21.15.1:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1092062604 0 0) 0x7fad880a96f0 con 0x7fad880b1b00 2024-03-25T18:37:38.183 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:37:38.182+0000 7fad8ccc5640 1 -- 172.21.15.1:0/1976397323 >> v1:172.21.15.115:6789/0 conn(0x7fad880b44b0 legacy=0x7fad88145860 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:37:38.183 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:37:38.182+0000 7fad8ccc5640 1 -- 172.21.15.1:0/1976397323 >> v1:172.21.15.1:6789/0 conn(0x7fad880b4e70 legacy=0x7fad8813f940 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:37:38.183 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:37:38.182+0000 7fad8ccc5640 10 monclient: _finish_hunting 0 2024-03-25T18:37:38.184 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:37:38.182+0000 7fad8ccc5640 1 monclient: found mon.noname-b 2024-03-25T18:37:38.184 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:37:38.182+0000 7fad8ccc5640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:37:38.184 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:37:38.182+0000 7fad8ccc5640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.1:6790/0 2024-03-25T18:37:38.184 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:37:38.182+0000 7fad8ccc5640 1 -- 172.21.15.1:0/1976397323 --> v1:172.21.15.1:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fad880a9b80 con 0x7fad880b1b00 2024-03-25T18:37:38.184 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:37:38.182+0000 7fad8ccc5640 10 monclient: _finish_auth 0 2024-03-25T18:37:38.184 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:37:38.182+0000 7fad8ccc5640 10 monclient: _check_auth_tickets 2024-03-25T18:37:38.184 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:37:38.182+0000 7fad8ccc5640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:37:38.184 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:37:38.182+0000 7fad8ccc5640 1 -- 172.21.15.1:0/1976397323 <== mon.2 v1:172.21.15.1:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1586965025 0 0) 0x7fad800019f0 con 0x7fad880b1b00 2024-03-25T18:37:38.184 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:37:38.182+0000 7fad8ccc5640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:37:38.184 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:37:38.182+0000 7fad8ccc5640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-03-25T18:37:38.184 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:37:38.182+0000 7fad8ccc5640 10 monclient: dump: 2024-03-25T18:37:38.184 INFO:teuthology.misc.health.smithi001.stderr:epoch 1 2024-03-25T18:37:38.184 INFO:teuthology.misc.health.smithi001.stderr:fsid c8ea72cf-9e99-4383-9b52-6c23d365fdfc 2024-03-25T18:37:38.184 INFO:teuthology.misc.health.smithi001.stderr:last_changed 2024-03-25T18:25:50.510519+0000 2024-03-25T18:37:38.184 INFO:teuthology.misc.health.smithi001.stderr:created 2024-03-25T18:25:50.510519+0000 2024-03-25T18:37:38.184 INFO:teuthology.misc.health.smithi001.stderr:min_mon_release 19 (squid) 2024-03-25T18:37:38.185 INFO:teuthology.misc.health.smithi001.stderr:election_strategy: 3 2024-03-25T18:37:38.185 INFO:teuthology.misc.health.smithi001.stderr:0: v1:172.21.15.1:6789/0 mon.a 2024-03-25T18:37:38.185 INFO:teuthology.misc.health.smithi001.stderr:1: v1:172.21.15.115:6789/0 mon.b 2024-03-25T18:37:38.185 INFO:teuthology.misc.health.smithi001.stderr:2: v1:172.21.15.1:6790/0 mon.c 2024-03-25T18:37:38.185 INFO:teuthology.misc.health.smithi001.stderr: 2024-03-25T18:37:38.185 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:37:38.182+0000 7fad8ff2b640 5 monclient: authenticate success, global_id 4694 2024-03-25T18:37:38.185 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:37:38.182+0000 7fad8ff2b640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T18:37:38.194 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:37:38.193+0000 7fad8ccc5640 1 -- 172.21.15.1:0/1976397323 <== mon.2 v1:172.21.15.1:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fad80001b50 con 0x7fad880b1b00 2024-03-25T18:37:38.194 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:37:38.194+0000 7fad8ccc5640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:37:38.194 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:37:38.194+0000 7fad8ccc5640 1 -- 172.21.15.1:0/1976397323 <== mon.2 v1:172.21.15.1:6790/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1586965025 0 0) 0x7fad80001f60 con 0x7fad880b1b00 2024-03-25T18:37:38.194 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:37:38.194+0000 7fad8ff2b640 10 monclient: get_monmap_and_config success 2024-03-25T18:37:38.194 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:37:38.194+0000 7fad8ff2b640 4 set_mon_vals no callback set 2024-03-25T18:37:38.194 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:37:38.194+0000 7fad8ccc5640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:37:38.194 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:37:38.194+0000 7fad8ccc5640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-25T18:37:38.195 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:37:38.194+0000 7fad8ccc5640 10 monclient: dump: 2024-03-25T18:37:38.195 INFO:teuthology.misc.health.smithi001.stderr:epoch 1 2024-03-25T18:37:38.195 INFO:teuthology.misc.health.smithi001.stderr:fsid c8ea72cf-9e99-4383-9b52-6c23d365fdfc 2024-03-25T18:37:38.195 INFO:teuthology.misc.health.smithi001.stderr:last_changed 2024-03-25T18:25:50.510519+0000 2024-03-25T18:37:38.195 INFO:teuthology.misc.health.smithi001.stderr:created 2024-03-25T18:25:50.510519+0000 2024-03-25T18:37:38.195 INFO:teuthology.misc.health.smithi001.stderr:min_mon_release 19 (squid) 2024-03-25T18:37:38.195 INFO:teuthology.misc.health.smithi001.stderr:election_strategy: 3 2024-03-25T18:37:38.195 INFO:teuthology.misc.health.smithi001.stderr:0: v1:172.21.15.1:6789/0 mon.a 2024-03-25T18:37:38.195 INFO:teuthology.misc.health.smithi001.stderr:1: v1:172.21.15.115:6789/0 mon.b 2024-03-25T18:37:38.195 INFO:teuthology.misc.health.smithi001.stderr:2: v1:172.21.15.1:6790/0 mon.c 2024-03-25T18:37:38.195 INFO:teuthology.misc.health.smithi001.stderr: 2024-03-25T18:37:38.196 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:37:38.194+0000 7fad8ff2b640 10 monclient: shutdown 2024-03-25T18:37:38.196 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:37:38.194+0000 7fad8ff2b640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:37:38.196 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:37:38.194+0000 7fad8ff2b640 1 -- 172.21.15.1:0/1976397323 >> v1:172.21.15.1:6790/0 conn(0x7fad880b1b00 legacy=0x7fad8814dc00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:37:38.196 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:37:38.194+0000 7fad8ff2b640 1 -- 172.21.15.1:0/1976397323 shutdown_connections 2024-03-25T18:37:38.197 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:37:38.194+0000 7fad8ff2b640 1 -- 172.21.15.1:0/1976397323 >> 172.21.15.1:0/1976397323 conn(0x7fad880617a0 msgr2=0x7fad88061bb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:37:38.197 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:37:38.195+0000 7fad8ff2b640 1 -- 172.21.15.1:0/1976397323 shutdown_connections 2024-03-25T18:37:38.197 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:37:38.195+0000 7fad8ff2b640 1 -- 172.21.15.1:0/1976397323 wait complete. 2024-03-25T18:37:38.197 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:37:38.195+0000 7fad8ff2b640 10 monclient: build_initial_monmap 2024-03-25T18:37:38.197 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:37:38.195+0000 7fad8ff2b640 10 monclient: monmap: 2024-03-25T18:37:38.197 INFO:teuthology.misc.health.smithi001.stderr:epoch 0 2024-03-25T18:37:38.197 INFO:teuthology.misc.health.smithi001.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T18:37:38.197 INFO:teuthology.misc.health.smithi001.stderr:last_changed 0.000000 2024-03-25T18:37:38.197 INFO:teuthology.misc.health.smithi001.stderr:created 0.000000 2024-03-25T18:37:38.197 INFO:teuthology.misc.health.smithi001.stderr:min_mon_release 0 (unknown) 2024-03-25T18:37:38.197 INFO:teuthology.misc.health.smithi001.stderr:election_strategy: 1 2024-03-25T18:37:38.197 INFO:teuthology.misc.health.smithi001.stderr:0: v1:172.21.15.1:6789/0 mon.noname-a 2024-03-25T18:37:38.197 INFO:teuthology.misc.health.smithi001.stderr:1: v1:172.21.15.115:6789/0 mon.noname-b 2024-03-25T18:37:38.197 INFO:teuthology.misc.health.smithi001.stderr:2: v1:172.21.15.1:6790/0 mon.noname-c 2024-03-25T18:37:38.197 INFO:teuthology.misc.health.smithi001.stderr: 2024-03-25T18:37:38.197 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:37:38.196+0000 7fad8ff2b640 1 librados: starting msgr at 2024-03-25T18:37:38.197 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:37:38.196+0000 7fad8ff2b640 1 librados: starting objecter 2024-03-25T18:37:38.198 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:37:38.196+0000 7fad8ff2b640 1 Processor -- start 2024-03-25T18:37:38.198 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:37:38.196+0000 7fad8ff2b640 1 -- start start 2024-03-25T18:37:38.198 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:37:38.196+0000 7fad8ff2b640 1 librados: setting wanted keys 2024-03-25T18:37:38.198 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:37:38.196+0000 7fad8ff2b640 1 librados: calling monclient init 2024-03-25T18:37:38.198 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:37:38.196+0000 7fad8ff2b640 10 monclient: init 2024-03-25T18:37:38.199 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:37:38.197+0000 7fad8ff2b640 10 monclient: _reopen_session rank -1 2024-03-25T18:37:38.199 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:37:38.197+0000 7fad8ff2b640 10 monclient: _add_conns ranks=[2,1,0] 2024-03-25T18:37:38.199 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:37:38.197+0000 7fad8ff2b640 10 monclient(hunting): picked mon.noname-c con 0x7fad880b1b00 addr v1:172.21.15.1:6790/0 2024-03-25T18:37:38.199 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:37:38.197+0000 7fad8ff2b640 10 monclient(hunting): picked mon.noname-b con 0x7fad880b44b0 addr v1:172.21.15.115:6789/0 2024-03-25T18:37:38.199 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:37:38.197+0000 7fad8ff2b640 10 monclient(hunting): picked mon.noname-a con 0x7fad880b4e70 addr v1:172.21.15.1:6789/0 2024-03-25T18:37:38.199 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:37:38.197+0000 7fad8ff2b640 1 -- --> v1:172.21.15.1:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fad88127ec0 con 0x7fad880b4e70 2024-03-25T18:37:38.199 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:37:38.197+0000 7fad8ff2b640 1 -- --> v1:172.21.15.115:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fad8813c350 con 0x7fad880b44b0 2024-03-25T18:37:38.199 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:37:38.197+0000 7fad8ff2b640 1 -- --> v1:172.21.15.1:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fad8813c5c0 con 0x7fad880b1b00 2024-03-25T18:37:38.199 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:37:38.197+0000 7fad8ff2b640 10 monclient(hunting): _renew_subs 2024-03-25T18:37:38.199 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:37:38.197+0000 7fad8ff2b640 10 monclient(hunting): authenticate will time out at 8138.667065s 2024-03-25T18:37:38.205 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:37:38.205+0000 7fad8d4c6640 1 --1- >> v1:172.21.15.115:6789/0 conn(0x7fad880b44b0 0x7fad88143660 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.115:6789/0 says I am v1:172.21.15.1:42668/0 (socket says 172.21.15.1:42668) 2024-03-25T18:37:38.205 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:37:38.205+0000 7fad8d4c6640 1 -- 172.21.15.1:0/1640677915 learned_addr learned my addr 172.21.15.1:0/1640677915 (peer_addr_for_me v1:172.21.15.1:0/0) 2024-03-25T18:37:38.207 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:37:38.207+0000 7fad7e7fc640 10 client.?.objecter ms_handle_connect 0x7fad880b44b0 2024-03-25T18:37:38.207 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:37:38.207+0000 7fad7e7fc640 10 client.?.objecter resend_mon_ops 2024-03-25T18:37:38.210 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:37:38.210+0000 7fad7e7fc640 10 client.?.objecter ms_handle_connect 0x7fad880b1b00 2024-03-25T18:37:38.210 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:37:38.210+0000 7fad7e7fc640 10 client.?.objecter resend_mon_ops 2024-03-25T18:37:38.220 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:37:38.219+0000 7fad7e7fc640 1 -- 172.21.15.1:0/1640677915 <== mon.2 v1:172.21.15.1:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3641879062 0 0) 0x7fad8813c5c0 con 0x7fad880b1b00 2024-03-25T18:37:38.220 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:37:38.219+0000 7fad7e7fc640 10 monclient(hunting): _negotiate 2024-03-25T18:37:38.220 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:37:38.219+0000 7fad7e7fc640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:37:38.220 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:37:38.219+0000 7fad7e7fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:37:38.220 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:37:38.219+0000 7fad7e7fc640 10 monclient(hunting): my global_id is 4697 2024-03-25T18:37:38.220 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:37:38.219+0000 7fad7e7fc640 1 -- 172.21.15.1:0/1640677915 --> v1:172.21.15.1:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fad6c004eb0 con 0x7fad880b1b00 2024-03-25T18:37:38.222 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:37:38.222+0000 7fad7e7fc640 10 client.?.objecter ms_handle_connect 0x7fad880b4e70 2024-03-25T18:37:38.222 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:37:38.222+0000 7fad7e7fc640 10 client.?.objecter resend_mon_ops 2024-03-25T18:37:38.224 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:37:38.223+0000 7fad7e7fc640 1 -- 172.21.15.1:0/1640677915 <== mon.1 v1:172.21.15.115:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 254977640 0 0) 0x7fad8813c350 con 0x7fad880b44b0 2024-03-25T18:37:38.224 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:37:38.223+0000 7fad7e7fc640 10 monclient(hunting): _negotiate 2024-03-25T18:37:38.224 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:37:38.223+0000 7fad7e7fc640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:37:38.224 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:37:38.223+0000 7fad7e7fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:37:38.224 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:37:38.223+0000 7fad7e7fc640 10 monclient(hunting): my global_id is 4705 2024-03-25T18:37:38.224 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:37:38.223+0000 7fad7e7fc640 1 -- 172.21.15.1:0/1640677915 --> v1:172.21.15.115:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fad8813c5c0 con 0x7fad880b44b0 2024-03-25T18:37:38.228 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:37:38.227+0000 7fad7e7fc640 1 -- 172.21.15.1:0/1640677915 <== mon.0 v1:172.21.15.1:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1150913180 0 0) 0x7fad880a9b80 con 0x7fad880b4e70 2024-03-25T18:37:38.228 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:37:38.227+0000 7fad7e7fc640 10 monclient(hunting): _negotiate 2024-03-25T18:37:38.228 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:37:38.227+0000 7fad7e7fc640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:37:38.228 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:37:38.227+0000 7fad7e7fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:37:38.228 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:37:38.227+0000 7fad7e7fc640 10 monclient(hunting): my global_id is 4689 2024-03-25T18:37:38.228 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:37:38.227+0000 7fad7e7fc640 1 -- 172.21.15.1:0/1640677915 --> v1:172.21.15.1:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fad8813c350 con 0x7fad880b4e70 2024-03-25T18:37:38.229 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:37:38.229+0000 7fad7e7fc640 1 -- 172.21.15.1:0/1640677915 <== mon.2 v1:172.21.15.1:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1942771949 0 0) 0x7fad6c004eb0 con 0x7fad880b1b00 2024-03-25T18:37:38.229 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:37:38.229+0000 7fad7e7fc640 1 -- 172.21.15.1:0/1640677915 --> v1:172.21.15.1:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fad880a9b80 con 0x7fad880b1b00 2024-03-25T18:37:38.230 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:37:38.229+0000 7fad7e7fc640 1 -- 172.21.15.1:0/1640677915 <== mon.2 v1:172.21.15.1:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1586965025 0 0) 0x7fad84001770 con 0x7fad880b1b00 2024-03-25T18:37:38.230 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:37:38.229+0000 7fad7e7fc640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:37:38.230 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:37:38.229+0000 7fad7e7fc640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-03-25T18:37:38.230 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:37:38.229+0000 7fad7e7fc640 10 monclient(hunting): dump: 2024-03-25T18:37:38.230 INFO:teuthology.misc.health.smithi001.stderr:epoch 1 2024-03-25T18:37:38.230 INFO:teuthology.misc.health.smithi001.stderr:fsid c8ea72cf-9e99-4383-9b52-6c23d365fdfc 2024-03-25T18:37:38.230 INFO:teuthology.misc.health.smithi001.stderr:last_changed 2024-03-25T18:25:50.510519+0000 2024-03-25T18:37:38.230 INFO:teuthology.misc.health.smithi001.stderr:created 2024-03-25T18:25:50.510519+0000 2024-03-25T18:37:38.230 INFO:teuthology.misc.health.smithi001.stderr:min_mon_release 19 (squid) 2024-03-25T18:37:38.230 INFO:teuthology.misc.health.smithi001.stderr:election_strategy: 3 2024-03-25T18:37:38.230 INFO:teuthology.misc.health.smithi001.stderr:0: v1:172.21.15.1:6789/0 mon.a 2024-03-25T18:37:38.230 INFO:teuthology.misc.health.smithi001.stderr:1: v1:172.21.15.115:6789/0 mon.b 2024-03-25T18:37:38.230 INFO:teuthology.misc.health.smithi001.stderr:2: v1:172.21.15.1:6790/0 mon.c 2024-03-25T18:37:38.230 INFO:teuthology.misc.health.smithi001.stderr: 2024-03-25T18:37:38.231 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:37:38.229+0000 7fad7e7fc640 10 monclient(hunting): _finish_auth 0 2024-03-25T18:37:38.235 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:37:38.234+0000 7fad7e7fc640 1 -- 172.21.15.1:0/1640677915 <== mon.2 v1:172.21.15.1:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 788800488 0 0) 0x7fad880a9b80 con 0x7fad880b1b00 2024-03-25T18:37:38.235 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:37:38.234+0000 7fad7e7fc640 1 -- 172.21.15.1:0/1640677915 >> v1:172.21.15.115:6789/0 conn(0x7fad880b44b0 legacy=0x7fad88143660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:37:38.235 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:37:38.234+0000 7fad7e7fc640 1 -- 172.21.15.1:0/1640677915 >> v1:172.21.15.1:6789/0 conn(0x7fad880b4e70 legacy=0x7fad881e1f80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:37:38.235 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:37:38.234+0000 7fad7e7fc640 10 monclient: _finish_hunting 0 2024-03-25T18:37:38.236 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:37:38.234+0000 7fad7e7fc640 1 monclient: found mon.c 2024-03-25T18:37:38.236 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:37:38.234+0000 7fad7e7fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:37:38.236 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:37:38.234+0000 7fad7e7fc640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.1:6790/0 2024-03-25T18:37:38.236 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:37:38.234+0000 7fad7e7fc640 1 -- 172.21.15.1:0/1640677915 --> v1:172.21.15.1:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fad8813c8a0 con 0x7fad880b1b00 2024-03-25T18:37:38.236 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:37:38.234+0000 7fad7e7fc640 10 monclient: _finish_auth 0 2024-03-25T18:37:38.236 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:37:38.234+0000 7fad7e7fc640 10 monclient: _check_auth_tickets 2024-03-25T18:37:38.236 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:37:38.234+0000 7fad7e7fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:37:38.236 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:37:38.234+0000 7fad8ff2b640 5 monclient: authenticate success, global_id 4697 2024-03-25T18:37:38.236 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:37:38.234+0000 7fad8ff2b640 10 monclient: _renew_subs 2024-03-25T18:37:38.236 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:37:38.234+0000 7fad8ff2b640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.1:6790/0 2024-03-25T18:37:38.236 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:37:38.234+0000 7fad8ff2b640 1 -- 172.21.15.1:0/1640677915 --> v1:172.21.15.1:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fad8813a240 con 0x7fad880b1b00 2024-03-25T18:37:38.236 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:37:38.234+0000 7fad8ff2b640 10 client.4697.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T18:37:38.236 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:37:38.234+0000 7fad8ff2b640 10 monclient: _renew_subs 2024-03-25T18:37:38.236 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:37:38.234+0000 7fad8ff2b640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.1:6790/0 2024-03-25T18:37:38.236 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:37:38.234+0000 7fad8ff2b640 1 -- 172.21.15.1:0/1640677915 --> v1:172.21.15.1:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7fad8813ccc0 con 0x7fad880b1b00 2024-03-25T18:37:38.236 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:37:38.234+0000 7fad8ff2b640 1 librados: init done 2024-03-25T18:37:38.237 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:37:38.235+0000 7fad8ff2b640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T18:37:38.237 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:37:38.235+0000 7fad8ff2b640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:37:38.237 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:37:38.235+0000 7fad8ff2b640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.1:6790/0 2024-03-25T18:37:38.237 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:37:38.235+0000 7fad8ff2b640 1 -- 172.21.15.1:0/1640677915 --> v1:172.21.15.1:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fad8813a430 con 0x7fad880b1b00 2024-03-25T18:37:38.544 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:37:38.544+0000 7fad7e7fc640 1 -- 172.21.15.1:0/1640677915 <== mon.2 v1:172.21.15.1:6790/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fad84001c20 con 0x7fad880b1b00 2024-03-25T18:37:38.544 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:37:38.544+0000 7fad7e7fc640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:37:38.544 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:37:38.544+0000 7fad7e7fc640 1 -- 172.21.15.1:0/1640677915 <== mon.2 v1:172.21.15.1:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1586965025 0 0) 0x7fad84004980 con 0x7fad880b1b00 2024-03-25T18:37:38.544 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:37:38.544+0000 7fad7e7fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:37:38.545 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:37:38.544+0000 7fad7e7fc640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-25T18:37:38.545 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:37:38.544+0000 7fad7e7fc640 10 monclient: dump: 2024-03-25T18:37:38.545 INFO:teuthology.misc.health.smithi001.stderr:epoch 1 2024-03-25T18:37:38.545 INFO:teuthology.misc.health.smithi001.stderr:fsid c8ea72cf-9e99-4383-9b52-6c23d365fdfc 2024-03-25T18:37:38.545 INFO:teuthology.misc.health.smithi001.stderr:last_changed 2024-03-25T18:25:50.510519+0000 2024-03-25T18:37:38.545 INFO:teuthology.misc.health.smithi001.stderr:created 2024-03-25T18:25:50.510519+0000 2024-03-25T18:37:38.545 INFO:teuthology.misc.health.smithi001.stderr:min_mon_release 19 (squid) 2024-03-25T18:37:38.545 INFO:teuthology.misc.health.smithi001.stderr:election_strategy: 3 2024-03-25T18:37:38.545 INFO:teuthology.misc.health.smithi001.stderr:0: v1:172.21.15.1:6789/0 mon.a 2024-03-25T18:37:38.545 INFO:teuthology.misc.health.smithi001.stderr:1: v1:172.21.15.115:6789/0 mon.b 2024-03-25T18:37:38.545 INFO:teuthology.misc.health.smithi001.stderr:2: v1:172.21.15.1:6790/0 mon.c 2024-03-25T18:37:38.545 INFO:teuthology.misc.health.smithi001.stderr: 2024-03-25T18:37:38.545 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:37:38.544+0000 7fad7ffff640 4 set_mon_vals no callback set 2024-03-25T18:37:38.546 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:37:38.544+0000 7fad7e7fc640 1 -- 172.21.15.1:0/1640677915 <== mon.2 v1:172.21.15.1:6790/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 21787975 0 0) 0x7fad84004ba0 con 0x7fad880b1b00 2024-03-25T18:37:38.546 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:37:38.545+0000 7fad8d4c6640 10 monclient: get_auth_request con 0x7fad6c06ddf0 auth_method 0 2024-03-25T18:37:38.546 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:37:38.545+0000 7fad7e7fc640 10 client.4697.objecter ms_handle_connect 0x7fad6c06ddf0 2024-03-25T18:37:38.552 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:37:38.551+0000 7fad7e7fc640 1 -- 172.21.15.1:0/1640677915 <== mon.2 v1:172.21.15.1:6790/0 8 ==== osd_map(11..11 src has 1..11) v4 ==== 1645+0+0 (unknown 2504898309 0 0) 0x7fad840546d0 con 0x7fad880b1b00 2024-03-25T18:37:38.552 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:37:38.551+0000 7fad7e7fc640 10 client.4697.objecter ms_dispatch 0x7fad88000b90 osd_map(11..11 src has 1..11) v4 2024-03-25T18:37:38.552 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:37:38.551+0000 7fad7e7fc640 3 client.4697.objecter handle_osd_map got epochs [11,11] > 0 2024-03-25T18:37:38.552 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:37:38.551+0000 7fad7e7fc640 3 client.4697.objecter handle_osd_map decoding full epoch 11 2024-03-25T18:37:38.552 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:37:38.551+0000 7fad7e7fc640 20 client.4697.objecter dump_active .. 0 homeless 2024-03-25T18:37:38.552 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:37:38.551+0000 7fad7e7fc640 1 -- 172.21.15.1:0/1640677915 <== mon.2 v1:172.21.15.1:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7fad840501e0 con 0x7fad880b1b00 2024-03-25T18:37:38.552 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:37:38.551+0000 7fad7e7fc640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:37:38.552 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:37:38.551+0000 7fad7e7fc640 10 monclient: _finish_command 1 = system:0 2024-03-25T18:37:38.816 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:37:38.815+0000 7fad8ff2b640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:37:38.816 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:37:38.816+0000 7fad8ff2b640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:37:38.816 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:37:38.816+0000 7fad8ff2b640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.1:6790/0 2024-03-25T18:37:38.817 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:37:38.816+0000 7fad8ff2b640 1 -- 172.21.15.1:0/1640677915 --> v1:172.21.15.1:6790/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fad8804f730 con 0x7fad880b1b00 2024-03-25T18:37:38.846 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:37:38.846+0000 7fad7e7fc640 1 -- 172.21.15.1:0/1640677915 <== mon.2 v1:172.21.15.1:6790/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v11) v1 ==== 74+0+7100 (unknown 238102980 0 2126609903) 0x7fad84052110 con 0x7fad880b1b00 2024-03-25T18:37:38.846 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:37:38.846+0000 7fad7e7fc640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:37:38.846 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:37:38.846+0000 7fad7e7fc640 10 monclient: _finish_command 2 = system:0 2024-03-25T18:37:38.847 INFO:teuthology.misc.health.smithi001.stdout: 2024-03-25T18:37:38.847 INFO:teuthology.misc.health.smithi001.stdout:{"epoch":11,"fsid":"c8ea72cf-9e99-4383-9b52-6c23d365fdfc","created":"2024-03-25T18:26:38.856638+0000","modified":"2024-03-25T18:36:43.033551+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T18:36:43.033551+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"b9f34922-3a84-49be-8426-807c1eceaefc","up":0,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["autoout","exists","new"]},{"osd":1,"uuid":"42bf306f-55e3-4634-9874-8b4b73ac6644","up":0,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["autoout","exists","new"]},{"osd":2,"uuid":"292cb23a-18a9-4207-a7cc-4d5727feaec0","up":0,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["autoout","exists","new"]},{"osd":3,"uuid":"45e0e9ab-898d-4673-9590-b4117bc5e84a","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"831a5a09-73ab-4a61-9872-088e76bdbc66","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"88cd903d-3b55-4840-9396-6e3cfe005282","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"89db7355-bc9d-46f4-8ab0-e8d9e744e026","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"cac47bee-0914-4f08-896f-348838882866","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":65536,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":65536,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":65536,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T18:37:38.849 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:37:38.848+0000 7fad8ff2b640 10 librados: watch_flush enter 2024-03-25T18:37:38.849 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:37:38.848+0000 7fad8ff2b640 10 librados: watch_flush exit 2024-03-25T18:37:38.849 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:37:38.848+0000 7fad8ff2b640 20 client.4697.objecter shutdown clearing up homeless session... 2024-03-25T18:37:38.849 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:37:38.848+0000 7fad8ff2b640 10 client.4697.objecter successfully canceled tick 2024-03-25T18:37:38.849 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:37:38.848+0000 7fad8ff2b640 1 -- 172.21.15.1:0/1640677915 >> v1:172.21.15.115:6800/4237379176 conn(0x7fad6c06ddf0 legacy=0x7fad6c0702b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:37:38.849 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:37:38.848+0000 7fad8ff2b640 10 monclient: shutdown 2024-03-25T18:37:38.849 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:37:38.848+0000 7fad8ff2b640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:37:38.849 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:37:38.848+0000 7fad8ff2b640 1 -- 172.21.15.1:0/1640677915 >> v1:172.21.15.1:6790/0 conn(0x7fad880b1b00 legacy=0x7fad8813afe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:37:38.849 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:37:38.848+0000 7fad8ff2b640 1 -- 172.21.15.1:0/1640677915 shutdown_connections 2024-03-25T18:37:38.850 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:37:38.848+0000 7fad8ff2b640 1 -- 172.21.15.1:0/1640677915 >> 172.21.15.1:0/1640677915 conn(0x7fad880617a0 msgr2=0x7fad880b64b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:37:38.850 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:37:38.848+0000 7fad8ff2b640 1 -- 172.21.15.1:0/1640677915 shutdown_connections 2024-03-25T18:37:38.850 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:37:38.848+0000 7fad8ff2b640 1 -- 172.21.15.1:0/1640677915 wait complete. 2024-03-25T18:37:38.850 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:37:38.848+0000 7fad8ff2b640 1 librados: shutdown 2024-03-25T18:37:38.859 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T18:37:45.666 DEBUG:teuthology.orchestra.run.smithi001:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T18:37:45.784 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:37:45.783+0000 7fd71e12b640 10 monclient: get_monmap_and_config 2024-03-25T18:37:45.784 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:37:45.783+0000 7fd71e12b640 10 monclient: build_initial_monmap 2024-03-25T18:37:45.784 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:37:45.783+0000 7fd71e12b640 10 monclient: monmap: 2024-03-25T18:37:45.784 INFO:teuthology.misc.health.smithi001.stderr:epoch 0 2024-03-25T18:37:45.784 INFO:teuthology.misc.health.smithi001.stderr:fsid c8ea72cf-9e99-4383-9b52-6c23d365fdfc 2024-03-25T18:37:45.784 INFO:teuthology.misc.health.smithi001.stderr:last_changed 2024-03-25T18:37:45.784416+0000 2024-03-25T18:37:45.784 INFO:teuthology.misc.health.smithi001.stderr:created 2024-03-25T18:37:45.784416+0000 2024-03-25T18:37:45.784 INFO:teuthology.misc.health.smithi001.stderr:min_mon_release 0 (unknown) 2024-03-25T18:37:45.784 INFO:teuthology.misc.health.smithi001.stderr:election_strategy: 3 2024-03-25T18:37:45.784 INFO:teuthology.misc.health.smithi001.stderr:0: v1:172.21.15.1:6789/0 mon.noname-a 2024-03-25T18:37:45.785 INFO:teuthology.misc.health.smithi001.stderr:1: v1:172.21.15.115:6789/0 mon.noname-c 2024-03-25T18:37:45.785 INFO:teuthology.misc.health.smithi001.stderr:2: v1:172.21.15.1:6790/0 mon.noname-b 2024-03-25T18:37:45.785 INFO:teuthology.misc.health.smithi001.stderr: 2024-03-25T18:37:45.786 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:37:45.784+0000 7fd71e12b640 1 Processor -- start 2024-03-25T18:37:45.786 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:37:45.784+0000 7fd71e12b640 1 -- start start 2024-03-25T18:37:45.786 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:37:45.784+0000 7fd71e12b640 10 monclient: init 2024-03-25T18:37:45.786 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:37:45.785+0000 7fd71e12b640 10 monclient: _reopen_session rank -1 2024-03-25T18:37:45.786 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:37:45.785+0000 7fd71e12b640 10 monclient: _add_conns ranks=[0,1,2] 2024-03-25T18:37:45.786 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:37:45.785+0000 7fd71e12b640 10 monclient(hunting): picked mon.noname-a con 0x7fd7180b4e70 addr v1:172.21.15.1:6789/0 2024-03-25T18:37:45.786 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:37:45.785+0000 7fd71e12b640 10 monclient(hunting): picked mon.noname-c con 0x7fd7180b44b0 addr v1:172.21.15.115:6789/0 2024-03-25T18:37:45.786 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:37:45.785+0000 7fd71e12b640 10 monclient(hunting): picked mon.noname-b con 0x7fd7180b1b00 addr v1:172.21.15.1:6790/0 2024-03-25T18:37:45.786 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:37:45.785+0000 7fd71e12b640 1 -- --> v1:172.21.15.1:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd7180a96f0 con 0x7fd7180b4e70 2024-03-25T18:37:45.786 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:37:45.785+0000 7fd71e12b640 1 -- --> v1:172.21.15.115:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd7180ab0e0 con 0x7fd7180b44b0 2024-03-25T18:37:45.786 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:37:45.785+0000 7fd71e12b640 1 -- --> v1:172.21.15.1:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd718145f70 con 0x7fd7180b1b00 2024-03-25T18:37:45.786 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:37:45.785+0000 7fd71e12b640 10 monclient(hunting): _renew_subs 2024-03-25T18:37:45.786 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:37:45.785+0000 7fd71e12b640 10 monclient(hunting): authenticate will time out at 8146.255214s 2024-03-25T18:37:45.795 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:37:45.794+0000 7fd716ffd640 1 --1- >> v1:172.21.15.115:6789/0 conn(0x7fd7180b44b0 0x7fd718145860 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.115:6789/0 says I am v1:172.21.15.1:35092/0 (socket says 172.21.15.1:35092) 2024-03-25T18:37:45.795 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:37:45.795+0000 7fd716ffd640 1 -- 172.21.15.1:0/1215247112 learned_addr learned my addr 172.21.15.1:0/1215247112 (peer_addr_for_me v1:172.21.15.1:0/0) 2024-03-25T18:37:45.807 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:37:45.807+0000 7fd7167fc640 1 -- 172.21.15.1:0/1215247112 <== mon.2 v1:172.21.15.1:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1423119160 0 0) 0x7fd718145f70 con 0x7fd7180b1b00 2024-03-25T18:37:45.807 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:37:45.807+0000 7fd7167fc640 10 monclient(hunting): _negotiate 2024-03-25T18:37:45.807 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:37:45.807+0000 7fd7167fc640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:37:45.808 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:37:45.807+0000 7fd7167fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:37:45.808 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:37:45.807+0000 7fd7167fc640 10 monclient(hunting): my global_id is 4700 2024-03-25T18:37:45.808 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:37:45.807+0000 7fd7167fc640 1 -- 172.21.15.1:0/1215247112 --> v1:172.21.15.1:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fd700003520 con 0x7fd7180b1b00 2024-03-25T18:37:45.808 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:37:45.807+0000 7fd7167fc640 1 -- 172.21.15.1:0/1215247112 <== mon.1 v1:172.21.15.115:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3025024670 0 0) 0x7fd7180ab0e0 con 0x7fd7180b44b0 2024-03-25T18:37:45.808 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:37:45.807+0000 7fd7167fc640 10 monclient(hunting): _negotiate 2024-03-25T18:37:45.808 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:37:45.807+0000 7fd7167fc640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:37:45.808 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:37:45.807+0000 7fd7167fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:37:45.808 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:37:45.807+0000 7fd7167fc640 10 monclient(hunting): my global_id is 4708 2024-03-25T18:37:45.808 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:37:45.807+0000 7fd7167fc640 1 -- 172.21.15.1:0/1215247112 --> v1:172.21.15.115:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fd718145f70 con 0x7fd7180b44b0 2024-03-25T18:37:45.818 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:37:45.817+0000 7fd7167fc640 1 -- 172.21.15.1:0/1215247112 <== mon.2 v1:172.21.15.1:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2103907952 0 0) 0x7fd700003520 con 0x7fd7180b1b00 2024-03-25T18:37:45.818 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:37:45.817+0000 7fd7167fc640 1 -- 172.21.15.1:0/1215247112 >> v1:172.21.15.115:6789/0 conn(0x7fd7180b44b0 legacy=0x7fd718145860 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:37:45.818 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:37:45.817+0000 7fd7167fc640 1 -- 172.21.15.1:0/1215247112 >> v1:172.21.15.1:6789/0 conn(0x7fd7180b4e70 legacy=0x7fd71813f940 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:37:45.818 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:37:45.817+0000 7fd7167fc640 10 monclient: _finish_hunting 0 2024-03-25T18:37:45.818 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:37:45.817+0000 7fd7167fc640 1 monclient: found mon.noname-b 2024-03-25T18:37:45.818 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:37:45.817+0000 7fd7167fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:37:45.818 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:37:45.817+0000 7fd7167fc640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.1:6790/0 2024-03-25T18:37:45.818 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:37:45.817+0000 7fd7167fc640 1 -- 172.21.15.1:0/1215247112 --> v1:172.21.15.1:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd7180a9b80 con 0x7fd7180b1b00 2024-03-25T18:37:45.818 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:37:45.817+0000 7fd7167fc640 10 monclient: _finish_auth 0 2024-03-25T18:37:45.818 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:37:45.817+0000 7fd7167fc640 10 monclient: _check_auth_tickets 2024-03-25T18:37:45.818 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:37:45.817+0000 7fd7167fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:37:45.819 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:37:45.817+0000 7fd7167fc640 1 -- 172.21.15.1:0/1215247112 <== mon.2 v1:172.21.15.1:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1586965025 0 0) 0x7fd710001a10 con 0x7fd7180b1b00 2024-03-25T18:37:45.819 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:37:45.817+0000 7fd71e12b640 5 monclient: authenticate success, global_id 4700 2024-03-25T18:37:45.819 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:37:45.817+0000 7fd71e12b640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T18:37:45.819 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:37:45.817+0000 7fd7167fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:37:45.819 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:37:45.817+0000 7fd7167fc640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-03-25T18:37:45.819 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:37:45.817+0000 7fd7167fc640 10 monclient: dump: 2024-03-25T18:37:45.819 INFO:teuthology.misc.health.smithi001.stderr:epoch 1 2024-03-25T18:37:45.819 INFO:teuthology.misc.health.smithi001.stderr:fsid c8ea72cf-9e99-4383-9b52-6c23d365fdfc 2024-03-25T18:37:45.819 INFO:teuthology.misc.health.smithi001.stderr:last_changed 2024-03-25T18:25:50.510519+0000 2024-03-25T18:37:45.819 INFO:teuthology.misc.health.smithi001.stderr:created 2024-03-25T18:25:50.510519+0000 2024-03-25T18:37:45.819 INFO:teuthology.misc.health.smithi001.stderr:min_mon_release 19 (squid) 2024-03-25T18:37:45.819 INFO:teuthology.misc.health.smithi001.stderr:election_strategy: 3 2024-03-25T18:37:45.819 INFO:teuthology.misc.health.smithi001.stderr:0: v1:172.21.15.1:6789/0 mon.a 2024-03-25T18:37:45.819 INFO:teuthology.misc.health.smithi001.stderr:1: v1:172.21.15.115:6789/0 mon.b 2024-03-25T18:37:45.819 INFO:teuthology.misc.health.smithi001.stderr:2: v1:172.21.15.1:6790/0 mon.c 2024-03-25T18:37:45.819 INFO:teuthology.misc.health.smithi001.stderr: 2024-03-25T18:37:45.820 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:37:45.817+0000 7fd71e12b640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T18:37:45.838 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:37:45.837+0000 7fd7167fc640 1 -- 172.21.15.1:0/1215247112 <== mon.2 v1:172.21.15.1:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fd710001b70 con 0x7fd7180b1b00 2024-03-25T18:37:45.838 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:37:45.837+0000 7fd7167fc640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:37:45.838 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:37:45.837+0000 7fd7167fc640 1 -- 172.21.15.1:0/1215247112 <== mon.2 v1:172.21.15.1:6790/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1586965025 0 0) 0x7fd710001f80 con 0x7fd7180b1b00 2024-03-25T18:37:45.838 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:37:45.837+0000 7fd71e12b640 10 monclient: get_monmap_and_config success 2024-03-25T18:37:45.838 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:37:45.837+0000 7fd71e12b640 4 set_mon_vals no callback set 2024-03-25T18:37:45.838 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:37:45.837+0000 7fd7167fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:37:45.838 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:37:45.837+0000 7fd7167fc640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-25T18:37:45.838 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:37:45.837+0000 7fd7167fc640 10 monclient: dump: 2024-03-25T18:37:45.838 INFO:teuthology.misc.health.smithi001.stderr:epoch 1 2024-03-25T18:37:45.838 INFO:teuthology.misc.health.smithi001.stderr:fsid c8ea72cf-9e99-4383-9b52-6c23d365fdfc 2024-03-25T18:37:45.838 INFO:teuthology.misc.health.smithi001.stderr:last_changed 2024-03-25T18:25:50.510519+0000 2024-03-25T18:37:45.839 INFO:teuthology.misc.health.smithi001.stderr:created 2024-03-25T18:25:50.510519+0000 2024-03-25T18:37:45.839 INFO:teuthology.misc.health.smithi001.stderr:min_mon_release 19 (squid) 2024-03-25T18:37:45.839 INFO:teuthology.misc.health.smithi001.stderr:election_strategy: 3 2024-03-25T18:37:45.839 INFO:teuthology.misc.health.smithi001.stderr:0: v1:172.21.15.1:6789/0 mon.a 2024-03-25T18:37:45.839 INFO:teuthology.misc.health.smithi001.stderr:1: v1:172.21.15.115:6789/0 mon.b 2024-03-25T18:37:45.839 INFO:teuthology.misc.health.smithi001.stderr:2: v1:172.21.15.1:6790/0 mon.c 2024-03-25T18:37:45.839 INFO:teuthology.misc.health.smithi001.stderr: 2024-03-25T18:37:45.840 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:37:45.838+0000 7fd71e12b640 10 monclient: shutdown 2024-03-25T18:37:45.840 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:37:45.838+0000 7fd71e12b640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:37:45.840 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:37:45.838+0000 7fd71e12b640 1 -- 172.21.15.1:0/1215247112 >> v1:172.21.15.1:6790/0 conn(0x7fd7180b1b00 legacy=0x7fd71814dc00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:37:45.840 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:37:45.838+0000 7fd71e12b640 1 -- 172.21.15.1:0/1215247112 shutdown_connections 2024-03-25T18:37:45.840 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:37:45.838+0000 7fd71e12b640 1 -- 172.21.15.1:0/1215247112 >> 172.21.15.1:0/1215247112 conn(0x7fd7180617a0 msgr2=0x7fd718061bb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:37:45.840 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:37:45.838+0000 7fd71e12b640 1 -- 172.21.15.1:0/1215247112 shutdown_connections 2024-03-25T18:37:45.840 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:37:45.839+0000 7fd71e12b640 1 -- 172.21.15.1:0/1215247112 wait complete. 2024-03-25T18:37:45.840 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:37:45.839+0000 7fd71e12b640 10 monclient: build_initial_monmap 2024-03-25T18:37:45.840 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:37:45.839+0000 7fd71e12b640 10 monclient: monmap: 2024-03-25T18:37:45.840 INFO:teuthology.misc.health.smithi001.stderr:epoch 0 2024-03-25T18:37:45.840 INFO:teuthology.misc.health.smithi001.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T18:37:45.840 INFO:teuthology.misc.health.smithi001.stderr:last_changed 0.000000 2024-03-25T18:37:45.841 INFO:teuthology.misc.health.smithi001.stderr:created 0.000000 2024-03-25T18:37:45.841 INFO:teuthology.misc.health.smithi001.stderr:min_mon_release 0 (unknown) 2024-03-25T18:37:45.841 INFO:teuthology.misc.health.smithi001.stderr:election_strategy: 1 2024-03-25T18:37:45.841 INFO:teuthology.misc.health.smithi001.stderr:0: v1:172.21.15.1:6789/0 mon.noname-a 2024-03-25T18:37:45.841 INFO:teuthology.misc.health.smithi001.stderr:1: v1:172.21.15.115:6789/0 mon.noname-b 2024-03-25T18:37:45.841 INFO:teuthology.misc.health.smithi001.stderr:2: v1:172.21.15.1:6790/0 mon.noname-c 2024-03-25T18:37:45.841 INFO:teuthology.misc.health.smithi001.stderr: 2024-03-25T18:37:45.841 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:37:45.840+0000 7fd71e12b640 1 librados: starting msgr at 2024-03-25T18:37:45.841 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:37:45.840+0000 7fd71e12b640 1 librados: starting objecter 2024-03-25T18:37:45.842 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:37:45.840+0000 7fd71e12b640 1 Processor -- start 2024-03-25T18:37:45.843 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:37:45.840+0000 7fd71e12b640 1 -- start start 2024-03-25T18:37:45.843 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:37:45.840+0000 7fd71e12b640 1 librados: setting wanted keys 2024-03-25T18:37:45.843 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:37:45.840+0000 7fd71e12b640 1 librados: calling monclient init 2024-03-25T18:37:45.843 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:37:45.840+0000 7fd71e12b640 10 monclient: init 2024-03-25T18:37:45.843 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:37:45.841+0000 7fd71e12b640 10 monclient: _reopen_session rank -1 2024-03-25T18:37:45.843 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:37:45.841+0000 7fd71e12b640 10 monclient: _add_conns ranks=[1,0,2] 2024-03-25T18:37:45.843 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:37:45.841+0000 7fd71e12b640 10 monclient(hunting): picked mon.noname-b con 0x7fd7180b1b00 addr v1:172.21.15.115:6789/0 2024-03-25T18:37:45.843 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:37:45.841+0000 7fd71e12b640 10 monclient(hunting): picked mon.noname-a con 0x7fd7180b44b0 addr v1:172.21.15.1:6789/0 2024-03-25T18:37:45.843 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:37:45.841+0000 7fd71e12b640 10 monclient(hunting): picked mon.noname-c con 0x7fd7180b4e70 addr v1:172.21.15.1:6790/0 2024-03-25T18:37:45.843 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:37:45.841+0000 7fd71e12b640 1 -- --> v1:172.21.15.1:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd718070cb0 con 0x7fd7180b44b0 2024-03-25T18:37:45.843 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:37:45.841+0000 7fd71e12b640 1 -- --> v1:172.21.15.115:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd718070720 con 0x7fd7180b1b00 2024-03-25T18:37:45.843 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:37:45.841+0000 7fd71e12b640 1 -- --> v1:172.21.15.1:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd7180708b0 con 0x7fd7180b4e70 2024-03-25T18:37:45.843 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:37:45.841+0000 7fd71e12b640 10 monclient(hunting): _renew_subs 2024-03-25T18:37:45.844 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:37:45.841+0000 7fd71e12b640 10 monclient(hunting): authenticate will time out at 8146.310939s 2024-03-25T18:37:45.846 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:37:45.845+0000 7fd7177fe640 1 --1- >> v1:172.21.15.115:6789/0 conn(0x7fd7180b1b00 0x7fd718070010 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.115:6789/0 says I am v1:172.21.15.1:35132/0 (socket says 172.21.15.1:35132) 2024-03-25T18:37:45.846 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:37:45.845+0000 7fd7177fe640 1 -- 172.21.15.1:0/2358222639 learned_addr learned my addr 172.21.15.1:0/2358222639 (peer_addr_for_me v1:172.21.15.1:0/0) 2024-03-25T18:37:45.847 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:37:45.846+0000 7fd6fffff640 10 client.?.objecter ms_handle_connect 0x7fd7180b4e70 2024-03-25T18:37:45.847 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:37:45.846+0000 7fd6fffff640 10 client.?.objecter resend_mon_ops 2024-03-25T18:37:45.847 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:37:45.847+0000 7fd6fffff640 10 client.?.objecter ms_handle_connect 0x7fd7180b1b00 2024-03-25T18:37:45.848 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:37:45.847+0000 7fd6fffff640 10 client.?.objecter resend_mon_ops 2024-03-25T18:37:45.863 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:37:45.862+0000 7fd6fffff640 10 client.?.objecter ms_handle_connect 0x7fd7180b44b0 2024-03-25T18:37:45.863 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:37:45.862+0000 7fd6fffff640 10 client.?.objecter resend_mon_ops 2024-03-25T18:37:45.923 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:37:45.922+0000 7fd6fffff640 1 -- 172.21.15.1:0/2358222639 <== mon.0 v1:172.21.15.1:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3649622902 0 0) 0x7fd718070cb0 con 0x7fd7180b44b0 2024-03-25T18:37:45.923 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:37:45.922+0000 7fd6fffff640 10 monclient(hunting): _negotiate 2024-03-25T18:37:45.923 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:37:45.922+0000 7fd6fffff640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:37:45.923 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:37:45.922+0000 7fd6fffff640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:37:45.923 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:37:45.922+0000 7fd6fffff640 10 monclient(hunting): my global_id is 4692 2024-03-25T18:37:45.923 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:37:45.922+0000 7fd6fffff640 1 -- 172.21.15.1:0/2358222639 --> v1:172.21.15.1:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fd704004eb0 con 0x7fd7180b44b0 2024-03-25T18:37:45.950 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:37:45.949+0000 7fd6fffff640 1 -- 172.21.15.1:0/2358222639 <== mon.0 v1:172.21.15.1:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1723955219 0 0) 0x7fd704004eb0 con 0x7fd7180b44b0 2024-03-25T18:37:45.950 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:37:45.949+0000 7fd6fffff640 1 -- 172.21.15.1:0/2358222639 --> v1:172.21.15.1:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fd718070cb0 con 0x7fd7180b44b0 2024-03-25T18:37:45.950 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:37:45.949+0000 7fd6fffff640 1 -- 172.21.15.1:0/2358222639 <== mon.0 v1:172.21.15.1:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1586965025 0 0) 0x7fd70c001930 con 0x7fd7180b44b0 2024-03-25T18:37:45.950 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:37:45.949+0000 7fd6fffff640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:37:45.950 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:37:45.949+0000 7fd6fffff640 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2024-03-25T18:37:45.950 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:37:45.949+0000 7fd6fffff640 10 monclient(hunting): dump: 2024-03-25T18:37:45.950 INFO:teuthology.misc.health.smithi001.stderr:epoch 1 2024-03-25T18:37:45.950 INFO:teuthology.misc.health.smithi001.stderr:fsid c8ea72cf-9e99-4383-9b52-6c23d365fdfc 2024-03-25T18:37:45.950 INFO:teuthology.misc.health.smithi001.stderr:last_changed 2024-03-25T18:25:50.510519+0000 2024-03-25T18:37:45.950 INFO:teuthology.misc.health.smithi001.stderr:created 2024-03-25T18:25:50.510519+0000 2024-03-25T18:37:45.950 INFO:teuthology.misc.health.smithi001.stderr:min_mon_release 19 (squid) 2024-03-25T18:37:45.950 INFO:teuthology.misc.health.smithi001.stderr:election_strategy: 3 2024-03-25T18:37:45.950 INFO:teuthology.misc.health.smithi001.stderr:0: v1:172.21.15.1:6789/0 mon.a 2024-03-25T18:37:45.950 INFO:teuthology.misc.health.smithi001.stderr:1: v1:172.21.15.115:6789/0 mon.b 2024-03-25T18:37:45.951 INFO:teuthology.misc.health.smithi001.stderr:2: v1:172.21.15.1:6790/0 mon.c 2024-03-25T18:37:45.951 INFO:teuthology.misc.health.smithi001.stderr: 2024-03-25T18:37:45.951 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:37:45.949+0000 7fd6fffff640 10 monclient(hunting): _finish_auth 0 2024-03-25T18:37:45.975 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:37:45.974+0000 7fd6fffff640 1 -- 172.21.15.1:0/2358222639 <== mon.1 v1:172.21.15.115:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1705672220 0 0) 0x7fd718070720 con 0x7fd7180b1b00 2024-03-25T18:37:45.976 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:37:45.974+0000 7fd6fffff640 10 monclient(hunting): _negotiate 2024-03-25T18:37:45.976 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:37:45.974+0000 7fd6fffff640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:37:45.976 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:37:45.974+0000 7fd6fffff640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:37:45.976 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:37:45.974+0000 7fd6fffff640 10 monclient(hunting): my global_id is 4711 2024-03-25T18:37:45.976 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:37:45.974+0000 7fd6fffff640 1 -- 172.21.15.1:0/2358222639 --> v1:172.21.15.115:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fd704004eb0 con 0x7fd7180b1b00 2024-03-25T18:37:45.976 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:37:45.975+0000 7fd6fffff640 1 -- 172.21.15.1:0/2358222639 <== mon.0 v1:172.21.15.1:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 3373048032 0 0) 0x7fd718070cb0 con 0x7fd7180b44b0 2024-03-25T18:37:45.976 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:37:45.975+0000 7fd6fffff640 1 -- 172.21.15.1:0/2358222639 >> v1:172.21.15.1:6790/0 conn(0x7fd7180b4e70 legacy=0x7fd7181dfbe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:37:45.976 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:37:45.975+0000 7fd6fffff640 1 -- 172.21.15.1:0/2358222639 >> v1:172.21.15.115:6789/0 conn(0x7fd7180b1b00 legacy=0x7fd718070010 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:37:45.976 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:37:45.975+0000 7fd6fffff640 10 monclient: _finish_hunting 0 2024-03-25T18:37:45.976 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:37:45.975+0000 7fd6fffff640 1 monclient: found mon.a 2024-03-25T18:37:45.976 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:37:45.975+0000 7fd6fffff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:37:45.976 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:37:45.975+0000 7fd6fffff640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.1:6789/0 2024-03-25T18:37:45.976 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:37:45.975+0000 7fd6fffff640 1 -- 172.21.15.1:0/2358222639 --> v1:172.21.15.1:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd718070a40 con 0x7fd7180b44b0 2024-03-25T18:37:45.976 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:37:45.975+0000 7fd6fffff640 10 monclient: _finish_auth 0 2024-03-25T18:37:45.976 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:37:45.975+0000 7fd6fffff640 10 monclient: _check_auth_tickets 2024-03-25T18:37:45.977 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:37:45.975+0000 7fd6fffff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:37:45.977 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:37:45.975+0000 7fd71e12b640 5 monclient: authenticate success, global_id 4692 2024-03-25T18:37:45.977 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:37:45.975+0000 7fd71e12b640 10 monclient: _renew_subs 2024-03-25T18:37:45.977 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:37:45.975+0000 7fd71e12b640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.1:6789/0 2024-03-25T18:37:45.977 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:37:45.975+0000 7fd71e12b640 1 -- 172.21.15.1:0/2358222639 --> v1:172.21.15.1:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd71813c3e0 con 0x7fd7180b44b0 2024-03-25T18:37:45.977 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:37:45.975+0000 7fd71e12b640 10 client.4692.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T18:37:45.977 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:37:45.975+0000 7fd71e12b640 10 monclient: _renew_subs 2024-03-25T18:37:45.977 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:37:45.975+0000 7fd71e12b640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.1:6789/0 2024-03-25T18:37:45.977 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:37:45.975+0000 7fd71e12b640 1 -- 172.21.15.1:0/2358222639 --> v1:172.21.15.1:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7fd71813c6e0 con 0x7fd7180b44b0 2024-03-25T18:37:45.977 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:37:45.975+0000 7fd71e12b640 1 librados: init done 2024-03-25T18:37:45.978 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:37:45.976+0000 7fd71e12b640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T18:37:45.978 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:37:45.976+0000 7fd71e12b640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:37:45.978 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:37:45.976+0000 7fd71e12b640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.1:6789/0 2024-03-25T18:37:45.978 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:37:45.976+0000 7fd71e12b640 1 -- 172.21.15.1:0/2358222639 --> v1:172.21.15.1:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd71804f730 con 0x7fd7180b44b0 2024-03-25T18:37:46.307 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:37:46.307+0000 7fd6fffff640 1 -- 172.21.15.1:0/2358222639 <== mon.0 v1:172.21.15.1:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fd70c0047e0 con 0x7fd7180b44b0 2024-03-25T18:37:46.308 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:37:46.307+0000 7fd6fffff640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:37:46.308 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:37:46.307+0000 7fd6fffff640 1 -- 172.21.15.1:0/2358222639 <== mon.0 v1:172.21.15.1:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1586965025 0 0) 0x7fd70c004950 con 0x7fd7180b44b0 2024-03-25T18:37:46.308 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:37:46.307+0000 7fd6fffff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:37:46.308 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:37:46.307+0000 7fd6fffff640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-25T18:37:46.308 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:37:46.307+0000 7fd6fffff640 10 monclient: dump: 2024-03-25T18:37:46.308 INFO:teuthology.misc.health.smithi001.stderr:epoch 1 2024-03-25T18:37:46.308 INFO:teuthology.misc.health.smithi001.stderr:fsid c8ea72cf-9e99-4383-9b52-6c23d365fdfc 2024-03-25T18:37:46.308 INFO:teuthology.misc.health.smithi001.stderr:last_changed 2024-03-25T18:25:50.510519+0000 2024-03-25T18:37:46.308 INFO:teuthology.misc.health.smithi001.stderr:created 2024-03-25T18:25:50.510519+0000 2024-03-25T18:37:46.308 INFO:teuthology.misc.health.smithi001.stderr:min_mon_release 19 (squid) 2024-03-25T18:37:46.308 INFO:teuthology.misc.health.smithi001.stderr:election_strategy: 3 2024-03-25T18:37:46.309 INFO:teuthology.misc.health.smithi001.stderr:0: v1:172.21.15.1:6789/0 mon.a 2024-03-25T18:37:46.309 INFO:teuthology.misc.health.smithi001.stderr:1: v1:172.21.15.115:6789/0 mon.b 2024-03-25T18:37:46.309 INFO:teuthology.misc.health.smithi001.stderr:2: v1:172.21.15.1:6790/0 mon.c 2024-03-25T18:37:46.309 INFO:teuthology.misc.health.smithi001.stderr: 2024-03-25T18:37:46.309 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:37:46.307+0000 7fd715ffb640 4 set_mon_vals no callback set 2024-03-25T18:37:46.309 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:37:46.308+0000 7fd6fffff640 1 -- 172.21.15.1:0/2358222639 <== mon.0 v1:172.21.15.1:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 21787975 0 0) 0x7fd70c01a9a0 con 0x7fd7180b44b0 2024-03-25T18:37:46.310 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:37:46.308+0000 7fd7177fe640 10 monclient: get_auth_request con 0x7fd70406dd90 auth_method 0 2024-03-25T18:37:46.310 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:37:46.309+0000 7fd6fffff640 10 client.4692.objecter ms_handle_connect 0x7fd70406dd90 2024-03-25T18:37:46.315 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:37:46.315+0000 7fd6fffff640 1 -- 172.21.15.1:0/2358222639 <== mon.0 v1:172.21.15.1:6789/0 8 ==== osd_map(11..11 src has 1..11) v4 ==== 1645+0+0 (unknown 2504898309 0 0) 0x7fd70c0545a0 con 0x7fd7180b44b0 2024-03-25T18:37:46.315 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:37:46.315+0000 7fd6fffff640 10 client.4692.objecter ms_dispatch 0x7fd718000b90 osd_map(11..11 src has 1..11) v4 2024-03-25T18:37:46.316 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:37:46.315+0000 7fd6fffff640 3 client.4692.objecter handle_osd_map got epochs [11,11] > 0 2024-03-25T18:37:46.316 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:37:46.315+0000 7fd6fffff640 3 client.4692.objecter handle_osd_map decoding full epoch 11 2024-03-25T18:37:46.316 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:37:46.315+0000 7fd6fffff640 20 client.4692.objecter dump_active .. 0 homeless 2024-03-25T18:37:46.316 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:37:46.315+0000 7fd6fffff640 1 -- 172.21.15.1:0/2358222639 <== mon.0 v1:172.21.15.1:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7fd70c04fee0 con 0x7fd7180b44b0 2024-03-25T18:37:46.316 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:37:46.315+0000 7fd6fffff640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:37:46.316 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:37:46.315+0000 7fd6fffff640 10 monclient: _finish_command 1 = system:0 2024-03-25T18:37:46.580 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:37:46.579+0000 7fd71e12b640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:37:46.580 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:37:46.579+0000 7fd71e12b640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:37:46.580 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:37:46.579+0000 7fd71e12b640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.1:6789/0 2024-03-25T18:37:46.580 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:37:46.579+0000 7fd71e12b640 1 -- 172.21.15.1:0/2358222639 --> v1:172.21.15.1:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fd718070e40 con 0x7fd7180b44b0 2024-03-25T18:37:46.628 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:37:46.628+0000 7fd6fffff640 1 -- 172.21.15.1:0/2358222639 <== mon.0 v1:172.21.15.1:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v11) v1 ==== 74+0+7100 (unknown 238102980 0 2126609903) 0x7fd70c051fe0 con 0x7fd7180b44b0 2024-03-25T18:37:46.628 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:37:46.628+0000 7fd6fffff640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:37:46.628 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:37:46.628+0000 7fd6fffff640 10 monclient: _finish_command 2 = system:0 2024-03-25T18:37:46.628 INFO:teuthology.misc.health.smithi001.stdout: 2024-03-25T18:37:46.628 INFO:teuthology.misc.health.smithi001.stdout:{"epoch":11,"fsid":"c8ea72cf-9e99-4383-9b52-6c23d365fdfc","created":"2024-03-25T18:26:38.856638+0000","modified":"2024-03-25T18:36:43.033551+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T18:36:43.033551+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"b9f34922-3a84-49be-8426-807c1eceaefc","up":0,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["autoout","exists","new"]},{"osd":1,"uuid":"42bf306f-55e3-4634-9874-8b4b73ac6644","up":0,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["autoout","exists","new"]},{"osd":2,"uuid":"292cb23a-18a9-4207-a7cc-4d5727feaec0","up":0,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["autoout","exists","new"]},{"osd":3,"uuid":"45e0e9ab-898d-4673-9590-b4117bc5e84a","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"831a5a09-73ab-4a61-9872-088e76bdbc66","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"88cd903d-3b55-4840-9396-6e3cfe005282","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"89db7355-bc9d-46f4-8ab0-e8d9e744e026","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"cac47bee-0914-4f08-896f-348838882866","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":65536,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":65536,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":65536,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T18:37:46.631 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:37:46.630+0000 7fd71e12b640 10 librados: watch_flush enter 2024-03-25T18:37:46.631 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:37:46.630+0000 7fd71e12b640 10 librados: watch_flush exit 2024-03-25T18:37:46.631 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:37:46.630+0000 7fd71e12b640 20 client.4692.objecter shutdown clearing up homeless session... 2024-03-25T18:37:46.631 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:37:46.630+0000 7fd71e12b640 10 client.4692.objecter successfully canceled tick 2024-03-25T18:37:46.631 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:37:46.630+0000 7fd71e12b640 1 -- 172.21.15.1:0/2358222639 >> v1:172.21.15.115:6800/4237379176 conn(0x7fd70406dd90 legacy=0x7fd704070250 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:37:46.631 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:37:46.630+0000 7fd71e12b640 10 monclient: shutdown 2024-03-25T18:37:46.631 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:37:46.630+0000 7fd71e12b640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:37:46.631 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:37:46.630+0000 7fd71e12b640 1 -- 172.21.15.1:0/2358222639 >> v1:172.21.15.1:6789/0 conn(0x7fd7180b44b0 legacy=0x7fd7181df4d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:37:46.632 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:37:46.630+0000 7fd71e12b640 1 -- 172.21.15.1:0/2358222639 shutdown_connections 2024-03-25T18:37:46.632 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:37:46.630+0000 7fd71e12b640 1 -- 172.21.15.1:0/2358222639 >> 172.21.15.1:0/2358222639 conn(0x7fd7180617a0 msgr2=0x7fd7180b2710 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:37:46.632 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:37:46.630+0000 7fd71e12b640 1 -- 172.21.15.1:0/2358222639 shutdown_connections 2024-03-25T18:37:46.632 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:37:46.630+0000 7fd71e12b640 1 -- 172.21.15.1:0/2358222639 wait complete. 2024-03-25T18:37:46.632 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:37:46.630+0000 7fd71e12b640 1 librados: shutdown 2024-03-25T18:37:46.642 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T18:37:48.396 INFO:tasks.ceph.mon.a.smithi001.stderr:==00:00:11:26.955 34876== Warning: unimplemented fcntl command: 1036 2024-03-25T18:37:48.401 INFO:tasks.ceph.mon.a.smithi001.stderr:==00:00:11:26.959 34876== Warning: unimplemented fcntl command: 1036 2024-03-25T18:37:48.435 INFO:tasks.ceph.mon.a.smithi001.stderr:==00:00:11:26.993 34876== Warning: unimplemented fcntl command: 1036 2024-03-25T18:37:48.459 INFO:tasks.ceph.mon.b.smithi115.stderr:==00:00:11:26.943 34836== Warning: unimplemented fcntl command: 1036 2024-03-25T18:37:48.463 INFO:tasks.ceph.mon.b.smithi115.stderr:==00:00:11:26.947 34836== Warning: unimplemented fcntl command: 1036 2024-03-25T18:37:48.464 INFO:tasks.ceph.mon.c.smithi001.stderr:==00:00:11:27.023 34877== Warning: unimplemented fcntl command: 1036 2024-03-25T18:37:48.473 INFO:tasks.ceph.mon.c.smithi001.stderr:==00:00:11:27.032 34877== Warning: unimplemented fcntl command: 1036 2024-03-25T18:37:48.494 INFO:tasks.ceph.mon.b.smithi115.stderr:==00:00:11:26.978 34836== Warning: unimplemented fcntl command: 1036 2024-03-25T18:37:48.497 INFO:tasks.ceph.mon.c.smithi001.stderr:==00:00:11:27.056 34877== Warning: unimplemented fcntl command: 1036 2024-03-25T18:37:53.449 DEBUG:teuthology.orchestra.run.smithi001:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T18:37:53.566 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:37:53.566+0000 7f83b3050640 10 monclient: get_monmap_and_config 2024-03-25T18:37:53.567 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:37:53.566+0000 7f83b3050640 10 monclient: build_initial_monmap 2024-03-25T18:37:53.567 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:37:53.566+0000 7f83b3050640 10 monclient: monmap: 2024-03-25T18:37:53.567 INFO:teuthology.misc.health.smithi001.stderr:epoch 0 2024-03-25T18:37:53.567 INFO:teuthology.misc.health.smithi001.stderr:fsid c8ea72cf-9e99-4383-9b52-6c23d365fdfc 2024-03-25T18:37:53.567 INFO:teuthology.misc.health.smithi001.stderr:last_changed 2024-03-25T18:37:53.566869+0000 2024-03-25T18:37:53.567 INFO:teuthology.misc.health.smithi001.stderr:created 2024-03-25T18:37:53.566869+0000 2024-03-25T18:37:53.567 INFO:teuthology.misc.health.smithi001.stderr:min_mon_release 0 (unknown) 2024-03-25T18:37:53.567 INFO:teuthology.misc.health.smithi001.stderr:election_strategy: 3 2024-03-25T18:37:53.567 INFO:teuthology.misc.health.smithi001.stderr:0: v1:172.21.15.1:6789/0 mon.noname-a 2024-03-25T18:37:53.567 INFO:teuthology.misc.health.smithi001.stderr:1: v1:172.21.15.115:6789/0 mon.noname-c 2024-03-25T18:37:53.567 INFO:teuthology.misc.health.smithi001.stderr:2: v1:172.21.15.1:6790/0 mon.noname-b 2024-03-25T18:37:53.568 INFO:teuthology.misc.health.smithi001.stderr: 2024-03-25T18:37:53.569 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:37:53.567+0000 7f83b3050640 1 Processor -- start 2024-03-25T18:37:53.569 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:37:53.567+0000 7f83b3050640 1 -- start start 2024-03-25T18:37:53.569 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:37:53.567+0000 7f83b3050640 10 monclient: init 2024-03-25T18:37:53.569 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:37:53.568+0000 7f83b3050640 10 monclient: _reopen_session rank -1 2024-03-25T18:37:53.569 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:37:53.568+0000 7f83b3050640 10 monclient: _add_conns ranks=[0,2,1] 2024-03-25T18:37:53.569 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:37:53.568+0000 7f83b3050640 10 monclient(hunting): picked mon.noname-a con 0x7f83ac0b4e70 addr v1:172.21.15.1:6789/0 2024-03-25T18:37:53.569 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:37:53.568+0000 7f83b3050640 10 monclient(hunting): picked mon.noname-b con 0x7f83ac0b44b0 addr v1:172.21.15.1:6790/0 2024-03-25T18:37:53.569 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:37:53.568+0000 7f83b3050640 10 monclient(hunting): picked mon.noname-c con 0x7f83ac0b1b00 addr v1:172.21.15.115:6789/0 2024-03-25T18:37:53.569 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:37:53.568+0000 7f83b3050640 1 -- --> v1:172.21.15.1:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f83ac0a96f0 con 0x7f83ac0b4e70 2024-03-25T18:37:53.570 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:37:53.568+0000 7f83b3050640 1 -- --> v1:172.21.15.115:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f83ac0ab0e0 con 0x7f83ac0b1b00 2024-03-25T18:37:53.570 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:37:53.568+0000 7f83b3050640 1 -- --> v1:172.21.15.1:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f83ac145f70 con 0x7f83ac0b44b0 2024-03-25T18:37:53.570 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:37:53.568+0000 7f83b3050640 10 monclient(hunting): _renew_subs 2024-03-25T18:37:53.570 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:37:53.568+0000 7f83b3050640 10 monclient(hunting): authenticate will time out at 8154.038204s 2024-03-25T18:37:53.572 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:37:53.572+0000 7f83b0dec640 1 --1- >> v1:172.21.15.1:6789/0 conn(0x7f83ac0b4e70 0x7f83ac13f940 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.1:6789/0 says I am v1:172.21.15.1:35808/0 (socket says 172.21.15.1:35808) 2024-03-25T18:37:53.573 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:37:53.572+0000 7f83b0dec640 1 -- 172.21.15.1:0/2430390077 learned_addr learned my addr 172.21.15.1:0/2430390077 (peer_addr_for_me v1:172.21.15.1:0/0) 2024-03-25T18:37:53.580 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:37:53.580+0000 7f83ab7fe640 1 -- 172.21.15.1:0/2430390077 <== mon.0 v1:172.21.15.1:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 515993128 0 0) 0x7f83ac0a96f0 con 0x7f83ac0b4e70 2024-03-25T18:37:53.580 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:37:53.580+0000 7f83ab7fe640 10 monclient(hunting): _negotiate 2024-03-25T18:37:53.580 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:37:53.580+0000 7f83ab7fe640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:37:53.581 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:37:53.580+0000 7f83ab7fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:37:53.581 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:37:53.580+0000 7f83ab7fe640 10 monclient(hunting): my global_id is 4695 2024-03-25T18:37:53.581 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:37:53.580+0000 7f83ab7fe640 1 -- 172.21.15.1:0/2430390077 --> v1:172.21.15.1:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f839c003520 con 0x7f83ac0b4e70 2024-03-25T18:37:53.584 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:37:53.584+0000 7f83ab7fe640 1 -- 172.21.15.1:0/2430390077 <== mon.1 v1:172.21.15.115:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 639579163 0 0) 0x7f83ac0ab0e0 con 0x7f83ac0b1b00 2024-03-25T18:37:53.584 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:37:53.584+0000 7f83ab7fe640 10 monclient(hunting): _negotiate 2024-03-25T18:37:53.585 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:37:53.584+0000 7f83ab7fe640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:37:53.585 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:37:53.584+0000 7f83ab7fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:37:53.585 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:37:53.584+0000 7f83ab7fe640 10 monclient(hunting): my global_id is 4714 2024-03-25T18:37:53.585 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:37:53.584+0000 7f83ab7fe640 1 -- 172.21.15.1:0/2430390077 --> v1:172.21.15.115:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f83ac0a96f0 con 0x7f83ac0b1b00 2024-03-25T18:37:53.594 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:37:53.594+0000 7f83ab7fe640 1 -- 172.21.15.1:0/2430390077 <== mon.0 v1:172.21.15.1:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3275307748 0 0) 0x7f839c003520 con 0x7f83ac0b4e70 2024-03-25T18:37:53.595 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:37:53.594+0000 7f83ab7fe640 1 -- 172.21.15.1:0/2430390077 >> v1:172.21.15.1:6790/0 conn(0x7f83ac0b44b0 legacy=0x7f83ac145860 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:37:53.595 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:37:53.594+0000 7f83ab7fe640 1 -- 172.21.15.1:0/2430390077 >> v1:172.21.15.115:6789/0 conn(0x7f83ac0b1b00 legacy=0x7f83ac14dc00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:37:53.595 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:37:53.594+0000 7f83ab7fe640 10 monclient: _finish_hunting 0 2024-03-25T18:37:53.595 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:37:53.594+0000 7f83ab7fe640 1 monclient: found mon.noname-a 2024-03-25T18:37:53.595 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:37:53.594+0000 7f83ab7fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:37:53.595 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:37:53.594+0000 7f83ab7fe640 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.1:6789/0 2024-03-25T18:37:53.595 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:37:53.594+0000 7f83ab7fe640 1 -- 172.21.15.1:0/2430390077 --> v1:172.21.15.1:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f83ac0a9b80 con 0x7f83ac0b4e70 2024-03-25T18:37:53.595 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:37:53.594+0000 7f83ab7fe640 10 monclient: _finish_auth 0 2024-03-25T18:37:53.595 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:37:53.594+0000 7f83ab7fe640 10 monclient: _check_auth_tickets 2024-03-25T18:37:53.595 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:37:53.594+0000 7f83ab7fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:37:53.595 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:37:53.594+0000 7f83ab7fe640 1 -- 172.21.15.1:0/2430390077 <== mon.0 v1:172.21.15.1:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1586965025 0 0) 0x7f83a0001690 con 0x7f83ac0b4e70 2024-03-25T18:37:53.595 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:37:53.594+0000 7f83ab7fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:37:53.595 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:37:53.594+0000 7f83ab7fe640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-03-25T18:37:53.596 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:37:53.594+0000 7f83ab7fe640 10 monclient: dump: 2024-03-25T18:37:53.596 INFO:teuthology.misc.health.smithi001.stderr:epoch 1 2024-03-25T18:37:53.596 INFO:teuthology.misc.health.smithi001.stderr:fsid c8ea72cf-9e99-4383-9b52-6c23d365fdfc 2024-03-25T18:37:53.596 INFO:teuthology.misc.health.smithi001.stderr:last_changed 2024-03-25T18:25:50.510519+0000 2024-03-25T18:37:53.596 INFO:teuthology.misc.health.smithi001.stderr:created 2024-03-25T18:25:50.510519+0000 2024-03-25T18:37:53.596 INFO:teuthology.misc.health.smithi001.stderr:min_mon_release 19 (squid) 2024-03-25T18:37:53.596 INFO:teuthology.misc.health.smithi001.stderr:election_strategy: 3 2024-03-25T18:37:53.596 INFO:teuthology.misc.health.smithi001.stderr:0: v1:172.21.15.1:6789/0 mon.a 2024-03-25T18:37:53.596 INFO:teuthology.misc.health.smithi001.stderr:1: v1:172.21.15.115:6789/0 mon.b 2024-03-25T18:37:53.596 INFO:teuthology.misc.health.smithi001.stderr:2: v1:172.21.15.1:6790/0 mon.c 2024-03-25T18:37:53.596 INFO:teuthology.misc.health.smithi001.stderr: 2024-03-25T18:37:53.596 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:37:53.594+0000 7f83b3050640 5 monclient: authenticate success, global_id 4695 2024-03-25T18:37:53.596 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:37:53.594+0000 7f83b3050640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T18:37:53.612 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:37:53.612+0000 7f83ab7fe640 1 -- 172.21.15.1:0/2430390077 <== mon.0 v1:172.21.15.1:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f83a00017f0 con 0x7f83ac0b4e70 2024-03-25T18:37:53.612 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:37:53.612+0000 7f83ab7fe640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:37:53.613 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:37:53.612+0000 7f83ab7fe640 1 -- 172.21.15.1:0/2430390077 <== mon.0 v1:172.21.15.1:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1586965025 0 0) 0x7f83a0001c00 con 0x7f83ac0b4e70 2024-03-25T18:37:53.613 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:37:53.612+0000 7f83b3050640 10 monclient: get_monmap_and_config success 2024-03-25T18:37:53.613 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:37:53.612+0000 7f83b3050640 4 set_mon_vals no callback set 2024-03-25T18:37:53.613 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:37:53.612+0000 7f83ab7fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:37:53.613 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:37:53.612+0000 7f83ab7fe640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-25T18:37:53.613 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:37:53.612+0000 7f83ab7fe640 10 monclient: dump: 2024-03-25T18:37:53.613 INFO:teuthology.misc.health.smithi001.stderr:epoch 1 2024-03-25T18:37:53.613 INFO:teuthology.misc.health.smithi001.stderr:fsid c8ea72cf-9e99-4383-9b52-6c23d365fdfc 2024-03-25T18:37:53.613 INFO:teuthology.misc.health.smithi001.stderr:last_changed 2024-03-25T18:25:50.510519+0000 2024-03-25T18:37:53.613 INFO:teuthology.misc.health.smithi001.stderr:created 2024-03-25T18:25:50.510519+0000 2024-03-25T18:37:53.613 INFO:teuthology.misc.health.smithi001.stderr:min_mon_release 19 (squid) 2024-03-25T18:37:53.613 INFO:teuthology.misc.health.smithi001.stderr:election_strategy: 3 2024-03-25T18:37:53.613 INFO:teuthology.misc.health.smithi001.stderr:0: v1:172.21.15.1:6789/0 mon.a 2024-03-25T18:37:53.613 INFO:teuthology.misc.health.smithi001.stderr:1: v1:172.21.15.115:6789/0 mon.b 2024-03-25T18:37:53.613 INFO:teuthology.misc.health.smithi001.stderr:2: v1:172.21.15.1:6790/0 mon.c 2024-03-25T18:37:53.614 INFO:teuthology.misc.health.smithi001.stderr: 2024-03-25T18:37:53.615 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:37:53.613+0000 7f83b3050640 10 monclient: shutdown 2024-03-25T18:37:53.615 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:37:53.613+0000 7f83b3050640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:37:53.615 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:37:53.613+0000 7f83b3050640 1 -- 172.21.15.1:0/2430390077 >> v1:172.21.15.1:6789/0 conn(0x7f83ac0b4e70 legacy=0x7f83ac13f940 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:37:53.615 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:37:53.613+0000 7f83b3050640 1 -- 172.21.15.1:0/2430390077 shutdown_connections 2024-03-25T18:37:53.615 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:37:53.613+0000 7f83b3050640 1 -- 172.21.15.1:0/2430390077 >> 172.21.15.1:0/2430390077 conn(0x7f83ac0617a0 msgr2=0x7f83ac061bb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:37:53.615 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:37:53.613+0000 7f83b3050640 1 -- 172.21.15.1:0/2430390077 shutdown_connections 2024-03-25T18:37:53.615 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:37:53.613+0000 7f83b3050640 1 -- 172.21.15.1:0/2430390077 wait complete. 2024-03-25T18:37:53.615 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:37:53.614+0000 7f83b3050640 10 monclient: build_initial_monmap 2024-03-25T18:37:53.615 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:37:53.614+0000 7f83b3050640 10 monclient: monmap: 2024-03-25T18:37:53.615 INFO:teuthology.misc.health.smithi001.stderr:epoch 0 2024-03-25T18:37:53.616 INFO:teuthology.misc.health.smithi001.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T18:37:53.616 INFO:teuthology.misc.health.smithi001.stderr:last_changed 0.000000 2024-03-25T18:37:53.616 INFO:teuthology.misc.health.smithi001.stderr:created 0.000000 2024-03-25T18:37:53.616 INFO:teuthology.misc.health.smithi001.stderr:min_mon_release 0 (unknown) 2024-03-25T18:37:53.616 INFO:teuthology.misc.health.smithi001.stderr:election_strategy: 1 2024-03-25T18:37:53.616 INFO:teuthology.misc.health.smithi001.stderr:0: v1:172.21.15.1:6789/0 mon.noname-a 2024-03-25T18:37:53.616 INFO:teuthology.misc.health.smithi001.stderr:1: v1:172.21.15.115:6789/0 mon.noname-b 2024-03-25T18:37:53.616 INFO:teuthology.misc.health.smithi001.stderr:2: v1:172.21.15.1:6790/0 mon.noname-c 2024-03-25T18:37:53.616 INFO:teuthology.misc.health.smithi001.stderr: 2024-03-25T18:37:53.616 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:37:53.614+0000 7f83b3050640 1 librados: starting msgr at 2024-03-25T18:37:53.616 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:37:53.614+0000 7f83b3050640 1 librados: starting objecter 2024-03-25T18:37:53.616 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:37:53.614+0000 7f83b3050640 1 Processor -- start 2024-03-25T18:37:53.616 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:37:53.614+0000 7f83b3050640 1 -- start start 2024-03-25T18:37:53.616 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:37:53.614+0000 7f83b3050640 1 librados: setting wanted keys 2024-03-25T18:37:53.616 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:37:53.614+0000 7f83b3050640 1 librados: calling monclient init 2024-03-25T18:37:53.616 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:37:53.614+0000 7f83b3050640 10 monclient: init 2024-03-25T18:37:53.617 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:37:53.615+0000 7f83b3050640 10 monclient: _reopen_session rank -1 2024-03-25T18:37:53.617 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:37:53.615+0000 7f83b3050640 10 monclient: _add_conns ranks=[2,0,1] 2024-03-25T18:37:53.617 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:37:53.615+0000 7f83b3050640 10 monclient(hunting): picked mon.noname-c con 0x7f83ac0b1b00 addr v1:172.21.15.1:6790/0 2024-03-25T18:37:53.617 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:37:53.615+0000 7f83b3050640 10 monclient(hunting): picked mon.noname-a con 0x7f83ac0b44b0 addr v1:172.21.15.1:6789/0 2024-03-25T18:37:53.618 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:37:53.616+0000 7f83b3050640 10 monclient(hunting): picked mon.noname-b con 0x7f83ac0b4e70 addr v1:172.21.15.115:6789/0 2024-03-25T18:37:53.618 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:37:53.616+0000 7f83b3050640 1 -- --> v1:172.21.15.1:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f83ac138120 con 0x7f83ac0b44b0 2024-03-25T18:37:53.618 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:37:53.616+0000 7f83b3050640 1 -- --> v1:172.21.15.115:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f83ac1d8710 con 0x7f83ac0b4e70 2024-03-25T18:37:53.618 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:37:53.616+0000 7f83b3050640 1 -- --> v1:172.21.15.1:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f83ac1d4bd0 con 0x7f83ac0b1b00 2024-03-25T18:37:53.618 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:37:53.616+0000 7f83b3050640 10 monclient(hunting): _renew_subs 2024-03-25T18:37:53.618 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:37:53.616+0000 7f83b3050640 10 monclient(hunting): authenticate will time out at 8154.085816s 2024-03-25T18:37:53.622 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:37:53.622+0000 7f83b0dec640 1 --1- >> v1:172.21.15.1:6790/0 conn(0x7f83ac0b1b00 0x7f83ac1d78f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.1:6790/0 says I am v1:172.21.15.1:34556/0 (socket says 172.21.15.1:34556) 2024-03-25T18:37:53.622 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:37:53.622+0000 7f83b0dec640 1 -- 172.21.15.1:0/3988127934 learned_addr learned my addr 172.21.15.1:0/3988127934 (peer_addr_for_me v1:172.21.15.1:0/0) 2024-03-25T18:37:53.629 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:37:53.629+0000 7f83a97fa640 10 client.?.objecter ms_handle_connect 0x7f83ac0b4e70 2024-03-25T18:37:53.629 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:37:53.629+0000 7f83a97fa640 10 client.?.objecter resend_mon_ops 2024-03-25T18:37:53.631 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:37:53.631+0000 7f83a97fa640 10 client.?.objecter ms_handle_connect 0x7f83ac0b1b00 2024-03-25T18:37:53.632 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:37:53.631+0000 7f83a97fa640 10 client.?.objecter resend_mon_ops 2024-03-25T18:37:53.633 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:37:53.633+0000 7f83a97fa640 10 client.?.objecter ms_handle_connect 0x7f83ac0b44b0 2024-03-25T18:37:53.633 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:37:53.633+0000 7f83a97fa640 10 client.?.objecter resend_mon_ops 2024-03-25T18:37:53.643 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:37:53.642+0000 7f83a97fa640 1 -- 172.21.15.1:0/3988127934 <== mon.0 v1:172.21.15.1:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 383795310 0 0) 0x7f83ac138120 con 0x7f83ac0b44b0 2024-03-25T18:37:53.643 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:37:53.642+0000 7f83a97fa640 10 monclient(hunting): _negotiate 2024-03-25T18:37:53.643 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:37:53.642+0000 7f83a97fa640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:37:53.643 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:37:53.642+0000 7f83a97fa640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:37:53.643 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:37:53.642+0000 7f83a97fa640 10 monclient(hunting): my global_id is 4698 2024-03-25T18:37:53.643 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:37:53.642+0000 7f83a97fa640 1 -- 172.21.15.1:0/3988127934 --> v1:172.21.15.1:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f8390004eb0 con 0x7f83ac0b44b0 2024-03-25T18:37:53.645 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:37:53.645+0000 7f83a97fa640 1 -- 172.21.15.1:0/3988127934 <== mon.1 v1:172.21.15.115:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1888288765 0 0) 0x7f83ac1d8710 con 0x7f83ac0b4e70 2024-03-25T18:37:53.645 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:37:53.645+0000 7f83a97fa640 10 monclient(hunting): _negotiate 2024-03-25T18:37:53.645 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:37:53.645+0000 7f83a97fa640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:37:53.645 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:37:53.645+0000 7f83a97fa640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:37:53.646 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:37:53.645+0000 7f83a97fa640 10 monclient(hunting): my global_id is 4717 2024-03-25T18:37:53.646 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:37:53.645+0000 7f83a97fa640 1 -- 172.21.15.1:0/3988127934 --> v1:172.21.15.115:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f83ac138120 con 0x7f83ac0b4e70 2024-03-25T18:37:53.649 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:37:53.648+0000 7f83a97fa640 1 -- 172.21.15.1:0/3988127934 <== mon.2 v1:172.21.15.1:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1881483637 0 0) 0x7f83ac1d4bd0 con 0x7f83ac0b1b00 2024-03-25T18:37:53.649 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:37:53.648+0000 7f83a97fa640 10 monclient(hunting): _negotiate 2024-03-25T18:37:53.649 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:37:53.648+0000 7f83a97fa640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:37:53.649 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:37:53.648+0000 7f83a97fa640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:37:53.649 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:37:53.648+0000 7f83a97fa640 10 monclient(hunting): my global_id is 4709 2024-03-25T18:37:53.649 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:37:53.648+0000 7f83a97fa640 1 -- 172.21.15.1:0/3988127934 --> v1:172.21.15.1:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f83ac1d8710 con 0x7f83ac0b1b00 2024-03-25T18:37:53.656 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:37:53.656+0000 7f83a97fa640 1 -- 172.21.15.1:0/3988127934 <== mon.0 v1:172.21.15.1:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1860204109 0 0) 0x7f8390004eb0 con 0x7f83ac0b44b0 2024-03-25T18:37:53.656 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:37:53.656+0000 7f83a97fa640 1 -- 172.21.15.1:0/3988127934 --> v1:172.21.15.1:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f83ac1d4bd0 con 0x7f83ac0b44b0 2024-03-25T18:37:53.657 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:37:53.656+0000 7f83a97fa640 1 -- 172.21.15.1:0/3988127934 <== mon.0 v1:172.21.15.1:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1586965025 0 0) 0x7f8398001930 con 0x7f83ac0b44b0 2024-03-25T18:37:53.657 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:37:53.656+0000 7f83a97fa640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:37:53.657 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:37:53.656+0000 7f83a97fa640 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2024-03-25T18:37:53.657 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:37:53.656+0000 7f83a97fa640 10 monclient(hunting): dump: 2024-03-25T18:37:53.657 INFO:teuthology.misc.health.smithi001.stderr:epoch 1 2024-03-25T18:37:53.657 INFO:teuthology.misc.health.smithi001.stderr:fsid c8ea72cf-9e99-4383-9b52-6c23d365fdfc 2024-03-25T18:37:53.657 INFO:teuthology.misc.health.smithi001.stderr:last_changed 2024-03-25T18:25:50.510519+0000 2024-03-25T18:37:53.657 INFO:teuthology.misc.health.smithi001.stderr:created 2024-03-25T18:25:50.510519+0000 2024-03-25T18:37:53.657 INFO:teuthology.misc.health.smithi001.stderr:min_mon_release 19 (squid) 2024-03-25T18:37:53.657 INFO:teuthology.misc.health.smithi001.stderr:election_strategy: 3 2024-03-25T18:37:53.657 INFO:teuthology.misc.health.smithi001.stderr:0: v1:172.21.15.1:6789/0 mon.a 2024-03-25T18:37:53.657 INFO:teuthology.misc.health.smithi001.stderr:1: v1:172.21.15.115:6789/0 mon.b 2024-03-25T18:37:53.658 INFO:teuthology.misc.health.smithi001.stderr:2: v1:172.21.15.1:6790/0 mon.c 2024-03-25T18:37:53.658 INFO:teuthology.misc.health.smithi001.stderr: 2024-03-25T18:37:53.658 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:37:53.656+0000 7f83a97fa640 10 monclient(hunting): _finish_auth 0 2024-03-25T18:37:53.663 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:37:53.662+0000 7f83a97fa640 1 -- 172.21.15.1:0/3988127934 <== mon.2 v1:172.21.15.1:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 4286792723 0 0) 0x7f83ac1d8710 con 0x7f83ac0b1b00 2024-03-25T18:37:53.663 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:37:53.662+0000 7f83a97fa640 1 -- 172.21.15.1:0/3988127934 --> v1:172.21.15.1:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f8390004eb0 con 0x7f83ac0b1b00 2024-03-25T18:37:53.663 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:37:53.662+0000 7f83a97fa640 1 -- 172.21.15.1:0/3988127934 <== mon.2 v1:172.21.15.1:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1586965025 0 0) 0x7f83a0001b00 con 0x7f83ac0b1b00 2024-03-25T18:37:53.663 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:37:53.662+0000 7f83a97fa640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:37:53.663 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:37:53.662+0000 7f83a97fa640 10 monclient(hunting): got monmap 1 from mon.c (according to old e1) 2024-03-25T18:37:53.663 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:37:53.662+0000 7f83a97fa640 10 monclient(hunting): dump: 2024-03-25T18:37:53.663 INFO:teuthology.misc.health.smithi001.stderr:epoch 1 2024-03-25T18:37:53.663 INFO:teuthology.misc.health.smithi001.stderr:fsid c8ea72cf-9e99-4383-9b52-6c23d365fdfc 2024-03-25T18:37:53.663 INFO:teuthology.misc.health.smithi001.stderr:last_changed 2024-03-25T18:25:50.510519+0000 2024-03-25T18:37:53.663 INFO:teuthology.misc.health.smithi001.stderr:created 2024-03-25T18:25:50.510519+0000 2024-03-25T18:37:53.663 INFO:teuthology.misc.health.smithi001.stderr:min_mon_release 19 (squid) 2024-03-25T18:37:53.664 INFO:teuthology.misc.health.smithi001.stderr:election_strategy: 3 2024-03-25T18:37:53.664 INFO:teuthology.misc.health.smithi001.stderr:0: v1:172.21.15.1:6789/0 mon.a 2024-03-25T18:37:53.664 INFO:teuthology.misc.health.smithi001.stderr:1: v1:172.21.15.115:6789/0 mon.b 2024-03-25T18:37:53.664 INFO:teuthology.misc.health.smithi001.stderr:2: v1:172.21.15.1:6790/0 mon.c 2024-03-25T18:37:53.664 INFO:teuthology.misc.health.smithi001.stderr: 2024-03-25T18:37:53.668 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:37:53.667+0000 7f83a97fa640 1 -- 172.21.15.1:0/3988127934 <== mon.0 v1:172.21.15.1:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 3389825556 0 0) 0x7f83ac1d4bd0 con 0x7f83ac0b44b0 2024-03-25T18:37:53.668 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:37:53.667+0000 7f83a97fa640 1 -- 172.21.15.1:0/3988127934 >> v1:172.21.15.1:6790/0 conn(0x7f83ac0b1b00 legacy=0x7f83ac1d78f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:37:53.668 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:37:53.667+0000 7f83a97fa640 1 -- 172.21.15.1:0/3988127934 >> v1:172.21.15.115:6789/0 conn(0x7f83ac0b4e70 legacy=0x7f83ac1e3130 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:37:53.668 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:37:53.667+0000 7f83a97fa640 10 monclient: _finish_hunting 0 2024-03-25T18:37:53.669 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:37:53.667+0000 7f83a97fa640 1 monclient: found mon.a 2024-03-25T18:37:53.669 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:37:53.667+0000 7f83a97fa640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:37:53.669 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:37:53.667+0000 7f83a97fa640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.1:6789/0 2024-03-25T18:37:53.669 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:37:53.667+0000 7f83a97fa640 1 -- 172.21.15.1:0/3988127934 --> v1:172.21.15.1:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f83ac1dbaa0 con 0x7f83ac0b44b0 2024-03-25T18:37:53.669 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:37:53.667+0000 7f83a97fa640 10 monclient: _finish_auth 0 2024-03-25T18:37:53.669 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:37:53.667+0000 7f83a97fa640 10 monclient: _check_auth_tickets 2024-03-25T18:37:53.669 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:37:53.667+0000 7f83a97fa640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:37:53.669 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:37:53.667+0000 7f83b3050640 5 monclient: authenticate success, global_id 4698 2024-03-25T18:37:53.669 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:37:53.667+0000 7f83b3050640 10 monclient: _renew_subs 2024-03-25T18:37:53.669 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:37:53.667+0000 7f83b3050640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.1:6789/0 2024-03-25T18:37:53.669 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:37:53.667+0000 7f83b3050640 1 -- 172.21.15.1:0/3988127934 --> v1:172.21.15.1:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f83ac1dbcf0 con 0x7f83ac0b44b0 2024-03-25T18:37:53.669 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:37:53.667+0000 7f83b3050640 10 client.4698.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T18:37:53.670 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:37:53.667+0000 7f83b3050640 10 monclient: _renew_subs 2024-03-25T18:37:53.670 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:37:53.667+0000 7f83b3050640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.1:6789/0 2024-03-25T18:37:53.670 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:37:53.667+0000 7f83b3050640 1 -- 172.21.15.1:0/3988127934 --> v1:172.21.15.1:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f83ac1db2f0 con 0x7f83ac0b44b0 2024-03-25T18:37:53.670 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:37:53.667+0000 7f83b3050640 1 librados: init done 2024-03-25T18:37:53.670 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:37:53.668+0000 7f83b3050640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T18:37:53.671 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:37:53.668+0000 7f83b3050640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:37:53.671 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:37:53.668+0000 7f83b3050640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.1:6789/0 2024-03-25T18:37:53.671 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:37:53.668+0000 7f83b3050640 1 -- 172.21.15.1:0/3988127934 --> v1:172.21.15.1:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f83ac04f730 con 0x7f83ac0b44b0 2024-03-25T18:37:53.982 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:37:53.981+0000 7f83a97fa640 1 -- 172.21.15.1:0/3988127934 <== mon.0 v1:172.21.15.1:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f8398003610 con 0x7f83ac0b44b0 2024-03-25T18:37:53.982 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:37:53.981+0000 7f83a97fa640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:37:53.982 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:37:53.981+0000 7f83a97fa640 1 -- 172.21.15.1:0/3988127934 <== mon.0 v1:172.21.15.1:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1586965025 0 0) 0x7f83980039a0 con 0x7f83ac0b44b0 2024-03-25T18:37:53.982 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:37:53.981+0000 7f83a97fa640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:37:53.982 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:37:53.981+0000 7f83a97fa640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-25T18:37:53.982 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:37:53.981+0000 7f83a97fa640 10 monclient: dump: 2024-03-25T18:37:53.982 INFO:teuthology.misc.health.smithi001.stderr:epoch 1 2024-03-25T18:37:53.982 INFO:teuthology.misc.health.smithi001.stderr:fsid c8ea72cf-9e99-4383-9b52-6c23d365fdfc 2024-03-25T18:37:53.982 INFO:teuthology.misc.health.smithi001.stderr:last_changed 2024-03-25T18:25:50.510519+0000 2024-03-25T18:37:53.983 INFO:teuthology.misc.health.smithi001.stderr:created 2024-03-25T18:25:50.510519+0000 2024-03-25T18:37:53.983 INFO:teuthology.misc.health.smithi001.stderr:min_mon_release 19 (squid) 2024-03-25T18:37:53.983 INFO:teuthology.misc.health.smithi001.stderr:election_strategy: 3 2024-03-25T18:37:53.983 INFO:teuthology.misc.health.smithi001.stderr:0: v1:172.21.15.1:6789/0 mon.a 2024-03-25T18:37:53.983 INFO:teuthology.misc.health.smithi001.stderr:1: v1:172.21.15.115:6789/0 mon.b 2024-03-25T18:37:53.983 INFO:teuthology.misc.health.smithi001.stderr:2: v1:172.21.15.1:6790/0 mon.c 2024-03-25T18:37:53.983 INFO:teuthology.misc.health.smithi001.stderr: 2024-03-25T18:37:53.983 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:37:53.981+0000 7f83aaffd640 4 set_mon_vals no callback set 2024-03-25T18:37:53.984 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:37:53.982+0000 7f83a97fa640 1 -- 172.21.15.1:0/3988127934 <== mon.0 v1:172.21.15.1:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 21787975 0 0) 0x7f8398003bc0 con 0x7f83ac0b44b0 2024-03-25T18:37:53.984 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:37:53.983+0000 7f83b0dec640 10 monclient: get_auth_request con 0x7f839006df70 auth_method 0 2024-03-25T18:37:53.984 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:37:53.983+0000 7f83a97fa640 10 client.4698.objecter ms_handle_connect 0x7f839006df70 2024-03-25T18:37:53.990 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:37:53.989+0000 7f83a97fa640 1 -- 172.21.15.1:0/3988127934 <== mon.0 v1:172.21.15.1:6789/0 8 ==== osd_map(11..11 src has 1..11) v4 ==== 1645+0+0 (unknown 2504898309 0 0) 0x7f83980535f0 con 0x7f83ac0b44b0 2024-03-25T18:37:53.990 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:37:53.989+0000 7f83a97fa640 10 client.4698.objecter ms_dispatch 0x7f83ac000b90 osd_map(11..11 src has 1..11) v4 2024-03-25T18:37:53.990 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:37:53.989+0000 7f83a97fa640 3 client.4698.objecter handle_osd_map got epochs [11,11] > 0 2024-03-25T18:37:53.990 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:37:53.989+0000 7f83a97fa640 3 client.4698.objecter handle_osd_map decoding full epoch 11 2024-03-25T18:37:53.990 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:37:53.989+0000 7f83a97fa640 20 client.4698.objecter dump_active .. 0 homeless 2024-03-25T18:37:53.990 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:37:53.990+0000 7f83a97fa640 1 -- 172.21.15.1:0/3988127934 <== mon.0 v1:172.21.15.1:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f839804f100 con 0x7f83ac0b44b0 2024-03-25T18:37:53.990 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:37:53.990+0000 7f83a97fa640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:37:53.991 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:37:53.990+0000 7f83a97fa640 10 monclient: _finish_command 1 = system:0 2024-03-25T18:37:54.209 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:37:54.208+0000 7f83b3050640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:37:54.209 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:37:54.208+0000 7f83b3050640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:37:54.209 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:37:54.208+0000 7f83b3050640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.1:6789/0 2024-03-25T18:37:54.209 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:37:54.208+0000 7f83b3050640 1 -- 172.21.15.1:0/3988127934 --> v1:172.21.15.1:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f83ac1d4d60 con 0x7f83ac0b44b0 2024-03-25T18:37:54.234 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:37:54.233+0000 7f83a97fa640 1 -- 172.21.15.1:0/3988127934 <== mon.0 v1:172.21.15.1:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v11) v1 ==== 74+0+7100 (unknown 238102980 0 2126609903) 0x7f8398051030 con 0x7f83ac0b44b0 2024-03-25T18:37:54.234 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:37:54.233+0000 7f83a97fa640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:37:54.234 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:37:54.233+0000 7f83a97fa640 10 monclient: _finish_command 2 = system:0 2024-03-25T18:37:54.234 INFO:teuthology.misc.health.smithi001.stdout: 2024-03-25T18:37:54.234 INFO:teuthology.misc.health.smithi001.stdout:{"epoch":11,"fsid":"c8ea72cf-9e99-4383-9b52-6c23d365fdfc","created":"2024-03-25T18:26:38.856638+0000","modified":"2024-03-25T18:36:43.033551+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T18:36:43.033551+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"b9f34922-3a84-49be-8426-807c1eceaefc","up":0,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["autoout","exists","new"]},{"osd":1,"uuid":"42bf306f-55e3-4634-9874-8b4b73ac6644","up":0,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["autoout","exists","new"]},{"osd":2,"uuid":"292cb23a-18a9-4207-a7cc-4d5727feaec0","up":0,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["autoout","exists","new"]},{"osd":3,"uuid":"45e0e9ab-898d-4673-9590-b4117bc5e84a","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"831a5a09-73ab-4a61-9872-088e76bdbc66","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"88cd903d-3b55-4840-9396-6e3cfe005282","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"89db7355-bc9d-46f4-8ab0-e8d9e744e026","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"cac47bee-0914-4f08-896f-348838882866","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":65536,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":65536,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":65536,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T18:37:54.237 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:37:54.236+0000 7f83b3050640 10 librados: watch_flush enter 2024-03-25T18:37:54.237 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:37:54.236+0000 7f83b3050640 10 librados: watch_flush exit 2024-03-25T18:37:54.237 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:37:54.236+0000 7f83b3050640 20 client.4698.objecter shutdown clearing up homeless session... 2024-03-25T18:37:54.237 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:37:54.236+0000 7f83b3050640 10 client.4698.objecter successfully canceled tick 2024-03-25T18:37:54.237 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:37:54.236+0000 7f83b3050640 1 -- 172.21.15.1:0/3988127934 >> v1:172.21.15.115:6800/4237379176 conn(0x7f839006df70 legacy=0x7f8390070430 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:37:54.237 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:37:54.236+0000 7f83b3050640 10 monclient: shutdown 2024-03-25T18:37:54.237 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:37:54.236+0000 7f83b3050640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:37:54.237 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:37:54.236+0000 7f83b3050640 1 -- 172.21.15.1:0/3988127934 >> v1:172.21.15.1:6789/0 conn(0x7f83ac0b44b0 legacy=0x7f83ac1d8000 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:37:54.237 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:37:54.236+0000 7f83b3050640 1 -- 172.21.15.1:0/3988127934 shutdown_connections 2024-03-25T18:37:54.238 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:37:54.236+0000 7f83b3050640 1 -- 172.21.15.1:0/3988127934 >> 172.21.15.1:0/3988127934 conn(0x7f83ac0617a0 msgr2=0x7f83ac0b2710 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:37:54.238 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:37:54.236+0000 7f83b3050640 1 -- 172.21.15.1:0/3988127934 shutdown_connections 2024-03-25T18:37:54.238 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:37:54.236+0000 7f83b3050640 1 -- 172.21.15.1:0/3988127934 wait complete. 2024-03-25T18:37:54.238 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:37:54.236+0000 7f83b3050640 1 librados: shutdown 2024-03-25T18:37:54.249 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T18:38:01.055 DEBUG:teuthology.orchestra.run.smithi001:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T18:38:01.156 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:38:01.155+0000 7fb1b4058640 10 monclient: get_monmap_and_config 2024-03-25T18:38:01.156 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:38:01.155+0000 7fb1b4058640 10 monclient: build_initial_monmap 2024-03-25T18:38:01.156 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:38:01.155+0000 7fb1b4058640 10 monclient: monmap: 2024-03-25T18:38:01.156 INFO:teuthology.misc.health.smithi001.stderr:epoch 0 2024-03-25T18:38:01.156 INFO:teuthology.misc.health.smithi001.stderr:fsid c8ea72cf-9e99-4383-9b52-6c23d365fdfc 2024-03-25T18:38:01.156 INFO:teuthology.misc.health.smithi001.stderr:last_changed 2024-03-25T18:38:01.156047+0000 2024-03-25T18:38:01.156 INFO:teuthology.misc.health.smithi001.stderr:created 2024-03-25T18:38:01.156047+0000 2024-03-25T18:38:01.156 INFO:teuthology.misc.health.smithi001.stderr:min_mon_release 0 (unknown) 2024-03-25T18:38:01.156 INFO:teuthology.misc.health.smithi001.stderr:election_strategy: 3 2024-03-25T18:38:01.157 INFO:teuthology.misc.health.smithi001.stderr:0: v1:172.21.15.1:6789/0 mon.noname-a 2024-03-25T18:38:01.157 INFO:teuthology.misc.health.smithi001.stderr:1: v1:172.21.15.115:6789/0 mon.noname-c 2024-03-25T18:38:01.157 INFO:teuthology.misc.health.smithi001.stderr:2: v1:172.21.15.1:6790/0 mon.noname-b 2024-03-25T18:38:01.157 INFO:teuthology.misc.health.smithi001.stderr: 2024-03-25T18:38:01.157 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:38:01.156+0000 7fb1b4058640 1 Processor -- start 2024-03-25T18:38:01.158 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:38:01.156+0000 7fb1b4058640 1 -- start start 2024-03-25T18:38:01.158 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:38:01.156+0000 7fb1b4058640 10 monclient: init 2024-03-25T18:38:01.158 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:38:01.156+0000 7fb1b4058640 10 monclient: _reopen_session rank -1 2024-03-25T18:38:01.158 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:38:01.156+0000 7fb1b4058640 10 monclient: _add_conns ranks=[1,2,0] 2024-03-25T18:38:01.158 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:38:01.156+0000 7fb1b4058640 10 monclient(hunting): picked mon.noname-c con 0x7fb1ac0b4e70 addr v1:172.21.15.115:6789/0 2024-03-25T18:38:01.158 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:38:01.156+0000 7fb1b4058640 10 monclient(hunting): picked mon.noname-b con 0x7fb1ac0b44b0 addr v1:172.21.15.1:6790/0 2024-03-25T18:38:01.158 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:38:01.156+0000 7fb1b4058640 10 monclient(hunting): picked mon.noname-a con 0x7fb1ac0b1b00 addr v1:172.21.15.1:6789/0 2024-03-25T18:38:01.158 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:38:01.156+0000 7fb1b4058640 1 -- --> v1:172.21.15.1:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb1ac0a96f0 con 0x7fb1ac0b1b00 2024-03-25T18:38:01.158 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:38:01.156+0000 7fb1b4058640 1 -- --> v1:172.21.15.115:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb1ac0ab0e0 con 0x7fb1ac0b4e70 2024-03-25T18:38:01.159 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:38:01.156+0000 7fb1b4058640 1 -- --> v1:172.21.15.1:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb1ac145f70 con 0x7fb1ac0b44b0 2024-03-25T18:38:01.159 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:38:01.156+0000 7fb1b4058640 10 monclient(hunting): _renew_subs 2024-03-25T18:38:01.159 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:38:01.156+0000 7fb1b4058640 10 monclient(hunting): authenticate will time out at 8161.626565s 2024-03-25T18:38:01.168 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:38:01.168+0000 7fb1b1df4640 1 --1- >> v1:172.21.15.115:6789/0 conn(0x7fb1ac0b4e70 0x7fb1ac13f940 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.115:6789/0 says I am v1:172.21.15.1:60846/0 (socket says 172.21.15.1:60846) 2024-03-25T18:38:01.168 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:38:01.168+0000 7fb1b1df4640 1 -- 172.21.15.1:0/1016539784 learned_addr learned my addr 172.21.15.1:0/1016539784 (peer_addr_for_me v1:172.21.15.1:0/0) 2024-03-25T18:38:01.184 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:38:01.184+0000 7fb1b0df2640 1 -- 172.21.15.1:0/1016539784 <== mon.1 v1:172.21.15.115:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1077664993 0 0) 0x7fb1ac0ab0e0 con 0x7fb1ac0b4e70 2024-03-25T18:38:01.185 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:38:01.184+0000 7fb1b0df2640 10 monclient(hunting): _negotiate 2024-03-25T18:38:01.185 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:38:01.184+0000 7fb1b0df2640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:38:01.185 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:38:01.184+0000 7fb1b0df2640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:38:01.185 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:38:01.184+0000 7fb1b0df2640 10 monclient(hunting): my global_id is 4720 2024-03-25T18:38:01.185 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:38:01.184+0000 7fb1b0df2640 1 -- 172.21.15.1:0/1016539784 --> v1:172.21.15.115:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fb198003520 con 0x7fb1ac0b4e70 2024-03-25T18:38:01.189 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:38:01.188+0000 7fb1b0df2640 1 -- 172.21.15.1:0/1016539784 <== mon.1 v1:172.21.15.115:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1846753671 0 0) 0x7fb198003520 con 0x7fb1ac0b4e70 2024-03-25T18:38:01.189 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:38:01.188+0000 7fb1b0df2640 1 -- 172.21.15.1:0/1016539784 >> v1:172.21.15.1:6790/0 conn(0x7fb1ac0b44b0 legacy=0x7fb1ac145860 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:38:01.189 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:38:01.188+0000 7fb1b0df2640 1 -- 172.21.15.1:0/1016539784 >> v1:172.21.15.1:6789/0 conn(0x7fb1ac0b1b00 legacy=0x7fb1ac14dc00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-25T18:38:01.189 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:38:01.188+0000 7fb1b0df2640 10 monclient: _finish_hunting 0 2024-03-25T18:38:01.189 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:38:01.188+0000 7fb1b0df2640 1 monclient: found mon.noname-c 2024-03-25T18:38:01.189 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:38:01.188+0000 7fb1b0df2640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:38:01.190 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:38:01.188+0000 7fb1b0df2640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.115:6789/0 2024-03-25T18:38:01.190 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:38:01.188+0000 7fb1b0df2640 1 -- 172.21.15.1:0/1016539784 --> v1:172.21.15.115:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb1ac0a9b80 con 0x7fb1ac0b4e70 2024-03-25T18:38:01.190 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:38:01.189+0000 7fb1b0df2640 10 monclient: _finish_auth 0 2024-03-25T18:38:01.190 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:38:01.189+0000 7fb1b0df2640 10 monclient: _check_auth_tickets 2024-03-25T18:38:01.190 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:38:01.189+0000 7fb1b0df2640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:38:01.190 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:38:01.189+0000 7fb1b0df2640 1 -- 172.21.15.1:0/1016539784 <== mon.1 v1:172.21.15.115:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1586965025 0 0) 0x7fb1a8001690 con 0x7fb1ac0b4e70 2024-03-25T18:38:01.190 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:38:01.189+0000 7fb1b0df2640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:38:01.190 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:38:01.189+0000 7fb1b0df2640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-03-25T18:38:01.190 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:38:01.189+0000 7fb1b0df2640 10 monclient: dump: 2024-03-25T18:38:01.190 INFO:teuthology.misc.health.smithi001.stderr:epoch 1 2024-03-25T18:38:01.190 INFO:teuthology.misc.health.smithi001.stderr:fsid c8ea72cf-9e99-4383-9b52-6c23d365fdfc 2024-03-25T18:38:01.190 INFO:teuthology.misc.health.smithi001.stderr:last_changed 2024-03-25T18:25:50.510519+0000 2024-03-25T18:38:01.190 INFO:teuthology.misc.health.smithi001.stderr:created 2024-03-25T18:25:50.510519+0000 2024-03-25T18:38:01.190 INFO:teuthology.misc.health.smithi001.stderr:min_mon_release 19 (squid) 2024-03-25T18:38:01.191 INFO:teuthology.misc.health.smithi001.stderr:election_strategy: 3 2024-03-25T18:38:01.191 INFO:teuthology.misc.health.smithi001.stderr:0: v1:172.21.15.1:6789/0 mon.a 2024-03-25T18:38:01.191 INFO:teuthology.misc.health.smithi001.stderr:1: v1:172.21.15.115:6789/0 mon.b 2024-03-25T18:38:01.191 INFO:teuthology.misc.health.smithi001.stderr:2: v1:172.21.15.1:6790/0 mon.c 2024-03-25T18:38:01.191 INFO:teuthology.misc.health.smithi001.stderr: 2024-03-25T18:38:01.191 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:38:01.189+0000 7fb1b4058640 5 monclient: authenticate success, global_id 4720 2024-03-25T18:38:01.191 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:38:01.189+0000 7fb1b4058640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T18:38:01.213 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:38:01.213+0000 7fb1b0df2640 1 -- 172.21.15.1:0/1016539784 <== mon.1 v1:172.21.15.115:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fb1a8001860 con 0x7fb1ac0b4e70 2024-03-25T18:38:01.213 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:38:01.213+0000 7fb1b0df2640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:38:01.213 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:38:01.213+0000 7fb1b0df2640 1 -- 172.21.15.1:0/1016539784 <== mon.1 v1:172.21.15.115:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1586965025 0 0) 0x7fb1a8001cb0 con 0x7fb1ac0b4e70 2024-03-25T18:38:01.213 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:38:01.213+0000 7fb1b4058640 10 monclient: get_monmap_and_config success 2024-03-25T18:38:01.214 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:38:01.213+0000 7fb1b4058640 4 set_mon_vals no callback set 2024-03-25T18:38:01.214 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:38:01.213+0000 7fb1b0df2640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:38:01.214 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:38:01.213+0000 7fb1b0df2640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-25T18:38:01.214 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:38:01.213+0000 7fb1b0df2640 10 monclient: dump: 2024-03-25T18:38:01.214 INFO:teuthology.misc.health.smithi001.stderr:epoch 1 2024-03-25T18:38:01.214 INFO:teuthology.misc.health.smithi001.stderr:fsid c8ea72cf-9e99-4383-9b52-6c23d365fdfc 2024-03-25T18:38:01.214 INFO:teuthology.misc.health.smithi001.stderr:last_changed 2024-03-25T18:25:50.510519+0000 2024-03-25T18:38:01.214 INFO:teuthology.misc.health.smithi001.stderr:created 2024-03-25T18:25:50.510519+0000 2024-03-25T18:38:01.214 INFO:teuthology.misc.health.smithi001.stderr:min_mon_release 19 (squid) 2024-03-25T18:38:01.214 INFO:teuthology.misc.health.smithi001.stderr:election_strategy: 3 2024-03-25T18:38:01.214 INFO:teuthology.misc.health.smithi001.stderr:0: v1:172.21.15.1:6789/0 mon.a 2024-03-25T18:38:01.214 INFO:teuthology.misc.health.smithi001.stderr:1: v1:172.21.15.115:6789/0 mon.b 2024-03-25T18:38:01.214 INFO:teuthology.misc.health.smithi001.stderr:2: v1:172.21.15.1:6790/0 mon.c 2024-03-25T18:38:01.214 INFO:teuthology.misc.health.smithi001.stderr: 2024-03-25T18:38:01.215 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:38:01.214+0000 7fb1b4058640 10 monclient: shutdown 2024-03-25T18:38:01.215 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:38:01.214+0000 7fb1b4058640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:38:01.215 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:38:01.214+0000 7fb1b4058640 1 -- 172.21.15.1:0/1016539784 >> v1:172.21.15.115:6789/0 conn(0x7fb1ac0b4e70 legacy=0x7fb1ac13f940 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:38:01.215 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:38:01.214+0000 7fb1b4058640 1 -- 172.21.15.1:0/1016539784 shutdown_connections 2024-03-25T18:38:01.215 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:38:01.214+0000 7fb1b4058640 1 -- 172.21.15.1:0/1016539784 >> 172.21.15.1:0/1016539784 conn(0x7fb1ac0617a0 msgr2=0x7fb1ac061bb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:38:01.215 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:38:01.214+0000 7fb1b4058640 1 -- 172.21.15.1:0/1016539784 shutdown_connections 2024-03-25T18:38:01.216 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:38:01.215+0000 7fb1b4058640 1 -- 172.21.15.1:0/1016539784 wait complete. 2024-03-25T18:38:01.217 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:38:01.215+0000 7fb1b4058640 10 monclient: build_initial_monmap 2024-03-25T18:38:01.217 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:38:01.215+0000 7fb1b4058640 10 monclient: monmap: 2024-03-25T18:38:01.217 INFO:teuthology.misc.health.smithi001.stderr:epoch 0 2024-03-25T18:38:01.217 INFO:teuthology.misc.health.smithi001.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T18:38:01.217 INFO:teuthology.misc.health.smithi001.stderr:last_changed 0.000000 2024-03-25T18:38:01.217 INFO:teuthology.misc.health.smithi001.stderr:created 0.000000 2024-03-25T18:38:01.217 INFO:teuthology.misc.health.smithi001.stderr:min_mon_release 0 (unknown) 2024-03-25T18:38:01.217 INFO:teuthology.misc.health.smithi001.stderr:election_strategy: 1 2024-03-25T18:38:01.217 INFO:teuthology.misc.health.smithi001.stderr:0: v1:172.21.15.1:6789/0 mon.noname-a 2024-03-25T18:38:01.218 INFO:teuthology.misc.health.smithi001.stderr:1: v1:172.21.15.115:6789/0 mon.noname-b 2024-03-25T18:38:01.218 INFO:teuthology.misc.health.smithi001.stderr:2: v1:172.21.15.1:6790/0 mon.noname-c 2024-03-25T18:38:01.218 INFO:teuthology.misc.health.smithi001.stderr: 2024-03-25T18:38:01.218 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:38:01.216+0000 7fb1b4058640 1 librados: starting msgr at 2024-03-25T18:38:01.218 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:38:01.216+0000 7fb1b4058640 1 librados: starting objecter 2024-03-25T18:38:01.218 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:38:01.216+0000 7fb1b4058640 1 Processor -- start 2024-03-25T18:38:01.218 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:38:01.216+0000 7fb1b4058640 1 -- start start 2024-03-25T18:38:01.218 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:38:01.216+0000 7fb1b4058640 1 librados: setting wanted keys 2024-03-25T18:38:01.218 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:38:01.216+0000 7fb1b4058640 1 librados: calling monclient init 2024-03-25T18:38:01.218 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:38:01.216+0000 7fb1b4058640 10 monclient: init 2024-03-25T18:38:01.218 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:38:01.216+0000 7fb1b4058640 10 monclient: _reopen_session rank -1 2024-03-25T18:38:01.218 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:38:01.216+0000 7fb1b4058640 10 monclient: _add_conns ranks=[2,1,0] 2024-03-25T18:38:01.218 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:38:01.216+0000 7fb1b4058640 10 monclient(hunting): picked mon.noname-c con 0x7fb1ac0b1b00 addr v1:172.21.15.1:6790/0 2024-03-25T18:38:01.218 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:38:01.216+0000 7fb1b4058640 10 monclient(hunting): picked mon.noname-b con 0x7fb1ac0b44b0 addr v1:172.21.15.115:6789/0 2024-03-25T18:38:01.219 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:38:01.216+0000 7fb1b4058640 10 monclient(hunting): picked mon.noname-a con 0x7fb1ac0b4e70 addr v1:172.21.15.1:6789/0 2024-03-25T18:38:01.219 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:38:01.216+0000 7fb1b4058640 1 -- --> v1:172.21.15.1:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb1ac13c790 con 0x7fb1ac0b4e70 2024-03-25T18:38:01.219 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:38:01.216+0000 7fb1b4058640 1 -- --> v1:172.21.15.115:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb1ac13b220 con 0x7fb1ac0b44b0 2024-03-25T18:38:01.219 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:38:01.216+0000 7fb1b4058640 1 -- --> v1:172.21.15.1:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb1ac145290 con 0x7fb1ac0b1b00 2024-03-25T18:38:01.219 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:38:01.216+0000 7fb1b4058640 10 monclient(hunting): _renew_subs 2024-03-25T18:38:01.219 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:38:01.216+0000 7fb1b4058640 10 monclient(hunting): authenticate will time out at 8161.686574s 2024-03-25T18:38:01.224 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:38:01.223+0000 7fb1b25f5640 1 --1- >> v1:172.21.15.1:6789/0 conn(0x7fb1ac0b4e70 0x7fb1ac143ab0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.1:6789/0 says I am v1:172.21.15.1:40656/0 (socket says 172.21.15.1:40656) 2024-03-25T18:38:01.224 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:38:01.223+0000 7fb1b25f5640 1 -- 172.21.15.1:0/933972791 learned_addr learned my addr 172.21.15.1:0/933972791 (peer_addr_for_me v1:172.21.15.1:0/0) 2024-03-25T18:38:01.226 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:38:01.226+0000 7fb1a27fc640 10 client.?.objecter ms_handle_connect 0x7fb1ac0b4e70 2024-03-25T18:38:01.227 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:38:01.226+0000 7fb1a27fc640 10 client.?.objecter resend_mon_ops 2024-03-25T18:38:01.227 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:38:01.227+0000 7fb1a27fc640 10 client.?.objecter ms_handle_connect 0x7fb1ac0b1b00 2024-03-25T18:38:01.227 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:38:01.227+0000 7fb1a27fc640 10 client.?.objecter resend_mon_ops 2024-03-25T18:38:01.231 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:38:01.230+0000 7fb1a27fc640 10 client.?.objecter ms_handle_connect 0x7fb1ac0b44b0 2024-03-25T18:38:01.231 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:38:01.230+0000 7fb1a27fc640 10 client.?.objecter resend_mon_ops 2024-03-25T18:38:01.231 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:38:01.231+0000 7fb1a27fc640 1 -- 172.21.15.1:0/933972791 <== mon.2 v1:172.21.15.1:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4036536036 0 0) 0x7fb1ac145290 con 0x7fb1ac0b1b00 2024-03-25T18:38:01.231 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:38:01.231+0000 7fb1a27fc640 10 monclient(hunting): _negotiate 2024-03-25T18:38:01.231 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:38:01.231+0000 7fb1a27fc640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:38:01.231 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:38:01.231+0000 7fb1a27fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:38:01.231 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:38:01.231+0000 7fb1a27fc640 10 monclient(hunting): my global_id is 4715 2024-03-25T18:38:01.232 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:38:01.231+0000 7fb1a27fc640 1 -- 172.21.15.1:0/933972791 --> v1:172.21.15.1:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fb190004eb0 con 0x7fb1ac0b1b00 2024-03-25T18:38:01.238 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:38:01.237+0000 7fb1a27fc640 1 -- 172.21.15.1:0/933972791 <== mon.2 v1:172.21.15.1:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 617776957 0 0) 0x7fb190004eb0 con 0x7fb1ac0b1b00 2024-03-25T18:38:01.238 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:38:01.237+0000 7fb1a27fc640 1 -- 172.21.15.1:0/933972791 --> v1:172.21.15.1:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fb1ac145290 con 0x7fb1ac0b1b00 2024-03-25T18:38:01.238 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:38:01.237+0000 7fb1a27fc640 1 -- 172.21.15.1:0/933972791 <== mon.2 v1:172.21.15.1:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1586965025 0 0) 0x7fb1a8001b70 con 0x7fb1ac0b1b00 2024-03-25T18:38:01.238 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:38:01.237+0000 7fb1a27fc640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:38:01.238 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:38:01.237+0000 7fb1a27fc640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-03-25T18:38:01.238 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:38:01.237+0000 7fb1a27fc640 10 monclient(hunting): dump: 2024-03-25T18:38:01.238 INFO:teuthology.misc.health.smithi001.stderr:epoch 1 2024-03-25T18:38:01.238 INFO:teuthology.misc.health.smithi001.stderr:fsid c8ea72cf-9e99-4383-9b52-6c23d365fdfc 2024-03-25T18:38:01.238 INFO:teuthology.misc.health.smithi001.stderr:last_changed 2024-03-25T18:25:50.510519+0000 2024-03-25T18:38:01.238 INFO:teuthology.misc.health.smithi001.stderr:created 2024-03-25T18:25:50.510519+0000 2024-03-25T18:38:01.238 INFO:teuthology.misc.health.smithi001.stderr:min_mon_release 19 (squid) 2024-03-25T18:38:01.238 INFO:teuthology.misc.health.smithi001.stderr:election_strategy: 3 2024-03-25T18:38:01.238 INFO:teuthology.misc.health.smithi001.stderr:0: v1:172.21.15.1:6789/0 mon.a 2024-03-25T18:38:01.239 INFO:teuthology.misc.health.smithi001.stderr:1: v1:172.21.15.115:6789/0 mon.b 2024-03-25T18:38:01.239 INFO:teuthology.misc.health.smithi001.stderr:2: v1:172.21.15.1:6790/0 mon.c 2024-03-25T18:38:01.239 INFO:teuthology.misc.health.smithi001.stderr: 2024-03-25T18:38:01.239 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:38:01.237+0000 7fb1a27fc640 10 monclient(hunting): _finish_auth 0 2024-03-25T18:38:01.244 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:38:01.244+0000 7fb1a27fc640 1 -- 172.21.15.1:0/933972791 <== mon.0 v1:172.21.15.1:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3396955365 0 0) 0x7fb1ac13c790 con 0x7fb1ac0b4e70 2024-03-25T18:38:01.244 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:38:01.244+0000 7fb1a27fc640 10 monclient(hunting): _negotiate 2024-03-25T18:38:01.244 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:38:01.244+0000 7fb1a27fc640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:38:01.244 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:38:01.244+0000 7fb1a27fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:38:01.244 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:38:01.244+0000 7fb1a27fc640 10 monclient(hunting): my global_id is 4701 2024-03-25T18:38:01.244 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:38:01.244+0000 7fb1a27fc640 1 -- 172.21.15.1:0/933972791 --> v1:172.21.15.1:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fb190004eb0 con 0x7fb1ac0b4e70 2024-03-25T18:38:01.248 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:38:01.248+0000 7fb1a27fc640 1 -- 172.21.15.1:0/933972791 <== mon.1 v1:172.21.15.115:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2270301284 0 0) 0x7fb1ac13b220 con 0x7fb1ac0b44b0 2024-03-25T18:38:01.248 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:38:01.248+0000 7fb1a27fc640 10 monclient(hunting): _negotiate 2024-03-25T18:38:01.249 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:38:01.248+0000 7fb1a27fc640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:38:01.249 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:38:01.248+0000 7fb1a27fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:38:01.249 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:38:01.248+0000 7fb1a27fc640 10 monclient(hunting): my global_id is 4723 2024-03-25T18:38:01.249 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:38:01.248+0000 7fb1a27fc640 1 -- 172.21.15.1:0/933972791 --> v1:172.21.15.115:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fb1ac13c790 con 0x7fb1ac0b44b0 2024-03-25T18:38:01.250 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:38:01.249+0000 7fb1a27fc640 1 -- 172.21.15.1:0/933972791 <== mon.2 v1:172.21.15.1:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 3899233167 0 0) 0x7fb1ac145290 con 0x7fb1ac0b1b00 2024-03-25T18:38:01.250 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:38:01.249+0000 7fb1a27fc640 1 -- 172.21.15.1:0/933972791 >> v1:172.21.15.115:6789/0 conn(0x7fb1ac0b44b0 legacy=0x7fb1ac12d0d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:38:01.250 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:38:01.249+0000 7fb1a27fc640 1 -- 172.21.15.1:0/933972791 >> v1:172.21.15.1:6789/0 conn(0x7fb1ac0b4e70 legacy=0x7fb1ac143ab0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:38:01.250 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:38:01.249+0000 7fb1a27fc640 10 monclient: _finish_hunting 0 2024-03-25T18:38:01.250 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:38:01.249+0000 7fb1a27fc640 1 monclient: found mon.c 2024-03-25T18:38:01.250 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:38:01.249+0000 7fb1a27fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:38:01.250 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:38:01.249+0000 7fb1a27fc640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.1:6790/0 2024-03-25T18:38:01.250 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:38:01.249+0000 7fb1a27fc640 1 -- 172.21.15.1:0/933972791 --> v1:172.21.15.1:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb1ac145420 con 0x7fb1ac0b1b00 2024-03-25T18:38:01.250 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:38:01.249+0000 7fb1a27fc640 10 monclient: _finish_auth 0 2024-03-25T18:38:01.250 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:38:01.249+0000 7fb1a27fc640 10 monclient: _check_auth_tickets 2024-03-25T18:38:01.250 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:38:01.249+0000 7fb1a27fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:38:01.251 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:38:01.249+0000 7fb1b4058640 5 monclient: authenticate success, global_id 4715 2024-03-25T18:38:01.251 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:38:01.249+0000 7fb1b4058640 10 monclient: _renew_subs 2024-03-25T18:38:01.251 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:38:01.249+0000 7fb1b4058640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.1:6790/0 2024-03-25T18:38:01.251 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:38:01.249+0000 7fb1b4058640 1 -- 172.21.15.1:0/933972791 --> v1:172.21.15.1:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb1ac13cb40 con 0x7fb1ac0b1b00 2024-03-25T18:38:01.251 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:38:01.249+0000 7fb1b4058640 10 client.4715.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T18:38:01.251 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:38:01.249+0000 7fb1b4058640 10 monclient: _renew_subs 2024-03-25T18:38:01.251 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:38:01.249+0000 7fb1b4058640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.1:6790/0 2024-03-25T18:38:01.251 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:38:01.249+0000 7fb1b4058640 1 -- 172.21.15.1:0/933972791 --> v1:172.21.15.1:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7fb1ac12d7e0 con 0x7fb1ac0b1b00 2024-03-25T18:38:01.251 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:38:01.249+0000 7fb1b4058640 1 librados: init done 2024-03-25T18:38:01.252 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:38:01.250+0000 7fb1b4058640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T18:38:01.252 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:38:01.250+0000 7fb1b4058640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:38:01.252 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:38:01.250+0000 7fb1b4058640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.1:6790/0 2024-03-25T18:38:01.252 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:38:01.250+0000 7fb1b4058640 1 -- 172.21.15.1:0/933972791 --> v1:172.21.15.1:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb1ac13b3b0 con 0x7fb1ac0b1b00 2024-03-25T18:38:01.564 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:38:01.563+0000 7fb1a27fc640 1 -- 172.21.15.1:0/933972791 <== mon.2 v1:172.21.15.1:6790/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fb1a8004a90 con 0x7fb1ac0b1b00 2024-03-25T18:38:01.564 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:38:01.563+0000 7fb1a27fc640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:38:01.564 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:38:01.563+0000 7fb1a27fc640 1 -- 172.21.15.1:0/933972791 <== mon.2 v1:172.21.15.1:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1586965025 0 0) 0x7fb1a8004da0 con 0x7fb1ac0b1b00 2024-03-25T18:38:01.564 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:38:01.563+0000 7fb1a27fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:38:01.564 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:38:01.563+0000 7fb1a27fc640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-25T18:38:01.564 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:38:01.563+0000 7fb1a27fc640 10 monclient: dump: 2024-03-25T18:38:01.564 INFO:teuthology.misc.health.smithi001.stderr:epoch 1 2024-03-25T18:38:01.565 INFO:teuthology.misc.health.smithi001.stderr:fsid c8ea72cf-9e99-4383-9b52-6c23d365fdfc 2024-03-25T18:38:01.565 INFO:teuthology.misc.health.smithi001.stderr:last_changed 2024-03-25T18:25:50.510519+0000 2024-03-25T18:38:01.565 INFO:teuthology.misc.health.smithi001.stderr:created 2024-03-25T18:25:50.510519+0000 2024-03-25T18:38:01.565 INFO:teuthology.misc.health.smithi001.stderr:min_mon_release 19 (squid) 2024-03-25T18:38:01.565 INFO:teuthology.misc.health.smithi001.stderr:election_strategy: 3 2024-03-25T18:38:01.565 INFO:teuthology.misc.health.smithi001.stderr:0: v1:172.21.15.1:6789/0 mon.a 2024-03-25T18:38:01.565 INFO:teuthology.misc.health.smithi001.stderr:1: v1:172.21.15.115:6789/0 mon.b 2024-03-25T18:38:01.565 INFO:teuthology.misc.health.smithi001.stderr:2: v1:172.21.15.1:6790/0 mon.c 2024-03-25T18:38:01.565 INFO:teuthology.misc.health.smithi001.stderr: 2024-03-25T18:38:01.565 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:38:01.563+0000 7fb1a3fff640 4 set_mon_vals no callback set 2024-03-25T18:38:01.566 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:38:01.564+0000 7fb1a27fc640 1 -- 172.21.15.1:0/933972791 <== mon.2 v1:172.21.15.1:6790/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 21787975 0 0) 0x7fb1a801ad70 con 0x7fb1ac0b1b00 2024-03-25T18:38:01.566 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:38:01.565+0000 7fb1b15f3640 10 monclient: get_auth_request con 0x7fb19006ddc0 auth_method 0 2024-03-25T18:38:01.566 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:38:01.565+0000 7fb1a27fc640 10 client.4715.objecter ms_handle_connect 0x7fb19006ddc0 2024-03-25T18:38:01.571 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:38:01.571+0000 7fb1a27fc640 1 -- 172.21.15.1:0/933972791 <== mon.2 v1:172.21.15.1:6790/0 8 ==== osd_map(11..11 src has 1..11) v4 ==== 1645+0+0 (unknown 2504898309 0 0) 0x7fb1a8054880 con 0x7fb1ac0b1b00 2024-03-25T18:38:01.571 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:38:01.571+0000 7fb1a27fc640 10 client.4715.objecter ms_dispatch 0x7fb1ac000b90 osd_map(11..11 src has 1..11) v4 2024-03-25T18:38:01.571 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:38:01.571+0000 7fb1a27fc640 3 client.4715.objecter handle_osd_map got epochs [11,11] > 0 2024-03-25T18:38:01.572 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:38:01.571+0000 7fb1a27fc640 3 client.4715.objecter handle_osd_map decoding full epoch 11 2024-03-25T18:38:01.572 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:38:01.571+0000 7fb1a27fc640 20 client.4715.objecter dump_active .. 0 homeless 2024-03-25T18:38:01.572 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:38:01.571+0000 7fb1a27fc640 1 -- 172.21.15.1:0/933972791 <== mon.2 v1:172.21.15.1:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7fb1a8050390 con 0x7fb1ac0b1b00 2024-03-25T18:38:01.572 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:38:01.571+0000 7fb1a27fc640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:38:01.572 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:38:01.571+0000 7fb1a27fc640 10 monclient: _finish_command 1 = system:0 2024-03-25T18:38:01.823 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:38:01.822+0000 7fb1b4058640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:38:01.823 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:38:01.822+0000 7fb1b4058640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:38:01.823 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:38:01.822+0000 7fb1b4058640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.1:6790/0 2024-03-25T18:38:01.823 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:38:01.822+0000 7fb1b4058640 1 -- 172.21.15.1:0/933972791 --> v1:172.21.15.1:6790/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fb1ac04f730 con 0x7fb1ac0b1b00 2024-03-25T18:38:01.842 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:38:01.842+0000 7fb1a27fc640 1 -- 172.21.15.1:0/933972791 <== mon.2 v1:172.21.15.1:6790/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v11) v1 ==== 74+0+7100 (unknown 238102980 0 2126609903) 0x7fb1a80522c0 con 0x7fb1ac0b1b00 2024-03-25T18:38:01.842 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:38:01.842+0000 7fb1a27fc640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:38:01.842 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:38:01.842+0000 7fb1a27fc640 10 monclient: _finish_command 2 = system:0 2024-03-25T18:38:01.843 INFO:teuthology.misc.health.smithi001.stdout: 2024-03-25T18:38:01.843 INFO:teuthology.misc.health.smithi001.stdout:{"epoch":11,"fsid":"c8ea72cf-9e99-4383-9b52-6c23d365fdfc","created":"2024-03-25T18:26:38.856638+0000","modified":"2024-03-25T18:36:43.033551+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T18:36:43.033551+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"b9f34922-3a84-49be-8426-807c1eceaefc","up":0,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["autoout","exists","new"]},{"osd":1,"uuid":"42bf306f-55e3-4634-9874-8b4b73ac6644","up":0,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["autoout","exists","new"]},{"osd":2,"uuid":"292cb23a-18a9-4207-a7cc-4d5727feaec0","up":0,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["autoout","exists","new"]},{"osd":3,"uuid":"45e0e9ab-898d-4673-9590-b4117bc5e84a","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"831a5a09-73ab-4a61-9872-088e76bdbc66","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"88cd903d-3b55-4840-9396-6e3cfe005282","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"89db7355-bc9d-46f4-8ab0-e8d9e744e026","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"cac47bee-0914-4f08-896f-348838882866","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":65536,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":65536,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":65536,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T18:38:01.844 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:38:01.844+0000 7fb1b4058640 10 librados: watch_flush enter 2024-03-25T18:38:01.845 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:38:01.844+0000 7fb1b4058640 10 librados: watch_flush exit 2024-03-25T18:38:01.845 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:38:01.844+0000 7fb1b4058640 20 client.4715.objecter shutdown clearing up homeless session... 2024-03-25T18:38:01.845 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:38:01.844+0000 7fb1b4058640 10 client.4715.objecter successfully canceled tick 2024-03-25T18:38:01.845 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:38:01.844+0000 7fb1b4058640 1 -- 172.21.15.1:0/933972791 >> v1:172.21.15.115:6800/4237379176 conn(0x7fb19006ddc0 legacy=0x7fb190070280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:38:01.845 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:38:01.844+0000 7fb1b4058640 10 monclient: shutdown 2024-03-25T18:38:01.845 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:38:01.844+0000 7fb1b4058640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:38:01.845 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:38:01.844+0000 7fb1b4058640 1 -- 172.21.15.1:0/933972791 >> v1:172.21.15.1:6790/0 conn(0x7fb1ac0b1b00 legacy=0x7fb1ac13a9e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:38:01.845 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:38:01.844+0000 7fb1b4058640 1 -- 172.21.15.1:0/933972791 shutdown_connections 2024-03-25T18:38:01.845 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:38:01.844+0000 7fb1b4058640 1 -- 172.21.15.1:0/933972791 >> 172.21.15.1:0/933972791 conn(0x7fb1ac0617a0 msgr2=0x7fb1ac0b2710 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:38:01.845 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:38:01.844+0000 7fb1b4058640 1 -- 172.21.15.1:0/933972791 shutdown_connections 2024-03-25T18:38:01.845 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:38:01.844+0000 7fb1b4058640 1 -- 172.21.15.1:0/933972791 wait complete. 2024-03-25T18:38:01.845 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:38:01.844+0000 7fb1b4058640 1 librados: shutdown 2024-03-25T18:38:01.855 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T18:38:08.458 INFO:tasks.ceph.mon.a.smithi001.stderr:==00:00:11:47.017 34876== Warning: unimplemented fcntl command: 1036 2024-03-25T18:38:08.471 INFO:tasks.ceph.mon.a.smithi001.stderr:==00:00:11:47.030 34876== Warning: unimplemented fcntl command: 1036 2024-03-25T18:38:08.506 INFO:tasks.ceph.mon.a.smithi001.stderr:==00:00:11:47.064 34876== Warning: unimplemented fcntl command: 1036 2024-03-25T18:38:08.541 INFO:tasks.ceph.mon.b.smithi115.stderr:==00:00:11:47.024 34836== Warning: unimplemented fcntl command: 1036 2024-03-25T18:38:08.545 INFO:tasks.ceph.mon.b.smithi115.stderr:==00:00:11:47.029 34836== Warning: unimplemented fcntl command: 1036 2024-03-25T18:38:08.545 INFO:tasks.ceph.mon.c.smithi001.stderr:==00:00:11:47.104 34877== Warning: unimplemented fcntl command: 1036 2024-03-25T18:38:08.555 INFO:tasks.ceph.mon.c.smithi001.stderr:==00:00:11:47.114 34877== Warning: unimplemented fcntl command: 1036 2024-03-25T18:38:08.581 INFO:tasks.ceph.mon.b.smithi115.stderr:==00:00:11:47.065 34836== Warning: unimplemented fcntl command: 1036 2024-03-25T18:38:08.584 INFO:tasks.ceph.mon.c.smithi001.stderr:==00:00:11:47.143 34877== Warning: unimplemented fcntl command: 1036 2024-03-25T18:38:08.661 DEBUG:teuthology.orchestra.run.smithi001:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T18:38:08.779 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:38:08.778+0000 7fc63c805640 10 monclient: get_monmap_and_config 2024-03-25T18:38:08.779 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:38:08.778+0000 7fc63c805640 10 monclient: build_initial_monmap 2024-03-25T18:38:08.779 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:38:08.778+0000 7fc63c805640 10 monclient: monmap: 2024-03-25T18:38:08.779 INFO:teuthology.misc.health.smithi001.stderr:epoch 0 2024-03-25T18:38:08.779 INFO:teuthology.misc.health.smithi001.stderr:fsid c8ea72cf-9e99-4383-9b52-6c23d365fdfc 2024-03-25T18:38:08.779 INFO:teuthology.misc.health.smithi001.stderr:last_changed 2024-03-25T18:38:08.778733+0000 2024-03-25T18:38:08.779 INFO:teuthology.misc.health.smithi001.stderr:created 2024-03-25T18:38:08.778733+0000 2024-03-25T18:38:08.779 INFO:teuthology.misc.health.smithi001.stderr:min_mon_release 0 (unknown) 2024-03-25T18:38:08.779 INFO:teuthology.misc.health.smithi001.stderr:election_strategy: 3 2024-03-25T18:38:08.779 INFO:teuthology.misc.health.smithi001.stderr:0: v1:172.21.15.1:6789/0 mon.noname-a 2024-03-25T18:38:08.780 INFO:teuthology.misc.health.smithi001.stderr:1: v1:172.21.15.115:6789/0 mon.noname-c 2024-03-25T18:38:08.780 INFO:teuthology.misc.health.smithi001.stderr:2: v1:172.21.15.1:6790/0 mon.noname-b 2024-03-25T18:38:08.780 INFO:teuthology.misc.health.smithi001.stderr: 2024-03-25T18:38:08.780 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:38:08.778+0000 7fc63c805640 1 Processor -- start 2024-03-25T18:38:08.780 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:38:08.778+0000 7fc63c805640 1 -- start start 2024-03-25T18:38:08.780 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:38:08.778+0000 7fc63c805640 10 monclient: init 2024-03-25T18:38:08.780 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:38:08.779+0000 7fc63c805640 10 monclient: _reopen_session rank -1 2024-03-25T18:38:08.781 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:38:08.779+0000 7fc63c805640 10 monclient: _add_conns ranks=[2,1,0] 2024-03-25T18:38:08.781 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:38:08.779+0000 7fc63c805640 10 monclient(hunting): picked mon.noname-b con 0x7fc6380b4ff0 addr v1:172.21.15.1:6790/0 2024-03-25T18:38:08.781 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:38:08.779+0000 7fc63c805640 10 monclient(hunting): picked mon.noname-c con 0x7fc6380b4670 addr v1:172.21.15.115:6789/0 2024-03-25T18:38:08.781 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:38:08.779+0000 7fc63c805640 10 monclient(hunting): picked mon.noname-a con 0x7fc63813c480 addr v1:172.21.15.1:6789/0 2024-03-25T18:38:08.781 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:38:08.779+0000 7fc63c805640 1 -- --> v1:172.21.15.1:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc6380ab0e0 con 0x7fc63813c480 2024-03-25T18:38:08.781 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:38:08.779+0000 7fc63c805640 1 -- --> v1:172.21.15.115:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc6380ab3c0 con 0x7fc6380b4670 2024-03-25T18:38:08.781 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:38:08.779+0000 7fc63c805640 1 -- --> v1:172.21.15.1:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc6380a9f30 con 0x7fc6380b4ff0 2024-03-25T18:38:08.781 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:38:08.779+0000 7fc63c805640 10 monclient(hunting): _renew_subs 2024-03-25T18:38:08.781 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:38:08.779+0000 7fc63c805640 10 monclient(hunting): authenticate will time out at 8169.249533s 2024-03-25T18:38:08.785 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:38:08.785+0000 7fc635d9b640 1 --1- >> v1:172.21.15.115:6789/0 conn(0x7fc6380b4670 0x7fc6381425c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.115:6789/0 says I am v1:172.21.15.1:41654/0 (socket says 172.21.15.1:41654) 2024-03-25T18:38:08.786 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:38:08.785+0000 7fc635d9b640 1 -- 172.21.15.1:0/2039141982 learned_addr learned my addr 172.21.15.1:0/2039141982 (peer_addr_for_me v1:172.21.15.1:0/0) 2024-03-25T18:38:08.798 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:38:08.797+0000 7fc63559a640 1 -- 172.21.15.1:0/2039141982 <== mon.1 v1:172.21.15.115:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2797460151 0 0) 0x7fc6380ab3c0 con 0x7fc6380b4670 2024-03-25T18:38:08.798 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:38:08.797+0000 7fc63559a640 10 monclient(hunting): _negotiate 2024-03-25T18:38:08.798 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:38:08.797+0000 7fc63559a640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:38:08.798 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:38:08.797+0000 7fc63559a640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:38:08.798 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:38:08.797+0000 7fc63559a640 10 monclient(hunting): my global_id is 4726 2024-03-25T18:38:08.798 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:38:08.797+0000 7fc63559a640 1 -- 172.21.15.1:0/2039141982 --> v1:172.21.15.115:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fc620003520 con 0x7fc6380b4670 2024-03-25T18:38:08.805 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:38:08.804+0000 7fc63559a640 1 -- 172.21.15.1:0/2039141982 <== mon.2 v1:172.21.15.1:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1978985003 0 0) 0x7fc6380a9f30 con 0x7fc6380b4ff0 2024-03-25T18:38:08.806 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:38:08.804+0000 7fc63559a640 10 monclient(hunting): _negotiate 2024-03-25T18:38:08.806 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:38:08.804+0000 7fc63559a640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:38:08.806 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:38:08.804+0000 7fc63559a640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:38:08.806 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:38:08.804+0000 7fc63559a640 10 monclient(hunting): my global_id is 4718 2024-03-25T18:38:08.806 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:38:08.804+0000 7fc63559a640 1 -- 172.21.15.1:0/2039141982 --> v1:172.21.15.1:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fc6380ab3c0 con 0x7fc6380b4ff0 2024-03-25T18:38:08.806 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:38:08.804+0000 7fc63559a640 1 -- 172.21.15.1:0/2039141982 <== mon.1 v1:172.21.15.115:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 4011480275 0 0) 0x7fc620003520 con 0x7fc6380b4670 2024-03-25T18:38:08.806 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:38:08.804+0000 7fc63559a640 1 -- 172.21.15.1:0/2039141982 >> v1:172.21.15.1:6790/0 conn(0x7fc6380b4ff0 legacy=0x7fc63813bd70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:38:08.807 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:38:08.804+0000 7fc63559a640 1 -- 172.21.15.1:0/2039141982 >> v1:172.21.15.1:6789/0 conn(0x7fc63813c480 legacy=0x7fc63814dc00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:38:08.807 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:38:08.804+0000 7fc63559a640 10 monclient: _finish_hunting 0 2024-03-25T18:38:08.807 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:38:08.804+0000 7fc63559a640 1 monclient: found mon.noname-c 2024-03-25T18:38:08.807 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:38:08.804+0000 7fc63559a640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:38:08.807 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:38:08.804+0000 7fc63559a640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.115:6789/0 2024-03-25T18:38:08.807 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:38:08.804+0000 7fc63559a640 1 -- 172.21.15.1:0/2039141982 --> v1:172.21.15.115:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc6380a9b70 con 0x7fc6380b4670 2024-03-25T18:38:08.807 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:38:08.804+0000 7fc63559a640 10 monclient: _finish_auth 0 2024-03-25T18:38:08.807 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:38:08.804+0000 7fc63559a640 10 monclient: _check_auth_tickets 2024-03-25T18:38:08.807 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:38:08.804+0000 7fc63559a640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:38:08.807 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:38:08.804+0000 7fc63559a640 1 -- 172.21.15.1:0/2039141982 <== mon.1 v1:172.21.15.115:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1586965025 0 0) 0x7fc62c001690 con 0x7fc6380b4670 2024-03-25T18:38:08.807 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:38:08.804+0000 7fc63559a640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:38:08.807 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:38:08.804+0000 7fc63559a640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-03-25T18:38:08.807 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:38:08.804+0000 7fc63559a640 10 monclient: dump: 2024-03-25T18:38:08.807 INFO:teuthology.misc.health.smithi001.stderr:epoch 1 2024-03-25T18:38:08.807 INFO:teuthology.misc.health.smithi001.stderr:fsid c8ea72cf-9e99-4383-9b52-6c23d365fdfc 2024-03-25T18:38:08.807 INFO:teuthology.misc.health.smithi001.stderr:last_changed 2024-03-25T18:25:50.510519+0000 2024-03-25T18:38:08.807 INFO:teuthology.misc.health.smithi001.stderr:created 2024-03-25T18:25:50.510519+0000 2024-03-25T18:38:08.808 INFO:teuthology.misc.health.smithi001.stderr:min_mon_release 19 (squid) 2024-03-25T18:38:08.808 INFO:teuthology.misc.health.smithi001.stderr:election_strategy: 3 2024-03-25T18:38:08.808 INFO:teuthology.misc.health.smithi001.stderr:0: v1:172.21.15.1:6789/0 mon.a 2024-03-25T18:38:08.808 INFO:teuthology.misc.health.smithi001.stderr:1: v1:172.21.15.115:6789/0 mon.b 2024-03-25T18:38:08.808 INFO:teuthology.misc.health.smithi001.stderr:2: v1:172.21.15.1:6790/0 mon.c 2024-03-25T18:38:08.808 INFO:teuthology.misc.health.smithi001.stderr: 2024-03-25T18:38:08.808 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:38:08.804+0000 7fc63c805640 5 monclient: authenticate success, global_id 4726 2024-03-25T18:38:08.808 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:38:08.805+0000 7fc63c805640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T18:38:08.817 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:38:08.817+0000 7fc63559a640 1 -- 172.21.15.1:0/2039141982 <== mon.1 v1:172.21.15.115:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fc62c0017f0 con 0x7fc6380b4670 2024-03-25T18:38:08.817 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:38:08.817+0000 7fc63559a640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:38:08.817 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:38:08.817+0000 7fc63559a640 1 -- 172.21.15.1:0/2039141982 <== mon.1 v1:172.21.15.115:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1586965025 0 0) 0x7fc62c001c00 con 0x7fc6380b4670 2024-03-25T18:38:08.817 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:38:08.817+0000 7fc63559a640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:38:08.817 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:38:08.817+0000 7fc63559a640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-25T18:38:08.817 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:38:08.817+0000 7fc63559a640 10 monclient: dump: 2024-03-25T18:38:08.817 INFO:teuthology.misc.health.smithi001.stderr:epoch 1 2024-03-25T18:38:08.817 INFO:teuthology.misc.health.smithi001.stderr:fsid c8ea72cf-9e99-4383-9b52-6c23d365fdfc 2024-03-25T18:38:08.817 INFO:teuthology.misc.health.smithi001.stderr:last_changed 2024-03-25T18:25:50.510519+0000 2024-03-25T18:38:08.818 INFO:teuthology.misc.health.smithi001.stderr:created 2024-03-25T18:25:50.510519+0000 2024-03-25T18:38:08.818 INFO:teuthology.misc.health.smithi001.stderr:min_mon_release 19 (squid) 2024-03-25T18:38:08.818 INFO:teuthology.misc.health.smithi001.stderr:election_strategy: 3 2024-03-25T18:38:08.818 INFO:teuthology.misc.health.smithi001.stderr:0: v1:172.21.15.1:6789/0 mon.a 2024-03-25T18:38:08.818 INFO:teuthology.misc.health.smithi001.stderr:1: v1:172.21.15.115:6789/0 mon.b 2024-03-25T18:38:08.818 INFO:teuthology.misc.health.smithi001.stderr:2: v1:172.21.15.1:6790/0 mon.c 2024-03-25T18:38:08.818 INFO:teuthology.misc.health.smithi001.stderr: 2024-03-25T18:38:08.818 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:38:08.817+0000 7fc63c805640 10 monclient: get_monmap_and_config success 2024-03-25T18:38:08.818 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:38:08.817+0000 7fc63c805640 4 set_mon_vals no callback set 2024-03-25T18:38:08.820 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:38:08.817+0000 7fc63c805640 10 monclient: shutdown 2024-03-25T18:38:08.820 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:38:08.817+0000 7fc63c805640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:38:08.821 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:38:08.817+0000 7fc63c805640 1 -- 172.21.15.1:0/2039141982 >> v1:172.21.15.115:6789/0 conn(0x7fc6380b4670 legacy=0x7fc6381425c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:38:08.821 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:38:08.818+0000 7fc63c805640 1 -- 172.21.15.1:0/2039141982 shutdown_connections 2024-03-25T18:38:08.821 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:38:08.818+0000 7fc63c805640 1 -- 172.21.15.1:0/2039141982 >> 172.21.15.1:0/2039141982 conn(0x7fc6380617a0 msgr2=0x7fc638061bb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:38:08.821 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:38:08.818+0000 7fc63c805640 1 -- 172.21.15.1:0/2039141982 shutdown_connections 2024-03-25T18:38:08.821 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:38:08.818+0000 7fc63c805640 1 -- 172.21.15.1:0/2039141982 wait complete. 2024-03-25T18:38:08.821 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:38:08.819+0000 7fc63c805640 10 monclient: build_initial_monmap 2024-03-25T18:38:08.821 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:38:08.819+0000 7fc63c805640 10 monclient: monmap: 2024-03-25T18:38:08.821 INFO:teuthology.misc.health.smithi001.stderr:epoch 0 2024-03-25T18:38:08.821 INFO:teuthology.misc.health.smithi001.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T18:38:08.821 INFO:teuthology.misc.health.smithi001.stderr:last_changed 0.000000 2024-03-25T18:38:08.821 INFO:teuthology.misc.health.smithi001.stderr:created 0.000000 2024-03-25T18:38:08.821 INFO:teuthology.misc.health.smithi001.stderr:min_mon_release 0 (unknown) 2024-03-25T18:38:08.821 INFO:teuthology.misc.health.smithi001.stderr:election_strategy: 1 2024-03-25T18:38:08.821 INFO:teuthology.misc.health.smithi001.stderr:0: v1:172.21.15.1:6789/0 mon.noname-a 2024-03-25T18:38:08.822 INFO:teuthology.misc.health.smithi001.stderr:1: v1:172.21.15.115:6789/0 mon.noname-b 2024-03-25T18:38:08.822 INFO:teuthology.misc.health.smithi001.stderr:2: v1:172.21.15.1:6790/0 mon.noname-c 2024-03-25T18:38:08.822 INFO:teuthology.misc.health.smithi001.stderr: 2024-03-25T18:38:08.822 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:38:08.819+0000 7fc63c805640 1 librados: starting msgr at 2024-03-25T18:38:08.822 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:38:08.819+0000 7fc63c805640 1 librados: starting objecter 2024-03-25T18:38:08.822 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:38:08.819+0000 7fc63c805640 1 Processor -- start 2024-03-25T18:38:08.822 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:38:08.819+0000 7fc63c805640 1 -- start start 2024-03-25T18:38:08.822 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:38:08.819+0000 7fc63c805640 1 librados: setting wanted keys 2024-03-25T18:38:08.822 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:38:08.819+0000 7fc63c805640 1 librados: calling monclient init 2024-03-25T18:38:08.822 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:38:08.819+0000 7fc63c805640 10 monclient: init 2024-03-25T18:38:08.822 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:38:08.820+0000 7fc63c805640 10 monclient: _reopen_session rank -1 2024-03-25T18:38:08.822 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:38:08.820+0000 7fc63c805640 10 monclient: _add_conns ranks=[2,1,0] 2024-03-25T18:38:08.822 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:38:08.820+0000 7fc63c805640 10 monclient(hunting): picked mon.noname-c con 0x7fc6380b4670 addr v1:172.21.15.1:6790/0 2024-03-25T18:38:08.822 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:38:08.820+0000 7fc63c805640 10 monclient(hunting): picked mon.noname-b con 0x7fc6380b4ff0 addr v1:172.21.15.115:6789/0 2024-03-25T18:38:08.823 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:38:08.820+0000 7fc63c805640 10 monclient(hunting): picked mon.noname-a con 0x7fc63813c480 addr v1:172.21.15.1:6789/0 2024-03-25T18:38:08.823 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:38:08.820+0000 7fc63c805640 1 -- --> v1:172.21.15.1:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc6380b0040 con 0x7fc63813c480 2024-03-25T18:38:08.823 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:38:08.820+0000 7fc63c805640 1 -- --> v1:172.21.15.115:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc6380b0320 con 0x7fc6380b4ff0 2024-03-25T18:38:08.823 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:38:08.820+0000 7fc63c805640 1 -- --> v1:172.21.15.1:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc6380b0600 con 0x7fc6380b4670 2024-03-25T18:38:08.823 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:38:08.820+0000 7fc63c805640 10 monclient(hunting): _renew_subs 2024-03-25T18:38:08.823 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:38:08.820+0000 7fc63c805640 10 monclient(hunting): authenticate will time out at 8169.289887s 2024-03-25T18:38:08.842 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:38:08.842+0000 7fc636d9d640 1 --1- >> v1:172.21.15.1:6789/0 conn(0x7fc63813c480 0x7fc63813a940 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.1:6789/0 says I am v1:172.21.15.1:43516/0 (socket says 172.21.15.1:43516) 2024-03-25T18:38:08.842 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:38:08.842+0000 7fc636d9d640 1 -- 172.21.15.1:0/1572074285 learned_addr learned my addr 172.21.15.1:0/1572074285 (peer_addr_for_me v1:172.21.15.1:0/0) 2024-03-25T18:38:08.843 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:38:08.843+0000 7fc626ffd640 10 client.?.objecter ms_handle_connect 0x7fc63813c480 2024-03-25T18:38:08.843 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:38:08.843+0000 7fc626ffd640 10 client.?.objecter resend_mon_ops 2024-03-25T18:38:08.850 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:38:08.850+0000 7fc626ffd640 10 client.?.objecter ms_handle_connect 0x7fc6380b4ff0 2024-03-25T18:38:08.850 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:38:08.850+0000 7fc626ffd640 10 client.?.objecter resend_mon_ops 2024-03-25T18:38:08.853 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:38:08.852+0000 7fc626ffd640 1 -- 172.21.15.1:0/1572074285 <== mon.0 v1:172.21.15.1:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 722012796 0 0) 0x7fc6380b0040 con 0x7fc63813c480 2024-03-25T18:38:08.853 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:38:08.852+0000 7fc626ffd640 10 monclient(hunting): _negotiate 2024-03-25T18:38:08.853 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:38:08.852+0000 7fc626ffd640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:38:08.853 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:38:08.852+0000 7fc626ffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:38:08.853 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:38:08.852+0000 7fc626ffd640 10 monclient(hunting): my global_id is 4707 2024-03-25T18:38:08.854 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:38:08.852+0000 7fc626ffd640 1 -- 172.21.15.1:0/1572074285 --> v1:172.21.15.1:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fc618004eb0 con 0x7fc63813c480 2024-03-25T18:38:08.856 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:38:08.856+0000 7fc626ffd640 1 -- 172.21.15.1:0/1572074285 <== mon.1 v1:172.21.15.115:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2152389689 0 0) 0x7fc6380b0320 con 0x7fc6380b4ff0 2024-03-25T18:38:08.856 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:38:08.856+0000 7fc626ffd640 10 monclient(hunting): _negotiate 2024-03-25T18:38:08.856 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:38:08.856+0000 7fc626ffd640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:38:08.856 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:38:08.856+0000 7fc626ffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:38:08.856 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:38:08.856+0000 7fc626ffd640 10 monclient(hunting): my global_id is 4729 2024-03-25T18:38:08.857 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:38:08.856+0000 7fc626ffd640 1 -- 172.21.15.1:0/1572074285 --> v1:172.21.15.115:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fc6380b0040 con 0x7fc6380b4ff0 2024-03-25T18:38:08.865 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:38:08.865+0000 7fc626ffd640 1 -- 172.21.15.1:0/1572074285 <== mon.1 v1:172.21.15.115:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3776017000 0 0) 0x7fc6380b0040 con 0x7fc6380b4ff0 2024-03-25T18:38:08.865 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:38:08.865+0000 7fc626ffd640 1 -- 172.21.15.1:0/1572074285 --> v1:172.21.15.115:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fc6380b0320 con 0x7fc6380b4ff0 2024-03-25T18:38:08.865 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:38:08.865+0000 7fc626ffd640 1 -- 172.21.15.1:0/1572074285 <== mon.1 v1:172.21.15.115:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1586965025 0 0) 0x7fc62c001b00 con 0x7fc6380b4ff0 2024-03-25T18:38:08.865 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:38:08.865+0000 7fc626ffd640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:38:08.865 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:38:08.865+0000 7fc626ffd640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-03-25T18:38:08.865 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:38:08.865+0000 7fc626ffd640 10 monclient(hunting): dump: 2024-03-25T18:38:08.865 INFO:teuthology.misc.health.smithi001.stderr:epoch 1 2024-03-25T18:38:08.866 INFO:teuthology.misc.health.smithi001.stderr:fsid c8ea72cf-9e99-4383-9b52-6c23d365fdfc 2024-03-25T18:38:08.866 INFO:teuthology.misc.health.smithi001.stderr:last_changed 2024-03-25T18:25:50.510519+0000 2024-03-25T18:38:08.866 INFO:teuthology.misc.health.smithi001.stderr:created 2024-03-25T18:25:50.510519+0000 2024-03-25T18:38:08.866 INFO:teuthology.misc.health.smithi001.stderr:min_mon_release 19 (squid) 2024-03-25T18:38:08.866 INFO:teuthology.misc.health.smithi001.stderr:election_strategy: 3 2024-03-25T18:38:08.866 INFO:teuthology.misc.health.smithi001.stderr:0: v1:172.21.15.1:6789/0 mon.a 2024-03-25T18:38:08.866 INFO:teuthology.misc.health.smithi001.stderr:1: v1:172.21.15.115:6789/0 mon.b 2024-03-25T18:38:08.866 INFO:teuthology.misc.health.smithi001.stderr:2: v1:172.21.15.1:6790/0 mon.c 2024-03-25T18:38:08.866 INFO:teuthology.misc.health.smithi001.stderr: 2024-03-25T18:38:08.866 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:38:08.865+0000 7fc626ffd640 10 monclient(hunting): _finish_auth 0 2024-03-25T18:38:08.874 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:38:08.872+0000 7fc626ffd640 1 -- 172.21.15.1:0/1572074285 <== mon.1 v1:172.21.15.115:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 3015774287 0 0) 0x7fc6380b0320 con 0x7fc6380b4ff0 2024-03-25T18:38:08.874 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:38:08.872+0000 7fc626ffd640 1 -- 172.21.15.1:0/1572074285 >> v1:172.21.15.1:6790/0 conn(0x7fc6380b4670 legacy=0x7fc6380ae210 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:38:08.874 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:38:08.872+0000 7fc626ffd640 1 -- 172.21.15.1:0/1572074285 >> v1:172.21.15.1:6789/0 conn(0x7fc63813c480 legacy=0x7fc63813a940 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:38:08.874 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:38:08.872+0000 7fc626ffd640 10 monclient: _finish_hunting 0 2024-03-25T18:38:08.874 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:38:08.872+0000 7fc626ffd640 1 monclient: found mon.b 2024-03-25T18:38:08.874 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:38:08.872+0000 7fc626ffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:38:08.874 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:38:08.873+0000 7fc626ffd640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.115:6789/0 2024-03-25T18:38:08.874 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:38:08.873+0000 7fc626ffd640 1 -- 172.21.15.1:0/1572074285 --> v1:172.21.15.115:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc6380af210 con 0x7fc6380b4ff0 2024-03-25T18:38:08.874 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:38:08.873+0000 7fc626ffd640 10 monclient: _finish_auth 0 2024-03-25T18:38:08.875 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:38:08.873+0000 7fc626ffd640 10 monclient: _check_auth_tickets 2024-03-25T18:38:08.875 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:38:08.873+0000 7fc626ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:38:08.875 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:38:08.873+0000 7fc63c805640 5 monclient: authenticate success, global_id 4729 2024-03-25T18:38:08.875 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:38:08.873+0000 7fc63c805640 10 monclient: _renew_subs 2024-03-25T18:38:08.875 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:38:08.873+0000 7fc63c805640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.115:6789/0 2024-03-25T18:38:08.875 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:38:08.873+0000 7fc63c805640 1 -- 172.21.15.1:0/1572074285 --> v1:172.21.15.115:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc6380aee90 con 0x7fc6380b4ff0 2024-03-25T18:38:08.875 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:38:08.873+0000 7fc63c805640 10 client.4729.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T18:38:08.875 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:38:08.873+0000 7fc63c805640 10 monclient: _renew_subs 2024-03-25T18:38:08.875 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:38:08.873+0000 7fc63c805640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.115:6789/0 2024-03-25T18:38:08.875 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:38:08.873+0000 7fc63c805640 1 -- 172.21.15.1:0/1572074285 --> v1:172.21.15.115:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7fc6381e3570 con 0x7fc6380b4ff0 2024-03-25T18:38:08.875 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:38:08.873+0000 7fc63c805640 1 librados: init done 2024-03-25T18:38:08.876 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:38:08.874+0000 7fc63c805640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T18:38:08.876 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:38:08.874+0000 7fc63c805640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:38:08.876 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:38:08.874+0000 7fc63c805640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.115:6789/0 2024-03-25T18:38:08.876 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:38:08.874+0000 7fc63c805640 1 -- 172.21.15.1:0/1572074285 --> v1:172.21.15.115:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc6380af080 con 0x7fc6380b4ff0 2024-03-25T18:38:09.155 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:38:09.154+0000 7fc626ffd640 1 -- 172.21.15.1:0/1572074285 <== mon.1 v1:172.21.15.115:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fc62c004a90 con 0x7fc6380b4ff0 2024-03-25T18:38:09.155 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:38:09.154+0000 7fc626ffd640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:38:09.155 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:38:09.154+0000 7fc626ffd640 1 -- 172.21.15.1:0/1572074285 <== mon.1 v1:172.21.15.115:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1586965025 0 0) 0x7fc62c004da0 con 0x7fc6380b4ff0 2024-03-25T18:38:09.155 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:38:09.154+0000 7fc626ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:38:09.155 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:38:09.154+0000 7fc626ffd640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-25T18:38:09.155 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:38:09.154+0000 7fc626ffd640 10 monclient: dump: 2024-03-25T18:38:09.155 INFO:teuthology.misc.health.smithi001.stderr:epoch 1 2024-03-25T18:38:09.156 INFO:teuthology.misc.health.smithi001.stderr:fsid c8ea72cf-9e99-4383-9b52-6c23d365fdfc 2024-03-25T18:38:09.156 INFO:teuthology.misc.health.smithi001.stderr:last_changed 2024-03-25T18:25:50.510519+0000 2024-03-25T18:38:09.156 INFO:teuthology.misc.health.smithi001.stderr:created 2024-03-25T18:25:50.510519+0000 2024-03-25T18:38:09.156 INFO:teuthology.misc.health.smithi001.stderr:min_mon_release 19 (squid) 2024-03-25T18:38:09.156 INFO:teuthology.misc.health.smithi001.stderr:election_strategy: 3 2024-03-25T18:38:09.156 INFO:teuthology.misc.health.smithi001.stderr:0: v1:172.21.15.1:6789/0 mon.a 2024-03-25T18:38:09.156 INFO:teuthology.misc.health.smithi001.stderr:1: v1:172.21.15.115:6789/0 mon.b 2024-03-25T18:38:09.156 INFO:teuthology.misc.health.smithi001.stderr:2: v1:172.21.15.1:6790/0 mon.c 2024-03-25T18:38:09.156 INFO:teuthology.misc.health.smithi001.stderr: 2024-03-25T18:38:09.156 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:38:09.154+0000 7fc634d99640 4 set_mon_vals no callback set 2024-03-25T18:38:09.157 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:38:09.155+0000 7fc626ffd640 1 -- 172.21.15.1:0/1572074285 <== mon.1 v1:172.21.15.115:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 21787975 0 0) 0x7fc62c01ad70 con 0x7fc6380b4ff0 2024-03-25T18:38:09.157 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:38:09.156+0000 7fc63659c640 10 monclient: get_auth_request con 0x7fc61806dd00 auth_method 0 2024-03-25T18:38:09.157 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:38:09.156+0000 7fc626ffd640 10 client.4729.objecter ms_handle_connect 0x7fc61806dd00 2024-03-25T18:38:09.162 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:38:09.161+0000 7fc626ffd640 1 -- 172.21.15.1:0/1572074285 <== mon.1 v1:172.21.15.115:6789/0 8 ==== osd_map(11..11 src has 1..11) v4 ==== 1645+0+0 (unknown 2504898309 0 0) 0x7fc62c054810 con 0x7fc6380b4ff0 2024-03-25T18:38:09.162 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:38:09.161+0000 7fc626ffd640 10 client.4729.objecter ms_dispatch 0x7fc638000b90 osd_map(11..11 src has 1..11) v4 2024-03-25T18:38:09.162 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:38:09.161+0000 7fc626ffd640 3 client.4729.objecter handle_osd_map got epochs [11,11] > 0 2024-03-25T18:38:09.162 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:38:09.161+0000 7fc626ffd640 3 client.4729.objecter handle_osd_map decoding full epoch 11 2024-03-25T18:38:09.162 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:38:09.161+0000 7fc626ffd640 20 client.4729.objecter dump_active .. 0 homeless 2024-03-25T18:38:09.164 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:38:09.163+0000 7fc626ffd640 1 -- 172.21.15.1:0/1572074285 <== mon.1 v1:172.21.15.115:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7fc62c050320 con 0x7fc6380b4ff0 2024-03-25T18:38:09.164 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:38:09.163+0000 7fc626ffd640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:38:09.164 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:38:09.163+0000 7fc626ffd640 10 monclient: _finish_command 1 = system:0 2024-03-25T18:38:09.431 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:38:09.430+0000 7fc63c805640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:38:09.431 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:38:09.430+0000 7fc63c805640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:38:09.431 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:38:09.431+0000 7fc63c805640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.115:6789/0 2024-03-25T18:38:09.431 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:38:09.431+0000 7fc63c805640 1 -- 172.21.15.1:0/1572074285 --> v1:172.21.15.115:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fc6380af3a0 con 0x7fc6380b4ff0 2024-03-25T18:38:09.536 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:38:09.535+0000 7fc626ffd640 1 -- 172.21.15.1:0/1572074285 <== mon.1 v1:172.21.15.115:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v11) v1 ==== 74+0+7100 (unknown 238102980 0 2126609903) 0x7fc62c052250 con 0x7fc6380b4ff0 2024-03-25T18:38:09.536 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:38:09.535+0000 7fc626ffd640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:38:09.536 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:38:09.535+0000 7fc626ffd640 10 monclient: _finish_command 2 = system:0 2024-03-25T18:38:09.536 INFO:teuthology.misc.health.smithi001.stdout: 2024-03-25T18:38:09.537 INFO:teuthology.misc.health.smithi001.stdout:{"epoch":11,"fsid":"c8ea72cf-9e99-4383-9b52-6c23d365fdfc","created":"2024-03-25T18:26:38.856638+0000","modified":"2024-03-25T18:36:43.033551+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T18:36:43.033551+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"b9f34922-3a84-49be-8426-807c1eceaefc","up":0,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["autoout","exists","new"]},{"osd":1,"uuid":"42bf306f-55e3-4634-9874-8b4b73ac6644","up":0,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["autoout","exists","new"]},{"osd":2,"uuid":"292cb23a-18a9-4207-a7cc-4d5727feaec0","up":0,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["autoout","exists","new"]},{"osd":3,"uuid":"45e0e9ab-898d-4673-9590-b4117bc5e84a","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"831a5a09-73ab-4a61-9872-088e76bdbc66","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"88cd903d-3b55-4840-9396-6e3cfe005282","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"89db7355-bc9d-46f4-8ab0-e8d9e744e026","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"cac47bee-0914-4f08-896f-348838882866","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":65536,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":65536,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":65536,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T18:38:09.539 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:38:09.537+0000 7fc63c805640 10 librados: watch_flush enter 2024-03-25T18:38:09.539 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:38:09.538+0000 7fc63c805640 10 librados: watch_flush exit 2024-03-25T18:38:09.539 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:38:09.538+0000 7fc63c805640 20 client.4729.objecter shutdown clearing up homeless session... 2024-03-25T18:38:09.539 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:38:09.538+0000 7fc63c805640 10 client.4729.objecter successfully canceled tick 2024-03-25T18:38:09.539 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:38:09.538+0000 7fc63c805640 1 -- 172.21.15.1:0/1572074285 >> v1:172.21.15.115:6800/4237379176 conn(0x7fc61806dd00 legacy=0x7fc6180701c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:38:09.539 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:38:09.538+0000 7fc63c805640 10 monclient: shutdown 2024-03-25T18:38:09.540 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:38:09.538+0000 7fc63c805640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:38:09.540 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:38:09.538+0000 7fc63c805640 1 -- 172.21.15.1:0/1572074285 >> v1:172.21.15.115:6789/0 conn(0x7fc6380b4ff0 legacy=0x7fc63813a230 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:38:09.540 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:38:09.538+0000 7fc63c805640 1 -- 172.21.15.1:0/1572074285 shutdown_connections 2024-03-25T18:38:09.540 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:38:09.538+0000 7fc63c805640 1 -- 172.21.15.1:0/1572074285 >> 172.21.15.1:0/1572074285 conn(0x7fc6380617a0 msgr2=0x7fc6380b6d80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:38:09.540 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:38:09.538+0000 7fc63c805640 1 -- 172.21.15.1:0/1572074285 shutdown_connections 2024-03-25T18:38:09.540 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:38:09.538+0000 7fc63c805640 1 -- 172.21.15.1:0/1572074285 wait complete. 2024-03-25T18:38:09.540 INFO:teuthology.misc.health.smithi001.stderr:2024-03-25T18:38:09.538+0000 7fc63c805640 1 librados: shutdown 2024-03-25T18:38:09.550 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T18:38:09.551 ERROR:teuthology.contextutil:Saw exception from nested tasks Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/contextutil.py", line 30, in nested vars.append(enter()) File "/usr/lib/python3.8/contextlib.py", line 113, in __enter__ return next(self.gen) File "/home/teuthworker/src/git.ceph.com_ceph-c_1fb5d10608dccee73edd6ef8a2c7b804a097c324/qa/tasks/ceph.py", line 401, in create_rbd_pool teuthology.wait_until_osds_up( File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/misc.py", line 861, in wait_until_osds_up while proceed(): File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/contextutil.py", line 134, in __call__ raise MaxWhileTries(error_msg) teuthology.exceptions.MaxWhileTries: reached maximum tries (91) after waiting for 540 seconds 2024-03-25T18:38:09.552 INFO:teuthology.misc:Shutting down osd daemons... 2024-03-25T18:38:09.552 DEBUG:tasks.ceph.osd.0:waiting for process to exit 2024-03-25T18:38:09.553 INFO:teuthology.orchestra.run:waiting for 300 2024-03-25T18:38:11.408 INFO:tasks.ceph.osd.0.smithi001.stderr:==00:00:11:27.016 35190== 2024-03-25T18:38:11.409 INFO:tasks.ceph.osd.0.smithi001.stderr:==00:00:11:27.016 35190== Exit program on first error (--exit-on-first-error=yes) 2024-03-25T18:38:11.447 INFO:tasks.ceph.osd.0.smithi001.stderr:daemon-helper: command failed with exit status 42 2024-03-25T18:38:15.654 DEBUG:teuthology.orchestra.run:got remote process result: 42 2024-03-25T18:38:15.655 ERROR:teuthology.orchestra.daemon.state:Error while waiting for process to exit Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_ceph-c_1fb5d10608dccee73edd6ef8a2c7b804a097c324/qa/tasks/ceph.py", line 1447, in run_daemon yield File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/contextutil.py", line 30, in nested vars.append(enter()) File "/usr/lib/python3.8/contextlib.py", line 113, in __enter__ return next(self.gen) File "/home/teuthworker/src/git.ceph.com_ceph-c_1fb5d10608dccee73edd6ef8a2c7b804a097c324/qa/tasks/ceph.py", line 401, in create_rbd_pool teuthology.wait_until_osds_up( File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/misc.py", line 861, in wait_until_osds_up while proceed(): File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/contextutil.py", line 134, in __call__ raise MaxWhileTries(error_msg) teuthology.exceptions.MaxWhileTries: reached maximum tries (91) after waiting for 540 seconds During handling of the above exception, another exception occurred: Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/orchestra/daemon/state.py", line 139, in stop run.wait([self.proc], timeout=timeout) File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/orchestra/run.py", line 181, in _raise_for_status raise CommandFailedError( teuthology.exceptions.CommandFailedError: Command failed on smithi001 with status 42: "cd /home/ubuntu/cephtest && sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term env 'OPENSSL_ia32cap=~0x1000000000000000' valgrind --trace-children=no --child-silent-after-fork=yes '--soname-synonyms=somalloc=*tcmalloc*' --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.0.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 0" 2024-03-25T18:38:15.656 INFO:tasks.ceph.osd.0:Stopped 2024-03-25T18:38:15.656 DEBUG:tasks.ceph.osd.1:waiting for process to exit 2024-03-25T18:38:15.656 INFO:teuthology.orchestra.run:waiting for 300 2024-03-25T18:38:17.634 INFO:tasks.ceph.osd.1.smithi001.stderr:==00:00:11:33.252 35189== 2024-03-25T18:38:17.655 INFO:tasks.ceph.osd.1.smithi001.stderr:==00:00:11:33.252 35189== Exit program on first error (--exit-on-first-error=yes) 2024-03-25T18:38:17.676 INFO:tasks.ceph.osd.1.smithi001.stderr:daemon-helper: command failed with exit status 42 2024-03-25T18:38:21.758 DEBUG:teuthology.orchestra.run:got remote process result: 42 2024-03-25T18:38:21.758 ERROR:teuthology.orchestra.daemon.state:Error while waiting for process to exit Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_ceph-c_1fb5d10608dccee73edd6ef8a2c7b804a097c324/qa/tasks/ceph.py", line 1447, in run_daemon yield File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/contextutil.py", line 30, in nested vars.append(enter()) File "/usr/lib/python3.8/contextlib.py", line 113, in __enter__ return next(self.gen) File "/home/teuthworker/src/git.ceph.com_ceph-c_1fb5d10608dccee73edd6ef8a2c7b804a097c324/qa/tasks/ceph.py", line 401, in create_rbd_pool teuthology.wait_until_osds_up( File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/misc.py", line 861, in wait_until_osds_up while proceed(): File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/contextutil.py", line 134, in __call__ raise MaxWhileTries(error_msg) teuthology.exceptions.MaxWhileTries: reached maximum tries (91) after waiting for 540 seconds During handling of the above exception, another exception occurred: Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/orchestra/daemon/state.py", line 139, in stop run.wait([self.proc], timeout=timeout) File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/orchestra/run.py", line 181, in _raise_for_status raise CommandFailedError( teuthology.exceptions.CommandFailedError: Command failed on smithi001 with status 42: "cd /home/ubuntu/cephtest && sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term env 'OPENSSL_ia32cap=~0x1000000000000000' valgrind --trace-children=no --child-silent-after-fork=yes '--soname-synonyms=somalloc=*tcmalloc*' --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.1.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 1" 2024-03-25T18:38:21.759 INFO:tasks.ceph.osd.1:Stopped 2024-03-25T18:38:21.759 DEBUG:tasks.ceph.osd.2:waiting for process to exit 2024-03-25T18:38:21.759 INFO:teuthology.orchestra.run:waiting for 300 2024-03-25T18:38:23.726 INFO:tasks.ceph.osd.2.smithi001.stderr:==00:00:11:39.319 35192== 2024-03-25T18:38:23.727 INFO:tasks.ceph.osd.2.smithi001.stderr:==00:00:11:39.319 35192== Exit program on first error (--exit-on-first-error=yes) 2024-03-25T18:38:23.768 INFO:tasks.ceph.osd.2.smithi001.stderr:daemon-helper: command failed with exit status 42 2024-03-25T18:38:27.860 DEBUG:teuthology.orchestra.run:got remote process result: 42 2024-03-25T18:38:27.861 ERROR:teuthology.orchestra.daemon.state:Error while waiting for process to exit Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_ceph-c_1fb5d10608dccee73edd6ef8a2c7b804a097c324/qa/tasks/ceph.py", line 1447, in run_daemon yield File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/contextutil.py", line 30, in nested vars.append(enter()) File "/usr/lib/python3.8/contextlib.py", line 113, in __enter__ return next(self.gen) File "/home/teuthworker/src/git.ceph.com_ceph-c_1fb5d10608dccee73edd6ef8a2c7b804a097c324/qa/tasks/ceph.py", line 401, in create_rbd_pool teuthology.wait_until_osds_up( File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/misc.py", line 861, in wait_until_osds_up while proceed(): File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/contextutil.py", line 134, in __call__ raise MaxWhileTries(error_msg) teuthology.exceptions.MaxWhileTries: reached maximum tries (91) after waiting for 540 seconds During handling of the above exception, another exception occurred: Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/orchestra/daemon/state.py", line 139, in stop run.wait([self.proc], timeout=timeout) File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/orchestra/run.py", line 181, in _raise_for_status raise CommandFailedError( teuthology.exceptions.CommandFailedError: Command failed on smithi001 with status 42: "cd /home/ubuntu/cephtest && sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term env 'OPENSSL_ia32cap=~0x1000000000000000' valgrind --trace-children=no --child-silent-after-fork=yes '--soname-synonyms=somalloc=*tcmalloc*' --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.2.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 2" 2024-03-25T18:38:27.861 INFO:tasks.ceph.osd.2:Stopped 2024-03-25T18:38:27.861 DEBUG:tasks.ceph.osd.3:waiting for process to exit 2024-03-25T18:38:27.861 INFO:teuthology.orchestra.run:waiting for 300 2024-03-25T18:38:28.537 INFO:tasks.ceph.mon.a.smithi001.stderr:==00:00:12:07.096 34876== Warning: unimplemented fcntl command: 1036 2024-03-25T18:38:28.546 INFO:tasks.ceph.mon.a.smithi001.stderr:==00:00:12:07.105 34876== Warning: unimplemented fcntl command: 1036 2024-03-25T18:38:28.570 INFO:tasks.ceph.mon.a.smithi001.stderr:==00:00:12:07.128 34876== Warning: unimplemented fcntl command: 1036 2024-03-25T18:38:28.604 INFO:tasks.ceph.mon.b.smithi115.stderr:==00:00:12:07.088 34836== Warning: unimplemented fcntl command: 1036 2024-03-25T18:38:28.607 INFO:tasks.ceph.mon.b.smithi115.stderr:==00:00:12:07.091 34836== Warning: unimplemented fcntl command: 1036 2024-03-25T18:38:28.616 INFO:tasks.ceph.mon.c.smithi001.stderr:==00:00:12:07.175 34877== Warning: unimplemented fcntl command: 1036 2024-03-25T18:38:28.626 INFO:tasks.ceph.mon.c.smithi001.stderr:==00:00:12:07.184 34877== Warning: unimplemented fcntl command: 1036 2024-03-25T18:38:28.629 INFO:tasks.ceph.mon.b.smithi115.stderr:==00:00:12:07.113 34836== Warning: unimplemented fcntl command: 1036 2024-03-25T18:38:28.661 INFO:tasks.ceph.mon.c.smithi001.stderr:==00:00:12:07.220 34877== Warning: unimplemented fcntl command: 1036 2024-03-25T18:38:29.617 INFO:tasks.ceph.osd.3.smithi001.stderr:==00:00:11:45.219 35191== 2024-03-25T18:38:29.617 INFO:tasks.ceph.osd.3.smithi001.stderr:==00:00:11:45.219 35191== Exit program on first error (--exit-on-first-error=yes) 2024-03-25T18:38:29.652 INFO:tasks.ceph.osd.3.smithi001.stderr:daemon-helper: command failed with exit status 42 2024-03-25T18:38:33.963 DEBUG:teuthology.orchestra.run:got remote process result: 42 2024-03-25T18:38:33.963 ERROR:teuthology.orchestra.daemon.state:Error while waiting for process to exit Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_ceph-c_1fb5d10608dccee73edd6ef8a2c7b804a097c324/qa/tasks/ceph.py", line 1447, in run_daemon yield File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/contextutil.py", line 30, in nested vars.append(enter()) File "/usr/lib/python3.8/contextlib.py", line 113, in __enter__ return next(self.gen) File "/home/teuthworker/src/git.ceph.com_ceph-c_1fb5d10608dccee73edd6ef8a2c7b804a097c324/qa/tasks/ceph.py", line 401, in create_rbd_pool teuthology.wait_until_osds_up( File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/misc.py", line 861, in wait_until_osds_up while proceed(): File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/contextutil.py", line 134, in __call__ raise MaxWhileTries(error_msg) teuthology.exceptions.MaxWhileTries: reached maximum tries (91) after waiting for 540 seconds During handling of the above exception, another exception occurred: Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/orchestra/daemon/state.py", line 139, in stop run.wait([self.proc], timeout=timeout) File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/orchestra/run.py", line 181, in _raise_for_status raise CommandFailedError( teuthology.exceptions.CommandFailedError: Command failed on smithi001 with status 42: "cd /home/ubuntu/cephtest && sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term env 'OPENSSL_ia32cap=~0x1000000000000000' valgrind --trace-children=no --child-silent-after-fork=yes '--soname-synonyms=somalloc=*tcmalloc*' --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.3.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 3" 2024-03-25T18:38:33.963 INFO:tasks.ceph.osd.3:Stopped 2024-03-25T18:38:33.964 DEBUG:tasks.ceph.osd.4:waiting for process to exit 2024-03-25T18:38:33.964 INFO:teuthology.orchestra.run:waiting for 300 2024-03-25T18:38:36.034 INFO:tasks.ceph.osd.4.smithi115.stderr:==00:00:11:51.651 35380== 2024-03-25T18:38:36.034 INFO:tasks.ceph.osd.4.smithi115.stderr:==00:00:11:51.651 35380== Exit program on first error (--exit-on-first-error=yes) 2024-03-25T18:38:36.074 INFO:tasks.ceph.osd.4.smithi115.stderr:daemon-helper: command failed with exit status 42 2024-03-25T18:38:40.066 DEBUG:teuthology.orchestra.run:got remote process result: 42 2024-03-25T18:38:40.066 ERROR:teuthology.orchestra.daemon.state:Error while waiting for process to exit Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_ceph-c_1fb5d10608dccee73edd6ef8a2c7b804a097c324/qa/tasks/ceph.py", line 1447, in run_daemon yield File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/contextutil.py", line 30, in nested vars.append(enter()) File "/usr/lib/python3.8/contextlib.py", line 113, in __enter__ return next(self.gen) File "/home/teuthworker/src/git.ceph.com_ceph-c_1fb5d10608dccee73edd6ef8a2c7b804a097c324/qa/tasks/ceph.py", line 401, in create_rbd_pool teuthology.wait_until_osds_up( File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/misc.py", line 861, in wait_until_osds_up while proceed(): File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/contextutil.py", line 134, in __call__ raise MaxWhileTries(error_msg) teuthology.exceptions.MaxWhileTries: reached maximum tries (91) after waiting for 540 seconds During handling of the above exception, another exception occurred: Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/orchestra/daemon/state.py", line 139, in stop run.wait([self.proc], timeout=timeout) File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/orchestra/run.py", line 181, in _raise_for_status raise CommandFailedError( teuthology.exceptions.CommandFailedError: Command failed on smithi115 with status 42: "cd /home/ubuntu/cephtest && sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term env 'OPENSSL_ia32cap=~0x1000000000000000' valgrind --trace-children=no --child-silent-after-fork=yes '--soname-synonyms=somalloc=*tcmalloc*' --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.4.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 4" 2024-03-25T18:38:40.067 INFO:tasks.ceph.osd.4:Stopped 2024-03-25T18:38:40.067 DEBUG:tasks.ceph.osd.5:waiting for process to exit 2024-03-25T18:38:40.067 INFO:teuthology.orchestra.run:waiting for 300 2024-03-25T18:38:42.066 INFO:tasks.ceph.osd.5.smithi115.stderr:==00:00:11:57.682 35381== 2024-03-25T18:38:42.066 INFO:tasks.ceph.osd.5.smithi115.stderr:==00:00:11:57.682 35381== Exit program on first error (--exit-on-first-error=yes) 2024-03-25T18:38:42.104 INFO:tasks.ceph.osd.5.smithi115.stderr:daemon-helper: command failed with exit status 42 2024-03-25T18:38:46.169 DEBUG:teuthology.orchestra.run:got remote process result: 42 2024-03-25T18:38:46.170 ERROR:teuthology.orchestra.daemon.state:Error while waiting for process to exit Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_ceph-c_1fb5d10608dccee73edd6ef8a2c7b804a097c324/qa/tasks/ceph.py", line 1447, in run_daemon yield File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/contextutil.py", line 30, in nested vars.append(enter()) File "/usr/lib/python3.8/contextlib.py", line 113, in __enter__ return next(self.gen) File "/home/teuthworker/src/git.ceph.com_ceph-c_1fb5d10608dccee73edd6ef8a2c7b804a097c324/qa/tasks/ceph.py", line 401, in create_rbd_pool teuthology.wait_until_osds_up( File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/misc.py", line 861, in wait_until_osds_up while proceed(): File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/contextutil.py", line 134, in __call__ raise MaxWhileTries(error_msg) teuthology.exceptions.MaxWhileTries: reached maximum tries (91) after waiting for 540 seconds During handling of the above exception, another exception occurred: Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/orchestra/daemon/state.py", line 139, in stop run.wait([self.proc], timeout=timeout) File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/orchestra/run.py", line 181, in _raise_for_status raise CommandFailedError( teuthology.exceptions.CommandFailedError: Command failed on smithi115 with status 42: "cd /home/ubuntu/cephtest && sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term env 'OPENSSL_ia32cap=~0x1000000000000000' valgrind --trace-children=no --child-silent-after-fork=yes '--soname-synonyms=somalloc=*tcmalloc*' --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.5.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 5" 2024-03-25T18:38:46.170 INFO:tasks.ceph.osd.5:Stopped 2024-03-25T18:38:46.170 DEBUG:tasks.ceph.osd.6:waiting for process to exit 2024-03-25T18:38:46.170 INFO:teuthology.orchestra.run:waiting for 300 2024-03-25T18:38:47.960 INFO:tasks.ceph.osd.6.smithi115.stderr:==00:00:12:03.577 35382== 2024-03-25T18:38:47.960 INFO:tasks.ceph.osd.6.smithi115.stderr:==00:00:12:03.577 35382== Exit program on first error (--exit-on-first-error=yes) 2024-03-25T18:38:48.000 INFO:tasks.ceph.osd.6.smithi115.stderr:daemon-helper: command failed with exit status 42 2024-03-25T18:38:48.579 INFO:tasks.ceph.mon.a.smithi001.stderr:==00:00:12:27.138 34876== Warning: unimplemented fcntl command: 1036 2024-03-25T18:38:48.583 INFO:tasks.ceph.mon.a.smithi001.stderr:==00:00:12:27.142 34876== Warning: unimplemented fcntl command: 1036 2024-03-25T18:38:48.601 INFO:tasks.ceph.mon.a.smithi001.stderr:==00:00:12:27.160 34876== Warning: unimplemented fcntl command: 1036 2024-03-25T18:38:48.623 INFO:tasks.ceph.mon.b.smithi115.stderr:==00:00:12:27.107 34836== Warning: unimplemented fcntl command: 1036 2024-03-25T18:38:48.625 INFO:tasks.ceph.mon.b.smithi115.stderr:==00:00:12:27.109 34836== Warning: unimplemented fcntl command: 1036 2024-03-25T18:38:48.629 INFO:tasks.ceph.mon.c.smithi001.stderr:==00:00:12:27.188 34877== Warning: unimplemented fcntl command: 1036 2024-03-25T18:38:48.633 INFO:tasks.ceph.mon.c.smithi001.stderr:==00:00:12:27.192 34877== Warning: unimplemented fcntl command: 1036 2024-03-25T18:38:48.641 INFO:tasks.ceph.mon.b.smithi115.stderr:==00:00:12:27.125 34836== Warning: unimplemented fcntl command: 1036 2024-03-25T18:38:48.651 INFO:tasks.ceph.mon.c.smithi001.stderr:==00:00:12:27.210 34877== Warning: unimplemented fcntl command: 1036 2024-03-25T18:38:52.272 DEBUG:teuthology.orchestra.run:got remote process result: 42 2024-03-25T18:38:52.273 ERROR:teuthology.orchestra.daemon.state:Error while waiting for process to exit Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_ceph-c_1fb5d10608dccee73edd6ef8a2c7b804a097c324/qa/tasks/ceph.py", line 1447, in run_daemon yield File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/contextutil.py", line 30, in nested vars.append(enter()) File "/usr/lib/python3.8/contextlib.py", line 113, in __enter__ return next(self.gen) File "/home/teuthworker/src/git.ceph.com_ceph-c_1fb5d10608dccee73edd6ef8a2c7b804a097c324/qa/tasks/ceph.py", line 401, in create_rbd_pool teuthology.wait_until_osds_up( File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/misc.py", line 861, in wait_until_osds_up while proceed(): File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/contextutil.py", line 134, in __call__ raise MaxWhileTries(error_msg) teuthology.exceptions.MaxWhileTries: reached maximum tries (91) after waiting for 540 seconds During handling of the above exception, another exception occurred: Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/orchestra/daemon/state.py", line 139, in stop run.wait([self.proc], timeout=timeout) File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/orchestra/run.py", line 181, in _raise_for_status raise CommandFailedError( teuthology.exceptions.CommandFailedError: Command failed on smithi115 with status 42: "cd /home/ubuntu/cephtest && sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term env 'OPENSSL_ia32cap=~0x1000000000000000' valgrind --trace-children=no --child-silent-after-fork=yes '--soname-synonyms=somalloc=*tcmalloc*' --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.6.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 6" 2024-03-25T18:38:52.273 INFO:tasks.ceph.osd.6:Stopped 2024-03-25T18:38:52.273 DEBUG:tasks.ceph.osd.7:waiting for process to exit 2024-03-25T18:38:52.273 INFO:teuthology.orchestra.run:waiting for 300 2024-03-25T18:38:53.724 INFO:tasks.ceph.osd.7.smithi115.stderr:==00:00:12:09.335 35383== 2024-03-25T18:38:53.724 INFO:tasks.ceph.osd.7.smithi115.stderr:==00:00:12:09.336 35383== Exit program on first error (--exit-on-first-error=yes) 2024-03-25T18:38:53.751 INFO:tasks.ceph.osd.7.smithi115.stderr:daemon-helper: command failed with exit status 42 2024-03-25T18:38:58.375 DEBUG:teuthology.orchestra.run:got remote process result: 42 2024-03-25T18:38:58.375 ERROR:teuthology.orchestra.daemon.state:Error while waiting for process to exit Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_ceph-c_1fb5d10608dccee73edd6ef8a2c7b804a097c324/qa/tasks/ceph.py", line 1447, in run_daemon yield File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/contextutil.py", line 30, in nested vars.append(enter()) File "/usr/lib/python3.8/contextlib.py", line 113, in __enter__ return next(self.gen) File "/home/teuthworker/src/git.ceph.com_ceph-c_1fb5d10608dccee73edd6ef8a2c7b804a097c324/qa/tasks/ceph.py", line 401, in create_rbd_pool teuthology.wait_until_osds_up( File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/misc.py", line 861, in wait_until_osds_up while proceed(): File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/contextutil.py", line 134, in __call__ raise MaxWhileTries(error_msg) teuthology.exceptions.MaxWhileTries: reached maximum tries (91) after waiting for 540 seconds During handling of the above exception, another exception occurred: Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/orchestra/daemon/state.py", line 139, in stop run.wait([self.proc], timeout=timeout) File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/orchestra/run.py", line 181, in _raise_for_status raise CommandFailedError( teuthology.exceptions.CommandFailedError: Command failed on smithi115 with status 42: "cd /home/ubuntu/cephtest && sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term env 'OPENSSL_ia32cap=~0x1000000000000000' valgrind --trace-children=no --child-silent-after-fork=yes '--soname-synonyms=somalloc=*tcmalloc*' --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.7.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 7" 2024-03-25T18:38:58.376 INFO:tasks.ceph.osd.7:Stopped 2024-03-25T18:38:58.376 INFO:teuthology.misc:Shutting down mgr daemons... 2024-03-25T18:38:58.377 DEBUG:tasks.ceph.mgr.y:waiting for process to exit 2024-03-25T18:38:58.377 INFO:teuthology.orchestra.run:waiting for 300 2024-03-25T18:38:58.404 INFO:tasks.ceph.mgr.y:Stopped 2024-03-25T18:38:58.404 DEBUG:tasks.ceph.mgr.x:waiting for process to exit 2024-03-25T18:38:58.404 INFO:teuthology.orchestra.run:waiting for 300 2024-03-25T18:38:58.405 INFO:tasks.ceph.mgr.x.smithi115.stderr:2024-03-25T18:38:58.405+0000 7f99d896e640 -1 received signal: Terminated from /usr/bin/python3 /bin/daemon-helper term ceph-mgr -f --cluster ceph -i x (PID: 34834) UID: 0 2024-03-25T18:38:58.405 INFO:tasks.ceph.mgr.x.smithi115.stderr:2024-03-25T18:38:58.405+0000 7f99d896e640 -1 mgr handle_mgr_signal *** Got signal Terminated *** 2024-03-25T18:38:58.435 INFO:tasks.ceph.mgr.x:Stopped 2024-03-25T18:38:58.435 INFO:teuthology.misc:Shutting down mon daemons... 2024-03-25T18:38:58.435 DEBUG:tasks.ceph.mon.a:waiting for process to exit 2024-03-25T18:38:58.435 INFO:teuthology.orchestra.run:waiting for 300 2024-03-25T18:38:58.460 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-03-25T18:38:58.447+0000 9f86640 -1 received signal: Terminated from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/mon.a.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck --leak-check=full --show-reachable=yes ceph-mon -f --cluster ceph -i a (PID: 34843) UID: 0 2024-03-25T18:38:58.461 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-03-25T18:38:58.449+0000 9f86640 -1 mon.a@0(leader) e1 *** Got Signal Terminated *** 2024-03-25T18:38:58.987 INFO:tasks.ceph.mon.a.smithi001.stderr:==00:00:12:37.545 34876== 2024-03-25T18:38:58.987 INFO:tasks.ceph.mon.a.smithi001.stderr:==00:00:12:37.545 34876== Exit program on first error (--exit-on-first-error=yes) 2024-03-25T18:38:58.996 INFO:tasks.ceph.mon.a.smithi001.stderr:daemon-helper: command failed with exit status 42 2024-03-25T18:39:04.537 DEBUG:teuthology.orchestra.run:got remote process result: 42 2024-03-25T18:39:04.538 ERROR:teuthology.orchestra.daemon.state:Error while waiting for process to exit Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_ceph-c_1fb5d10608dccee73edd6ef8a2c7b804a097c324/qa/tasks/ceph.py", line 1447, in run_daemon yield File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/contextutil.py", line 30, in nested vars.append(enter()) File "/usr/lib/python3.8/contextlib.py", line 113, in __enter__ return next(self.gen) File "/home/teuthworker/src/git.ceph.com_ceph-c_1fb5d10608dccee73edd6ef8a2c7b804a097c324/qa/tasks/ceph.py", line 401, in create_rbd_pool teuthology.wait_until_osds_up( File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/misc.py", line 861, in wait_until_osds_up while proceed(): File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/contextutil.py", line 134, in __call__ raise MaxWhileTries(error_msg) teuthology.exceptions.MaxWhileTries: reached maximum tries (91) after waiting for 540 seconds During handling of the above exception, another exception occurred: Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/orchestra/daemon/state.py", line 139, in stop run.wait([self.proc], timeout=timeout) File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/orchestra/run.py", line 181, in _raise_for_status raise CommandFailedError( teuthology.exceptions.CommandFailedError: Command failed on smithi001 with status 42: "cd /home/ubuntu/cephtest && sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term env 'OPENSSL_ia32cap=~0x1000000000000000' valgrind --trace-children=no --child-silent-after-fork=yes '--soname-synonyms=somalloc=*tcmalloc*' --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/mon.a.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck --leak-check=full --show-reachable=yes ceph-mon -f --cluster ceph -i a" 2024-03-25T18:39:04.538 INFO:tasks.ceph.mon.a:Stopped 2024-03-25T18:39:04.538 DEBUG:tasks.ceph.mon.c:waiting for process to exit 2024-03-25T18:39:04.539 INFO:teuthology.orchestra.run:waiting for 300 2024-03-25T18:39:04.555 INFO:tasks.ceph.mon.c.smithi001.stderr:2024-03-25T18:39:04.543+0000 9f86640 -1 received signal: Terminated from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/mon.c.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck --leak-check=full --show-reachable=yes ceph-mon -f --cluster ceph -i c (PID: 34845) UID: 0 2024-03-25T18:39:04.555 INFO:tasks.ceph.mon.c.smithi001.stderr:2024-03-25T18:39:04.544+0000 9f86640 -1 mon.c@2(peon) e1 *** Got Signal Terminated *** 2024-03-25T18:39:11.334 INFO:tasks.ceph.mon.c.smithi001.stderr:==00:00:12:49.892 34877== 2024-03-25T18:39:11.334 INFO:tasks.ceph.mon.c.smithi001.stderr:==00:00:12:49.892 34877== Exit program on first error (--exit-on-first-error=yes) 2024-03-25T18:39:11.343 INFO:tasks.ceph.mon.c.smithi001.stderr:daemon-helper: command failed with exit status 42 2024-03-25T18:39:16.742 DEBUG:teuthology.orchestra.run:got remote process result: 42 2024-03-25T18:39:16.742 ERROR:teuthology.orchestra.daemon.state:Error while waiting for process to exit Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_ceph-c_1fb5d10608dccee73edd6ef8a2c7b804a097c324/qa/tasks/ceph.py", line 1447, in run_daemon yield File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/contextutil.py", line 30, in nested vars.append(enter()) File "/usr/lib/python3.8/contextlib.py", line 113, in __enter__ return next(self.gen) File "/home/teuthworker/src/git.ceph.com_ceph-c_1fb5d10608dccee73edd6ef8a2c7b804a097c324/qa/tasks/ceph.py", line 401, in create_rbd_pool teuthology.wait_until_osds_up( File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/misc.py", line 861, in wait_until_osds_up while proceed(): File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/contextutil.py", line 134, in __call__ raise MaxWhileTries(error_msg) teuthology.exceptions.MaxWhileTries: reached maximum tries (91) after waiting for 540 seconds During handling of the above exception, another exception occurred: Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/orchestra/daemon/state.py", line 139, in stop run.wait([self.proc], timeout=timeout) File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/orchestra/run.py", line 181, in _raise_for_status raise CommandFailedError( teuthology.exceptions.CommandFailedError: Command failed on smithi001 with status 42: "cd /home/ubuntu/cephtest && sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term env 'OPENSSL_ia32cap=~0x1000000000000000' valgrind --trace-children=no --child-silent-after-fork=yes '--soname-synonyms=somalloc=*tcmalloc*' --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/mon.c.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck --leak-check=full --show-reachable=yes ceph-mon -f --cluster ceph -i c" 2024-03-25T18:39:16.743 INFO:tasks.ceph.mon.c:Stopped 2024-03-25T18:39:16.743 DEBUG:tasks.ceph.mon.b:waiting for process to exit 2024-03-25T18:39:16.743 INFO:teuthology.orchestra.run:waiting for 300 2024-03-25T18:39:16.759 INFO:tasks.ceph.mon.b.smithi115.stderr:2024-03-25T18:39:16.747+0000 9f86640 -1 received signal: Terminated from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/mon.b.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck --leak-check=full --show-reachable=yes ceph-mon -f --cluster ceph -i b (PID: 34832) UID: 0 2024-03-25T18:39:16.760 INFO:tasks.ceph.mon.b.smithi115.stderr:2024-03-25T18:39:16.749+0000 9f86640 -1 mon.b@1(probing) e1 *** Got Signal Terminated *** 2024-03-25T18:39:17.195 INFO:tasks.ceph.mon.b.smithi115.stderr:==00:00:12:55.679 34836== 2024-03-25T18:39:17.195 INFO:tasks.ceph.mon.b.smithi115.stderr:==00:00:12:55.679 34836== Exit program on first error (--exit-on-first-error=yes) 2024-03-25T18:39:17.204 INFO:tasks.ceph.mon.b.smithi115.stderr:daemon-helper: command failed with exit status 42 2024-03-25T18:39:22.845 DEBUG:teuthology.orchestra.run:got remote process result: 42 2024-03-25T18:39:22.845 ERROR:teuthology.orchestra.daemon.state:Error while waiting for process to exit Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_ceph-c_1fb5d10608dccee73edd6ef8a2c7b804a097c324/qa/tasks/ceph.py", line 1447, in run_daemon yield File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/contextutil.py", line 30, in nested vars.append(enter()) File "/usr/lib/python3.8/contextlib.py", line 113, in __enter__ return next(self.gen) File "/home/teuthworker/src/git.ceph.com_ceph-c_1fb5d10608dccee73edd6ef8a2c7b804a097c324/qa/tasks/ceph.py", line 401, in create_rbd_pool teuthology.wait_until_osds_up( File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/misc.py", line 861, in wait_until_osds_up while proceed(): File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/contextutil.py", line 134, in __call__ raise MaxWhileTries(error_msg) teuthology.exceptions.MaxWhileTries: reached maximum tries (91) after waiting for 540 seconds During handling of the above exception, another exception occurred: Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/orchestra/daemon/state.py", line 139, in stop run.wait([self.proc], timeout=timeout) File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/orchestra/run.py", line 181, in _raise_for_status raise CommandFailedError( teuthology.exceptions.CommandFailedError: Command failed on smithi115 with status 42: "cd /home/ubuntu/cephtest && sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term env 'OPENSSL_ia32cap=~0x1000000000000000' valgrind --trace-children=no --child-silent-after-fork=yes '--soname-synonyms=somalloc=*tcmalloc*' --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/mon.b.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck --leak-check=full --show-reachable=yes ceph-mon -f --cluster ceph -i b" 2024-03-25T18:39:22.845 INFO:tasks.ceph.mon.b:Stopped 2024-03-25T18:39:22.846 INFO:tasks.ceph:Checking cluster log for badness... 2024-03-25T18:39:22.846 DEBUG:teuthology.orchestra.run.smithi001:> sudo egrep '\[ERR\]|\[WRN\]|\[SEC\]' /var/log/ceph/ceph.log | egrep -v '\(MDS_ALL_DOWN\)' | egrep -v '\(MDS_UP_LESS_THAN_MAX\)' | egrep -v 'but it is still running' | egrep -v 'objects unfound and apparently lost' | egrep -v 'overall HEALTH_' | egrep -v '\(OSDMAP_FLAGS\)' | egrep -v '\(OSD_' | egrep -v '\(PG_' | egrep -v '\(POOL_' | egrep -v '\(CACHE_POOL_' | egrep -v '\(SMALLER_PGP_NUM\)' | egrep -v '\(OBJECT_' | egrep -v '\(SLOW_OPS\)' | egrep -v '\(REQUEST_SLOW\)' | egrep -v '\(TOO_FEW_PGS\)' | egrep -v 'slow request' | egrep -v 'timeout on replica' | egrep -v 'late reservation from' | egrep -v MON_DOWN | egrep -v OSDMAP_FLAGS | egrep -v OSD_DOWN | egrep -v PG_DEGRADED | egrep -v PG_AVAILABILITY | egrep -v POOL_APP_NOT_ENABLED | egrep -v '\(OSD_SLOW_PING_TIME' | egrep -v 'reached quota' | egrep -v 'overall HEALTH_' | egrep -v '\(CACHE_POOL_NO_HIT_SET\)' | egrep -v '\(POOL_FULL\)' | egrep -v '\(SMALLER_PGP_NUM\)' | egrep -v '\(SLOW_OPS\)' | egrep -v '\(CACHE_POOL_NEAR_FULL\)' | egrep -v '\(POOL_APP_NOT_ENABLED\)' | egrep -v '\(PG_AVAILABILITY\)' | egrep -v '\(OBJECT_MISPLACED\)' | egrep -v 'slow request' | egrep -v 'overall HEALTH_' | egrep -v '\(PG_' | egrep -v '\(MON_DOWN\)' | egrep -v '\(SLOW_OPS\)' | egrep -v 'slow request' | head -n 1 2024-03-25T18:39:22.895 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/ceph-0 on ubuntu@smithi001.front.sepia.ceph.com 2024-03-25T18:39:22.895 DEBUG:teuthology.orchestra.run.smithi001:> sync && sudo umount -f /var/lib/ceph/osd/ceph-0 2024-03-25T18:39:22.952 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/ceph-1 on ubuntu@smithi001.front.sepia.ceph.com 2024-03-25T18:39:22.952 DEBUG:teuthology.orchestra.run.smithi001:> sync && sudo umount -f /var/lib/ceph/osd/ceph-1 2024-03-25T18:39:23.019 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/ceph-2 on ubuntu@smithi001.front.sepia.ceph.com 2024-03-25T18:39:23.019 DEBUG:teuthology.orchestra.run.smithi001:> sync && sudo umount -f /var/lib/ceph/osd/ceph-2 2024-03-25T18:39:23.075 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/ceph-3 on ubuntu@smithi001.front.sepia.ceph.com 2024-03-25T18:39:23.076 DEBUG:teuthology.orchestra.run.smithi001:> sync && sudo umount -f /var/lib/ceph/osd/ceph-3 2024-03-25T18:39:23.143 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/ceph-4 on ubuntu@smithi115.front.sepia.ceph.com 2024-03-25T18:39:23.143 DEBUG:teuthology.orchestra.run.smithi115:> sync && sudo umount -f /var/lib/ceph/osd/ceph-4 2024-03-25T18:39:23.271 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/ceph-5 on ubuntu@smithi115.front.sepia.ceph.com 2024-03-25T18:39:23.271 DEBUG:teuthology.orchestra.run.smithi115:> sync && sudo umount -f /var/lib/ceph/osd/ceph-5 2024-03-25T18:39:23.325 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/ceph-6 on ubuntu@smithi115.front.sepia.ceph.com 2024-03-25T18:39:23.325 DEBUG:teuthology.orchestra.run.smithi115:> sync && sudo umount -f /var/lib/ceph/osd/ceph-6 2024-03-25T18:39:23.394 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/ceph-7 on ubuntu@smithi115.front.sepia.ceph.com 2024-03-25T18:39:23.395 DEBUG:teuthology.orchestra.run.smithi115:> sync && sudo umount -f /var/lib/ceph/osd/ceph-7 2024-03-25T18:39:23.469 INFO:tasks.ceph:Archiving mon data... 2024-03-25T18:39:23.471 DEBUG:teuthology.misc:Transferring archived files from smithi001:/var/lib/ceph/mon/ceph-a to /home/teuthworker/archive/amathuri-2024-03-25_13:28:33-rados:verify-main-distro-default-smithi/7621279/data/mon.a.tgz 2024-03-25T18:39:23.471 DEBUG:teuthology.orchestra.run.smithi001:> mktemp 2024-03-25T18:39:23.488 INFO:teuthology.orchestra.run.smithi001.stdout:/tmp/tmp.5mN10AV9CX 2024-03-25T18:39:23.488 DEBUG:teuthology.orchestra.run.smithi001:> sudo tar cz -f - -C /var/lib/ceph/mon/ceph-a -- . > /tmp/tmp.5mN10AV9CX 2024-03-25T18:39:23.597 DEBUG:teuthology.orchestra.run.smithi001:> sudo chmod 0666 /tmp/tmp.5mN10AV9CX 2024-03-25T18:39:23.683 DEBUG:teuthology.orchestra.remote:smithi001:/tmp/tmp.5mN10AV9CX is 134KB 2024-03-25T18:39:23.735 DEBUG:teuthology.orchestra.run.smithi001:> rm -fr /tmp/tmp.5mN10AV9CX 2024-03-25T18:39:23.751 DEBUG:teuthology.misc:Transferring archived files from smithi001:/var/lib/ceph/mon/ceph-c to /home/teuthworker/archive/amathuri-2024-03-25_13:28:33-rados:verify-main-distro-default-smithi/7621279/data/mon.c.tgz 2024-03-25T18:39:23.751 DEBUG:teuthology.orchestra.run.smithi001:> mktemp 2024-03-25T18:39:23.805 INFO:teuthology.orchestra.run.smithi001.stdout:/tmp/tmp.deLOW3M57C 2024-03-25T18:39:23.805 DEBUG:teuthology.orchestra.run.smithi001:> sudo tar cz -f - -C /var/lib/ceph/mon/ceph-c -- . > /tmp/tmp.deLOW3M57C 2024-03-25T18:39:23.895 DEBUG:teuthology.orchestra.run.smithi001:> sudo chmod 0666 /tmp/tmp.deLOW3M57C 2024-03-25T18:39:23.980 DEBUG:teuthology.orchestra.remote:smithi001:/tmp/tmp.deLOW3M57C is 134KB 2024-03-25T18:39:24.032 DEBUG:teuthology.orchestra.run.smithi001:> rm -fr /tmp/tmp.deLOW3M57C 2024-03-25T18:39:24.047 DEBUG:teuthology.misc:Transferring archived files from smithi115:/var/lib/ceph/mon/ceph-b to /home/teuthworker/archive/amathuri-2024-03-25_13:28:33-rados:verify-main-distro-default-smithi/7621279/data/mon.b.tgz 2024-03-25T18:39:24.047 DEBUG:teuthology.orchestra.run.smithi115:> mktemp 2024-03-25T18:39:24.065 INFO:teuthology.orchestra.run.smithi115.stdout:/tmp/tmp.X4xMWpVM7o 2024-03-25T18:39:24.066 DEBUG:teuthology.orchestra.run.smithi115:> sudo tar cz -f - -C /var/lib/ceph/mon/ceph-b -- . > /tmp/tmp.X4xMWpVM7o 2024-03-25T18:39:24.194 DEBUG:teuthology.orchestra.run.smithi115:> sudo chmod 0666 /tmp/tmp.X4xMWpVM7o 2024-03-25T18:39:24.285 DEBUG:teuthology.orchestra.remote:smithi115:/tmp/tmp.X4xMWpVM7o is 134KB 2024-03-25T18:39:24.337 DEBUG:teuthology.orchestra.run.smithi115:> rm -fr /tmp/tmp.X4xMWpVM7o 2024-03-25T18:39:24.351 INFO:tasks.ceph:Cleaning ceph cluster... 2024-03-25T18:39:24.352 DEBUG:teuthology.orchestra.run.smithi001:> sudo rm -rf -- /etc/ceph/ceph.conf /etc/ceph/ceph.keyring /home/ubuntu/cephtest/ceph.data /home/ubuntu/cephtest/ceph.monmap /home/ubuntu/cephtest/../*.pid 2024-03-25T18:39:24.354 DEBUG:teuthology.orchestra.run.smithi115:> sudo rm -rf -- /etc/ceph/ceph.conf /etc/ceph/ceph.keyring /home/ubuntu/cephtest/ceph.data /home/ubuntu/cephtest/ceph.monmap /home/ubuntu/cephtest/../*.pid 2024-03-25T18:39:24.743 DEBUG:tasks.ceph:valgrind exception message: valgrind error: Leak_StillReachable operator new[](unsigned long) UnknownInlinedFun UnknownInlinedFun 2024-03-25T18:39:24.931 INFO:tasks.ceph:Archiving crash dumps... 2024-03-25T18:39:24.933 DEBUG:teuthology.misc:Transferring archived files from smithi001:/var/lib/ceph/crash to /home/teuthworker/archive/amathuri-2024-03-25_13:28:33-rados:verify-main-distro-default-smithi/7621279/remote/smithi001/crash 2024-03-25T18:39:24.934 DEBUG:teuthology.orchestra.run.smithi001:> sudo tar c -f - -C /var/lib/ceph/crash -- . 2024-03-25T18:39:24.962 DEBUG:teuthology.misc:Transferring archived files from smithi115:/var/lib/ceph/crash to /home/teuthworker/archive/amathuri-2024-03-25_13:28:33-rados:verify-main-distro-default-smithi/7621279/remote/smithi115/crash 2024-03-25T18:39:24.963 DEBUG:teuthology.orchestra.run.smithi115:> sudo tar c -f - -C /var/lib/ceph/crash -- . 2024-03-25T18:39:24.990 INFO:tasks.ceph:Compressing logs... 2024-03-25T18:39:24.990 DEBUG:teuthology.orchestra.run.smithi001:> time sudo find /var/log/ceph -name '*.log' -print0 | sudo xargs --max-args=1 --max-procs=0 --verbose -0 --no-run-if-empty -- gzip -5 --verbose -- 2024-03-25T18:39:25.006 DEBUG:teuthology.orchestra.run.smithi115:> time sudo find /var/log/ceph -name '*.log' -print0 | sudo xargs --max-args=1 --max-procs=0 --verbose -0 --no-run-if-empty -- gzip -5 --verbose -- 2024-03-25T18:39:25.032 INFO:teuthology.orchestra.run.smithi001.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40144.log 2024-03-25T18:39:25.032 INFO:teuthology.orchestra.run.smithi001.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-mon.a.log 2024-03-25T18:39:25.075 INFO:teuthology.orchestra.run.smithi115.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-osd.6.log 2024-03-25T18:39:25.075 INFO:teuthology.orchestra.run.smithi115.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.35190.log 2024-03-25T18:39:25.075 INFO:teuthology.orchestra.run.smithi115.stderr:/var/log/ceph/ceph-osd.6.log: gzip -5 --verbose -- /var/log/ceph/valgrind/osd.7.log 2024-03-25T18:39:25.075 INFO:teuthology.orchestra.run.smithi115.stderr:/var/log/ceph/ceph-client.admin.35190.log: gzip -5 --verbose -- /var/log/ceph/valgrind/osd.5.log 2024-03-25T18:39:25.075 INFO:teuthology.orchestra.run.smithi115.stderr:/var/log/ceph/valgrind/osd.7.log: gzip -5 --verbose -- /var/log/ceph/valgrind/osd.6.log 2024-03-25T18:39:25.075 INFO:teuthology.orchestra.run.smithi115.stderr: 87.6% -- replaced with /var/log/ceph/ceph-client.admin.35190.log.gz 2024-03-25T18:39:25.076 INFO:teuthology.orchestra.run.smithi115.stderr: 72.0% -- replaced with /var/log/ceph/valgrind/osd.7.log.gz 2024-03-25T18:39:25.076 INFO:teuthology.orchestra.run.smithi115.stderr:/var/log/ceph/valgrind/osd.5.log: gzip -5 --verbose -- /var/log/ceph/valgrind/mon.b.log 2024-03-25T18:39:25.076 INFO:teuthology.orchestra.run.smithi115.stderr: 72.0% -- replaced with /var/log/ceph/valgrind/osd.5.log.gz 2024-03-25T18:39:25.076 INFO:teuthology.orchestra.run.smithi115.stderr:/var/log/ceph/valgrind/osd.6.log: gzip -5 --verbose -- /var/log/ceph/valgrind/osd.4.log 2024-03-25T18:39:25.076 INFO:teuthology.orchestra.run.smithi115.stderr: 72.0% -- replaced with /var/log/ceph/valgrind/osd.6.log.gz 2024-03-25T18:39:25.076 INFO:teuthology.orchestra.run.smithi115.stderr:/var/log/ceph/valgrind/mon.b.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.35281.log 2024-03-25T18:39:25.076 INFO:teuthology.orchestra.run.smithi115.stderr: 69.6% -- replaced with /var/log/ceph/valgrind/mon.b.log.gz 2024-03-25T18:39:25.076 INFO:teuthology.orchestra.run.smithi115.stderr:/var/log/ceph/valgrind/osd.4.log: gzip -5 --verbose -- /var/log/ceph/ceph.audit.log 2024-03-25T18:39:25.076 INFO:teuthology.orchestra.run.smithi115.stderr: 72.0% -- replaced with /var/log/ceph/valgrind/osd.4.log.gz 2024-03-25T18:39:25.076 INFO:teuthology.orchestra.run.smithi115.stderr:/var/log/ceph/ceph-client.admin.35281.log: gzip -5 --verbose -- /var/log/ceph/ceph-osd.4.log 2024-03-25T18:39:25.076 INFO:teuthology.orchestra.run.smithi115.stderr:/var/log/ceph/ceph.audit.log: gzip -5 --verbose -- /var/log/ceph/ceph-osd.7.log 2024-03-25T18:39:25.076 INFO:teuthology.orchestra.run.smithi115.stderr: 87.7% -- replaced with /var/log/ceph/ceph-client.admin.35281.log.gz 2024-03-25T18:39:25.076 INFO:teuthology.orchestra.run.smithi115.stderr:/var/log/ceph/ceph-osd.4.log: gzip -5 --verbose -- /var/log/ceph/ceph-osd.5.log 2024-03-25T18:39:25.076 INFO:teuthology.orchestra.run.smithi115.stderr: 88.6% -- replaced with /var/log/ceph/ceph.audit.log.gz 2024-03-25T18:39:25.076 INFO:teuthology.orchestra.run.smithi115.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.35236.log 2024-03-25T18:39:25.077 INFO:teuthology.orchestra.run.smithi115.stderr:/var/log/ceph/ceph-osd.7.log: /var/log/ceph/ceph-osd.5.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.35150.log 2024-03-25T18:39:25.077 INFO:teuthology.orchestra.run.smithi115.stderr:/var/log/ceph/ceph-client.admin.35236.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.34990.log 2024-03-25T18:39:25.077 INFO:teuthology.orchestra.run.smithi115.stderr:/var/log/ceph/ceph-client.admin.35150.log: 87.5%gzip -5 --verbose -- /var/log/ceph/ceph-mgr.x.log 2024-03-25T18:39:25.077 INFO:teuthology.orchestra.run.smithi115.stderr: -- replaced with /var/log/ceph/ceph-client.admin.35236.log.gz 2024-03-25T18:39:25.077 INFO:teuthology.orchestra.run.smithi115.stderr:/var/log/ceph/ceph-client.admin.34990.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.35070.log 2024-03-25T18:39:25.077 INFO:teuthology.orchestra.run.smithi115.stderr: 87.5% -- replaced with /var/log/ceph/ceph-client.admin.35150.log.gz 2024-03-25T18:39:25.077 INFO:teuthology.orchestra.run.smithi115.stderr:/var/log/ceph/ceph-mgr.x.log: 86.5% -- replaced with /var/log/ceph/ceph-client.admin.34990.log.gz 2024-03-25T18:39:25.077 INFO:teuthology.orchestra.run.smithi115.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.35030.log 2024-03-25T18:39:25.077 INFO:teuthology.orchestra.run.smithi115.stderr:/var/log/ceph/ceph-client.admin.35070.log: gzip -5 --verbose -- /var/log/ceph/ceph.log 2024-03-25T18:39:25.077 INFO:teuthology.orchestra.run.smithi115.stderr:/var/log/ceph/ceph-client.admin.35030.log: 87.6%gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.35110.log 2024-03-25T18:39:25.077 INFO:teuthology.orchestra.run.smithi115.stderr: -- replaced with /var/log/ceph/ceph-client.admin.35070.log.gz 2024-03-25T18:39:25.077 INFO:teuthology.orchestra.run.smithi115.stderr:/var/log/ceph/ceph.log: 87.3% -- replaced with /var/log/ceph/ceph-client.admin.35030.log.gz 2024-03-25T18:39:25.077 INFO:teuthology.orchestra.run.smithi115.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-mon.b.log 2024-03-25T18:39:25.077 INFO:teuthology.orchestra.run.smithi115.stderr:/var/log/ceph/ceph-client.admin.35110.log: 87.4% -- replaced with /var/log/ceph/ceph.log.gz 2024-03-25T18:39:25.078 INFO:teuthology.orchestra.run.smithi115.stderr: 87.3% -- replaced with /var/log/ceph/ceph-client.admin.35110.log.gz 2024-03-25T18:39:25.088 INFO:teuthology.orchestra.run.smithi001.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39427.log 2024-03-25T18:39:25.088 INFO:teuthology.orchestra.run.smithi001.stderr:/var/log/ceph/ceph-client.admin.40144.log: /var/log/ceph/ceph-mon.a.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41861.log 2024-03-25T18:39:25.088 INFO:teuthology.orchestra.run.smithi001.stderr:/var/log/ceph/ceph-client.admin.39427.log: 86.7% -- replaced with /var/log/ceph/ceph-client.admin.40144.log.gz 2024-03-25T18:39:25.088 INFO:teuthology.orchestra.run.smithi001.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39153.log 2024-03-25T18:39:25.088 INFO:teuthology.orchestra.run.smithi001.stderr:/var/log/ceph/ceph-client.admin.41861.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41979.log 2024-03-25T18:39:25.088 INFO:teuthology.orchestra.run.smithi001.stderr: 86.7% -- replaced with /var/log/ceph/ceph-client.admin.39427.log.gz 2024-03-25T18:39:25.088 INFO:teuthology.orchestra.run.smithi001.stderr:/var/log/ceph/ceph-client.admin.39153.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40417.log 2024-03-25T18:39:25.088 INFO:teuthology.orchestra.run.smithi001.stderr: 86.6% -- replaced with /var/log/ceph/ceph-client.admin.41861.log.gz 2024-03-25T18:39:25.088 INFO:teuthology.orchestra.run.smithi001.stderr:/var/log/ceph/ceph-client.admin.41979.log: 86.1% -- replaced with /var/log/ceph/ceph-client.admin.39153.log.gz 2024-03-25T18:39:25.088 INFO:teuthology.orchestra.run.smithi001.stderr:gzip -5 --verbose -- /var/log/ceph/valgrind/mon.c.log 2024-03-25T18:39:25.089 INFO:teuthology.orchestra.run.smithi001.stderr:/var/log/ceph/ceph-client.admin.40417.log: gzip -5 --verbose -- /var/log/ceph/valgrind/osd.2.log 2024-03-25T18:39:25.089 INFO:teuthology.orchestra.run.smithi001.stderr: 86.1% -- replaced with /var/log/ceph/ceph-client.admin.41979.log.gz 2024-03-25T18:39:25.089 INFO:teuthology.orchestra.run.smithi001.stderr:/var/log/ceph/valgrind/mon.c.log: gzip -5 --verbose -- /var/log/ceph/valgrind/osd.0.log 2024-03-25T18:39:25.089 INFO:teuthology.orchestra.run.smithi001.stderr: 85.6% -- replaced with /var/log/ceph/ceph-client.admin.40417.log.gz 2024-03-25T18:39:25.089 INFO:teuthology.orchestra.run.smithi001.stderr: 69.6% -- replaced with /var/log/ceph/valgrind/mon.c.log.gz 2024-03-25T18:39:25.089 INFO:teuthology.orchestra.run.smithi001.stderr:gzip/var/log/ceph/valgrind/osd.2.log: -5 --verbose -- /var/log/ceph/valgrind/osd.1.log 2024-03-25T18:39:25.089 INFO:teuthology.orchestra.run.smithi001.stderr: 72.0% -- replaced with /var/log/ceph/valgrind/osd.2.log.gz 2024-03-25T18:39:25.089 INFO:teuthology.orchestra.run.smithi001.stderr:/var/log/ceph/valgrind/osd.0.log: gzip -5 --verbose -- /var/log/ceph/valgrind/mon.a.log 2024-03-25T18:39:25.089 INFO:teuthology.orchestra.run.smithi001.stderr:/var/log/ceph/valgrind/osd.1.log: 72.0% -- replaced with /var/log/ceph/valgrind/osd.0.log.gz 2024-03-25T18:39:25.089 INFO:teuthology.orchestra.run.smithi001.stderr:gzip -5 --verbose -- /var/log/ceph/valgrind/osd.3.log 2024-03-25T18:39:25.089 INFO:teuthology.orchestra.run.smithi001.stderr: 72.0% -- replaced with /var/log/ceph/valgrind/osd.1.log.gz 2024-03-25T18:39:25.089 INFO:teuthology.orchestra.run.smithi001.stderr:/var/log/ceph/valgrind/mon.a.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.38423.log 2024-03-25T18:39:25.089 INFO:teuthology.orchestra.run.smithi001.stderr: 69.6% -- replaced with /var/log/ceph/valgrind/mon.a.log.gz 2024-03-25T18:39:25.089 INFO:teuthology.orchestra.run.smithi001.stderr:/var/log/ceph/valgrind/osd.3.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39231.log 2024-03-25T18:39:25.090 INFO:teuthology.orchestra.run.smithi001.stderr:/var/log/ceph/ceph-client.admin.38423.log: 72.0% -- replaced with /var/log/ceph/valgrind/osd.3.log.gz 2024-03-25T18:39:25.090 INFO:teuthology.orchestra.run.smithi001.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41354.log 2024-03-25T18:39:25.090 INFO:teuthology.orchestra.run.smithi001.stderr:/var/log/ceph/ceph-client.admin.39231.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41549.log 2024-03-25T18:39:25.090 INFO:teuthology.orchestra.run.smithi001.stderr: 87.4% -- replaced with /var/log/ceph/ceph-client.admin.38423.log.gz 2024-03-25T18:39:25.090 INFO:teuthology.orchestra.run.smithi001.stderr:/var/log/ceph/ceph-client.admin.41354.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40846.log 86.5% 2024-03-25T18:39:25.090 INFO:teuthology.orchestra.run.smithi001.stderr: -- replaced with /var/log/ceph/ceph-client.admin.39231.log.gz 2024-03-25T18:39:25.090 INFO:teuthology.orchestra.run.smithi001.stderr:/var/log/ceph/ceph-client.admin.41549.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.34885.log 2024-03-25T18:39:25.090 INFO:teuthology.orchestra.run.smithi001.stderr: 86.9% -- replaced with /var/log/ceph/ceph-client.admin.41354.log.gz 2024-03-25T18:39:25.090 INFO:teuthology.orchestra.run.smithi001.stderr:/var/log/ceph/ceph-client.admin.40846.log: gzip 86.7% -5 -- replaced with /var/log/ceph/ceph-client.admin.41549.log.gz --verbose 2024-03-25T18:39:25.090 INFO:teuthology.orchestra.run.smithi001.stderr: -- /var/log/ceph/ceph-client.admin.40339.log 2024-03-25T18:39:25.090 INFO:teuthology.orchestra.run.smithi001.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.38880.log 2024-03-25T18:39:25.090 INFO:teuthology.orchestra.run.smithi001.stderr:/var/log/ceph/ceph-client.admin.34885.log: 86.2% -- replaced with /var/log/ceph/ceph-client.admin.40846.log.gz 2024-03-25T18:39:25.090 INFO:teuthology.orchestra.run.smithi001.stderr:/var/log/ceph/ceph-client.admin.40339.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41159.log 2024-03-25T18:39:25.090 INFO:teuthology.orchestra.run.smithi001.stderr:/var/log/ceph/ceph-client.admin.38880.log: 86.9% -- replaced with /var/log/ceph/ceph-client.admin.40339.log.gz 2024-03-25T18:39:25.090 INFO:teuthology.orchestra.run.smithi001.stderr: 90.1%gzip -- replaced with /var/log/ceph/ceph-client.admin.34885.log.gz 2024-03-25T18:39:25.091 INFO:teuthology.orchestra.run.smithi001.stderr: -5 --verbose -- /var/log/ceph/ceph-client.admin.41901.log 2024-03-25T18:39:25.091 INFO:teuthology.orchestra.run.smithi001.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40963.log 2024-03-25T18:39:25.091 INFO:teuthology.orchestra.run.smithi001.stderr:/var/log/ceph/ceph-client.admin.41159.log: /var/log/ceph/ceph-client.admin.41901.log: 87.0% -- replaced with /var/log/ceph/ceph-client.admin.38880.log.gz 2024-03-25T18:39:25.091 INFO:teuthology.orchestra.run.smithi001.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41119.log 2024-03-25T18:39:25.091 INFO:teuthology.orchestra.run.smithi001.stderr: 86.9% -- replaced with /var/log/ceph/ceph-client.admin.41159.log.gz 2024-03-25T18:39:25.091 INFO:teuthology.orchestra.run.smithi001.stderr:/var/log/ceph/ceph-client.admin.40963.log: 86.3% -- replaced with /var/log/ceph/ceph-client.admin.41901.log.gz 2024-03-25T18:39:25.091 INFO:teuthology.orchestra.run.smithi001.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39622.log 2024-03-25T18:39:25.091 INFO:teuthology.orchestra.run.smithi001.stderr:/var/log/ceph/ceph-client.admin.41119.log: 86.4% -- replaced with /var/log/ceph/ceph-client.admin.40963.log.gz 2024-03-25T18:39:25.091 INFO:teuthology.orchestra.run.smithi001.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40105.log 2024-03-25T18:39:25.091 INFO:teuthology.orchestra.run.smithi001.stderr:/var/log/ceph/ceph-client.admin.39622.log: 86.6% -- replaced with /var/log/ceph/ceph-client.admin.41119.log.gz 2024-03-25T18:39:25.091 INFO:teuthology.orchestra.run.smithi001.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.42018.log 2024-03-25T18:39:25.091 INFO:teuthology.orchestra.run.smithi001.stderr: 86.2% -- replaced with /var/log/ceph/ceph-client.admin.39622.log.gz 2024-03-25T18:39:25.091 INFO:teuthology.orchestra.run.smithi001.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39675.log 2024-03-25T18:39:25.091 INFO:teuthology.orchestra.run.smithi001.stderr:/var/log/ceph/ceph-client.admin.40105.log: /var/log/ceph/ceph-client.admin.42018.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.38958.log 2024-03-25T18:39:25.091 INFO:teuthology.orchestra.run.smithi001.stderr:/var/log/ceph/ceph-client.admin.39675.log: gzip -5 --verbose -- /var/log/ceph/ceph.audit.log 2024-03-25T18:39:25.091 INFO:teuthology.orchestra.run.smithi001.stderr: 86.6% -- replaced with /var/log/ceph/ceph-client.admin.40105.log.gz 2024-03-25T18:39:25.091 INFO:teuthology.orchestra.run.smithi001.stderr: 86.1% -- replaced with /var/log/ceph/ceph-client.admin.42018.log.gz 2024-03-25T18:39:25.092 INFO:teuthology.orchestra.run.smithi001.stderr:/var/log/ceph/ceph-client.admin.38958.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39910.log 2024-03-25T18:39:25.092 INFO:teuthology.orchestra.run.smithi001.stderr: 86.6% -- replaced with /var/log/ceph/ceph-client.admin.39675.log.gz 2024-03-25T18:39:25.092 INFO:teuthology.orchestra.run.smithi001.stderr:/var/log/ceph/ceph.audit.log: gzip 86.9% -- replaced with /var/log/ceph/ceph-client.admin.38958.log.gz -5 2024-03-25T18:39:25.092 INFO:teuthology.orchestra.run.smithi001.stderr: --verbose -- /var/log/ceph/ceph-client.admin.41002.log 2024-03-25T18:39:25.092 INFO:teuthology.orchestra.run.smithi001.stderr:/var/log/ceph/ceph-client.admin.39910.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39270.log 2024-03-25T18:39:25.092 INFO:teuthology.orchestra.run.smithi001.stderr: 86.5% -- replaced with /var/log/ceph/ceph-client.admin.39910.log.gz 2024-03-25T18:39:25.092 INFO:teuthology.orchestra.run.smithi001.stderr:/var/log/ceph/ceph-client.admin.41002.log: 93.6% -- replaced with /var/log/ceph/ceph.audit.log.gz 2024-03-25T18:39:25.092 INFO:teuthology.orchestra.run.smithi001.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40261.log 2024-03-25T18:39:25.092 INFO:teuthology.orchestra.run.smithi001.stderr:/var/log/ceph/ceph-client.admin.39270.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41471.log 2024-03-25T18:39:25.092 INFO:teuthology.orchestra.run.smithi001.stderr: 86.2% -- replaced with /var/log/ceph/ceph-client.admin.41002.log.gz 2024-03-25T18:39:25.092 INFO:teuthology.orchestra.run.smithi001.stderr:/var/log/ceph/ceph-client.admin.40261.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41080.log 2024-03-25T18:39:25.092 INFO:teuthology.orchestra.run.smithi001.stderr: 86.6% -- replaced with /var/log/ceph/ceph-client.admin.39270.log.gz 2024-03-25T18:39:25.092 INFO:teuthology.orchestra.run.smithi001.stderr: 86.8%/var/log/ceph/ceph-client.admin.41471.log: -- replaced with /var/log/ceph/ceph-client.admin.40261.log.gz 2024-03-25T18:39:25.092 INFO:teuthology.orchestra.run.smithi001.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40300.log 2024-03-25T18:39:25.092 INFO:teuthology.orchestra.run.smithi001.stderr:/var/log/ceph/ceph-client.admin.41080.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41666.log 2024-03-25T18:39:25.093 INFO:teuthology.orchestra.run.smithi001.stderr: 86.6% -- replaced with /var/log/ceph/ceph-client.admin.41471.log.gz 2024-03-25T18:39:25.093 INFO:teuthology.orchestra.run.smithi001.stderr:/var/log/ceph/ceph-client.admin.40300.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40183.log 2024-03-25T18:39:25.093 INFO:teuthology.orchestra.run.smithi001.stderr: 86.5% -- replaced with /var/log/ceph/ceph-client.admin.41080.log.gz 2024-03-25T18:39:25.093 INFO:teuthology.orchestra.run.smithi001.stderr: 86.5% -- replaced with /var/log/ceph/ceph-client.admin.40300.log.gz 2024-03-25T18:39:25.093 INFO:teuthology.orchestra.run.smithi001.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39544.log 2024-03-25T18:39:25.093 INFO:teuthology.orchestra.run.smithi001.stderr:/var/log/ceph/ceph-client.admin.41666.log: /var/log/ceph/ceph-client.admin.40183.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.38722.log 2024-03-25T18:39:25.093 INFO:teuthology.orchestra.run.smithi001.stderr: 86.6% -- replaced with /var/log/ceph/ceph-client.admin.41666.log.gz 2024-03-25T18:39:25.093 INFO:teuthology.orchestra.run.smithi001.stderr:/var/log/ceph/ceph-client.admin.39544.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.38997.log 2024-03-25T18:39:25.093 INFO:teuthology.orchestra.run.smithi001.stderr: 86.9% -- replaced with /var/log/ceph/ceph-client.admin.40183.log.gz 2024-03-25T18:39:25.093 INFO:teuthology.orchestra.run.smithi001.stderr:/var/log/ceph/ceph-client.admin.38722.log: 87.3% -- replaced with /var/log/ceph/ceph-client.admin.39544.log.gz 2024-03-25T18:39:25.093 INFO:teuthology.orchestra.run.smithi001.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39114.log 2024-03-25T18:39:25.093 INFO:teuthology.orchestra.run.smithi001.stderr:/var/log/ceph/ceph-client.admin.38997.log: 87.1% -- replaced with /var/log/ceph/ceph-client.admin.38722.log.gz 2024-03-25T18:39:25.093 INFO:teuthology.orchestra.run.smithi001.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40027.log 2024-03-25T18:39:25.093 INFO:teuthology.orchestra.run.smithi001.stderr:/var/log/ceph/ceph-client.admin.39114.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40924.log 2024-03-25T18:39:25.093 INFO:teuthology.orchestra.run.smithi001.stderr: 86.7% -- replaced with /var/log/ceph/ceph-client.admin.38997.log.gz 2024-03-25T18:39:25.094 INFO:teuthology.orchestra.run.smithi001.stderr:/var/log/ceph/ceph-client.admin.40027.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41588.log 2024-03-25T18:39:25.094 INFO:teuthology.orchestra.run.smithi001.stderr: 86.9% -- replaced with /var/log/ceph/ceph-client.admin.39114.log.gz 2024-03-25T18:39:25.094 INFO:teuthology.orchestra.run.smithi001.stderr: 86.1% -- replaced with /var/log/ceph/ceph-client.admin.40027.log.gz 2024-03-25T18:39:25.094 INFO:teuthology.orchestra.run.smithi001.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41432.log 2024-03-25T18:39:25.094 INFO:teuthology.orchestra.run.smithi001.stderr:/var/log/ceph/ceph-client.admin.40924.log: /var/log/ceph/ceph-client.admin.41588.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40768.log 2024-03-25T18:39:25.094 INFO:teuthology.orchestra.run.smithi001.stderr:/var/log/ceph/ceph-client.admin.41432.log: gzip -5 --verbose -- /var/log/ceph/ceph-osd.1.log 2024-03-25T18:39:25.094 INFO:teuthology.orchestra.run.smithi001.stderr: 87.0% -- replaced with /var/log/ceph/ceph-client.admin.40924.log.gz 2024-03-25T18:39:25.094 INFO:teuthology.orchestra.run.smithi001.stderr: 86.6% -- replaced with /var/log/ceph/ceph-client.admin.41588.log.gz 2024-03-25T18:39:25.094 INFO:teuthology.orchestra.run.smithi001.stderr:/var/log/ceph/ceph-client.admin.40768.log: 86.1% -- replaced with /var/log/ceph/ceph-client.admin.41432.log.gz 2024-03-25T18:39:25.094 INFO:teuthology.orchestra.run.smithi001.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.36195.log 2024-03-25T18:39:25.094 INFO:teuthology.orchestra.run.smithi001.stderr:/var/log/ceph/ceph-osd.1.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40222.log 2024-03-25T18:39:25.094 INFO:teuthology.orchestra.run.smithi001.stderr: 86.5% -- replaced with /var/log/ceph/ceph-client.admin.40768.log.gz 2024-03-25T18:39:25.094 INFO:teuthology.orchestra.run.smithi001.stderr:/var/log/ceph/ceph-client.admin.36195.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39832.log 2024-03-25T18:39:25.094 INFO:teuthology.orchestra.run.smithi001.stderr:/var/log/ceph/ceph-client.admin.40222.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39388.log 2024-03-25T18:39:25.095 INFO:teuthology.orchestra.run.smithi001.stderr: 87.4% -- replaced with /var/log/ceph/ceph-client.admin.36195.log.gz 2024-03-25T18:39:25.095 INFO:teuthology.orchestra.run.smithi001.stderr:/var/log/ceph/ceph-client.admin.39832.log: 87.2% -- replaced with /var/log/ceph/ceph-client.admin.40222.log.gz 2024-03-25T18:39:25.095 INFO:teuthology.orchestra.run.smithi001.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40807.log 2024-03-25T18:39:25.095 INFO:teuthology.orchestra.run.smithi001.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39754.log 2024-03-25T18:39:25.095 INFO:teuthology.orchestra.run.smithi001.stderr:/var/log/ceph/ceph-client.admin.39388.log: 86.8% -- replaced with /var/log/ceph/ceph-client.admin.39832.log.gz 2024-03-25T18:39:25.095 INFO:teuthology.orchestra.run.smithi001.stderr:/var/log/ceph/ceph-client.admin.40807.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.35348.log 2024-03-25T18:39:25.095 INFO:teuthology.orchestra.run.smithi001.stderr:/var/log/ceph/ceph-client.admin.39754.log: 87.1% -- replaced with /var/log/ceph/ceph-client.admin.39388.log.gz 2024-03-25T18:39:25.095 INFO:teuthology.orchestra.run.smithi001.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-osd.3.log 2024-03-25T18:39:25.095 INFO:teuthology.orchestra.run.smithi001.stderr: 86.0% -- replaced with /var/log/ceph/ceph-client.admin.40807.log.gz 2024-03-25T18:39:25.095 INFO:teuthology.orchestra.run.smithi001.stderr:/var/log/ceph/ceph-client.admin.35348.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40495.log 2024-03-25T18:39:25.095 INFO:teuthology.orchestra.run.smithi001.stderr: 86.8% -- replaced with /var/log/ceph/ceph-client.admin.39754.log.gz 2024-03-25T18:39:25.095 INFO:teuthology.orchestra.run.smithi001.stderr:/var/log/ceph/ceph-osd.3.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40729.log 2024-03-25T18:39:25.095 INFO:teuthology.orchestra.run.smithi001.stderr: 87.5% -- replaced with /var/log/ceph/ceph-client.admin.35348.log.gz 2024-03-25T18:39:25.095 INFO:teuthology.orchestra.run.smithi001.stderr:gzip/var/log/ceph/ceph-client.admin.40495.log: -5 --verbose -- /var/log/ceph/ceph-client.admin.39583.log 2024-03-25T18:39:25.096 INFO:teuthology.orchestra.run.smithi001.stderr:/var/log/ceph/ceph-client.admin.40729.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39036.log 2024-03-25T18:39:25.096 INFO:teuthology.orchestra.run.smithi001.stderr: 86.8% -- replaced with /var/log/ceph/ceph-client.admin.40495.log.gz 2024-03-25T18:39:25.096 INFO:teuthology.orchestra.run.smithi001.stderr:/var/log/ceph/ceph-client.admin.39583.log: gzip -5 --verbose -- 87.0% -- replaced with /var/log/ceph/ceph-client.admin.40729.log.gz 2024-03-25T18:39:25.096 INFO:teuthology.orchestra.run.smithi001.stderr: /var/log/ceph/ceph-client.admin.39192.log 2024-03-25T18:39:25.096 INFO:teuthology.orchestra.run.smithi001.stderr:/var/log/ceph/ceph-client.admin.39036.log: 87.0% -- replaced with /var/log/ceph/ceph-client.admin.39583.log.gz 2024-03-25T18:39:25.096 INFO:teuthology.orchestra.run.smithi001.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41510.log 2024-03-25T18:39:25.096 INFO:teuthology.orchestra.run.smithi001.stderr:/var/log/ceph/ceph-client.admin.39192.log: gzip -5 --verbose 86.9% -- replaced with /var/log/ceph/ceph-client.admin.39036.log.gz 2024-03-25T18:39:25.096 INFO:teuthology.orchestra.run.smithi001.stderr: -- /var/log/ceph/ceph-client.admin.41940.log 2024-03-25T18:39:25.096 INFO:teuthology.orchestra.run.smithi001.stderr:/var/log/ceph/ceph-client.admin.41510.log: 86.7% -- replaced with /var/log/ceph/ceph-client.admin.39192.log.gz 2024-03-25T18:39:25.096 INFO:teuthology.orchestra.run.smithi001.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40378.log 2024-03-25T18:39:25.096 INFO:teuthology.orchestra.run.smithi001.stderr:/var/log/ceph/ceph-client.admin.41940.log: gzip -5 --verbose 86.9% -- /var/log/ceph/ceph-client.admin.41822.log -- replaced with /var/log/ceph/ceph-client.admin.41510.log.gz 2024-03-25T18:39:25.096 INFO:teuthology.orchestra.run.smithi001.stderr: 2024-03-25T18:39:25.096 INFO:teuthology.orchestra.run.smithi001.stderr:/var/log/ceph/ceph-client.admin.40378.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39988.log 2024-03-25T18:39:25.096 INFO:teuthology.orchestra.run.smithi001.stderr: 86.7% -- replaced with /var/log/ceph/ceph-client.admin.41940.log.gz 2024-03-25T18:39:25.096 INFO:teuthology.orchestra.run.smithi001.stderr: 86.9%/var/log/ceph/ceph-client.admin.41822.log: -- replaced with /var/log/ceph/ceph-client.admin.40378.log.gz 2024-03-25T18:39:25.097 INFO:teuthology.orchestra.run.smithi001.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.38802.log 2024-03-25T18:39:25.097 INFO:teuthology.orchestra.run.smithi001.stderr:/var/log/ceph/ceph-client.admin.39988.log: gzip -5 --verbose -- /var/log/ceph/ceph-mgr.y.log 2024-03-25T18:39:25.097 INFO:teuthology.orchestra.run.smithi001.stderr: 86.8% -- replaced with /var/log/ceph/ceph-client.admin.41822.log.gz 2024-03-25T18:39:25.097 INFO:teuthology.orchestra.run.smithi001.stderr:/var/log/ceph/ceph-client.admin.38802.log: 86.9% -- replaced with /var/log/ceph/ceph-client.admin.39988.log.gz 2024-03-25T18:39:25.097 INFO:teuthology.orchestra.run.smithi001.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.35215.log 2024-03-25T18:39:25.097 INFO:teuthology.orchestra.run.smithi001.stderr:/var/log/ceph/ceph-mgr.y.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39349.log 2024-03-25T18:39:25.097 INFO:teuthology.orchestra.run.smithi001.stderr: 87.3% -- replaced with /var/log/ceph/ceph-client.admin.38802.log.gz 2024-03-25T18:39:25.097 INFO:teuthology.orchestra.run.smithi001.stderr:/var/log/ceph/ceph-client.admin.35215.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40651.log 2024-03-25T18:39:25.097 INFO:teuthology.orchestra.run.smithi001.stderr:/var/log/ceph/ceph-client.admin.39349.log: 86.1% -- replaced with /var/log/ceph/ceph-client.admin.35215.log.gz 2024-03-25T18:39:25.097 INFO:teuthology.orchestra.run.smithi001.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41783.log 2024-03-25T18:39:25.097 INFO:teuthology.orchestra.run.smithi001.stderr:/var/log/ceph/ceph-client.admin.40651.log: 86.4% -- replaced with /var/log/ceph/ceph-client.admin.39349.log.gz 2024-03-25T18:39:25.097 INFO:teuthology.orchestra.run.smithi001.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39715.log 2024-03-25T18:39:25.097 INFO:teuthology.orchestra.run.smithi001.stderr:/var/log/ceph/ceph-client.admin.41783.log: 86.2%gzip -5 --verbose -- /var/log/ceph/ceph-osd.0.log 2024-03-25T18:39:25.097 INFO:teuthology.orchestra.run.smithi001.stderr: -- replaced with /var/log/ceph/ceph-client.admin.40651.log.gz 2024-03-25T18:39:25.097 INFO:teuthology.orchestra.run.smithi001.stderr:/var/log/ceph/ceph-client.admin.39715.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41315.log 2024-03-25T18:39:25.098 INFO:teuthology.orchestra.run.smithi001.stderr: 87.0% -- replaced with /var/log/ceph/ceph-client.admin.41783.log.gz 2024-03-25T18:39:25.098 INFO:teuthology.orchestra.run.smithi001.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40534.log 2024-03-25T18:39:25.098 INFO:teuthology.orchestra.run.smithi001.stderr: 86.7% -- replaced with /var/log/ceph/ceph-client.admin.39715.log.gz/var/log/ceph/ceph-osd.0.log: 2024-03-25T18:39:25.098 INFO:teuthology.orchestra.run.smithi001.stderr:/var/log/ceph/ceph-client.admin.41315.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39505.log 2024-03-25T18:39:25.098 INFO:teuthology.orchestra.run.smithi001.stderr:/var/log/ceph/ceph-client.admin.40534.log: gzip -5 --verbose -- /var/log/ceph/ceph.log 2024-03-25T18:39:25.098 INFO:teuthology.orchestra.run.smithi001.stderr: 86.4% -- replaced with /var/log/ceph/ceph-client.admin.41315.log.gz/var/log/ceph/ceph-client.admin.39505.log: 2024-03-25T18:39:25.098 INFO:teuthology.orchestra.run.smithi001.stderr: 86.3% -- replaced with /var/log/ceph/ceph-client.admin.40534.log.gz 2024-03-25T18:39:25.098 INFO:teuthology.orchestra.run.smithi001.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41744.log 2024-03-25T18:39:25.098 INFO:teuthology.orchestra.run.smithi001.stderr: 86.9% -- replaced with /var/log/ceph/ceph-client.admin.39505.log.gz 2024-03-25T18:39:25.098 INFO:teuthology.orchestra.run.smithi001.stderr:/var/log/ceph/ceph.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.38841.log 2024-03-25T18:39:25.098 INFO:teuthology.orchestra.run.smithi001.stderr:/var/log/ceph/ceph-client.admin.41744.log: gzip -5 --verbose -- /var/log/ceph/ceph-osd.2.log 2024-03-25T18:39:25.098 INFO:teuthology.orchestra.run.smithi001.stderr: 86.0%/var/log/ceph/ceph-client.admin.38841.log: -- replaced with /var/log/ceph/ceph-client.admin.41744.log.gz 2024-03-25T18:39:25.098 INFO:teuthology.orchestra.run.smithi001.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41627.log 2024-03-25T18:39:25.098 INFO:teuthology.orchestra.run.smithi001.stderr: 92.9% -- replaced with /var/log/ceph/ceph.log.gz 2024-03-25T18:39:25.098 INFO:teuthology.orchestra.run.smithi001.stderr:/var/log/ceph/ceph-osd.2.log: 87.0% -- replaced with /var/log/ceph/ceph-client.admin.38841.log.gz 2024-03-25T18:39:25.099 INFO:teuthology.orchestra.run.smithi001.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40573.log 2024-03-25T18:39:25.099 INFO:teuthology.orchestra.run.smithi001.stderr:/var/log/ceph/ceph-client.admin.41627.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.35188.log 2024-03-25T18:39:25.099 INFO:teuthology.orchestra.run.smithi001.stderr:/var/log/ceph/ceph-client.admin.40573.log: 86.6% -- replaced with /var/log/ceph/ceph-client.admin.41627.log.gz 2024-03-25T18:39:25.099 INFO:teuthology.orchestra.run.smithi001.stderr:gzip -5 --verbose -- /var/log/ceph/ceph.tmp-client.admin.29034.log 2024-03-25T18:39:25.099 INFO:teuthology.orchestra.run.smithi001.stderr:/var/log/ceph/ceph-client.admin.35188.log: 85.9% -- replaced with /var/log/ceph/ceph-client.admin.40573.log.gz 2024-03-25T18:39:25.099 INFO:teuthology.orchestra.run.smithi001.stderr: 86.3% -- replaced with /var/log/ceph/ceph-client.admin.35188.log.gz 2024-03-25T18:39:25.099 INFO:teuthology.orchestra.run.smithi001.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.38761.log 2024-03-25T18:39:25.099 INFO:teuthology.orchestra.run.smithi001.stderr:/var/log/ceph/ceph.tmp-client.admin.29034.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41393.log 2024-03-25T18:39:25.099 INFO:teuthology.orchestra.run.smithi001.stderr: 0.0% -- replaced with /var/log/ceph/ceph.tmp-client.admin.29034.log.gz 2024-03-25T18:39:25.099 INFO:teuthology.orchestra.run.smithi001.stderr:/var/log/ceph/ceph-client.admin.38761.log: 85.9% -- replaced with /var/log/ceph/ceph-client.admin.38761.log.gz 2024-03-25T18:39:25.099 INFO:teuthology.orchestra.run.smithi001.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39309.log 2024-03-25T18:39:25.099 INFO:teuthology.orchestra.run.smithi001.stderr:/var/log/ceph/ceph-client.admin.41393.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39793.log 2024-03-25T18:39:25.099 INFO:teuthology.orchestra.run.smithi001.stderr:/var/log/ceph/ceph-client.admin.39309.log: 86.0% -- replaced with /var/log/ceph/ceph-client.admin.41393.log.gz 2024-03-25T18:39:25.099 INFO:teuthology.orchestra.run.smithi001.stderr: 93.9% -- replaced with /var/log/ceph/ceph-mgr.y.log.gzgzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41276.log 2024-03-25T18:39:25.099 INFO:teuthology.orchestra.run.smithi001.stderr: 2024-03-25T18:39:25.100 INFO:teuthology.orchestra.run.smithi001.stderr:/var/log/ceph/ceph-client.admin.39793.log: 86.4%gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40066.log 2024-03-25T18:39:25.100 INFO:teuthology.orchestra.run.smithi001.stderr: -- replaced with /var/log/ceph/ceph-client.admin.39309.log.gz 2024-03-25T18:39:25.100 INFO:teuthology.orchestra.run.smithi001.stderr:/var/log/ceph/ceph-client.admin.41276.log: 86.2% -- replaced with /var/log/ceph/ceph-client.admin.39793.log.gz 2024-03-25T18:39:25.100 INFO:teuthology.orchestra.run.smithi001.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41198.log 2024-03-25T18:39:25.100 INFO:teuthology.orchestra.run.smithi001.stderr: 86.3% -- replaced with /var/log/ceph/ceph-client.admin.41276.log.gz 2024-03-25T18:39:25.100 INFO:teuthology.orchestra.run.smithi001.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.38919.log 2024-03-25T18:39:25.100 INFO:teuthology.orchestra.run.smithi001.stderr:/var/log/ceph/ceph-client.admin.40066.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39466.log 2024-03-25T18:39:25.100 INFO:teuthology.orchestra.run.smithi001.stderr:/var/log/ceph/ceph-client.admin.41198.log: 86.9% -- replaced with /var/log/ceph/ceph-client.admin.40066.log.gz/var/log/ceph/ceph-client.admin.38919.log: 2024-03-25T18:39:25.100 INFO:teuthology.orchestra.run.smithi001.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39949.log 2024-03-25T18:39:25.100 INFO:teuthology.orchestra.run.smithi001.stderr: 87.2% -- replaced with /var/log/ceph/ceph-client.admin.38919.log.gz 2024-03-25T18:39:25.100 INFO:teuthology.orchestra.run.smithi001.stderr:/var/log/ceph/ceph-client.admin.39466.log: 87.1% -- replaced with /var/log/ceph/ceph-client.admin.41198.log.gz 2024-03-25T18:39:25.100 INFO:teuthology.orchestra.run.smithi001.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41705.log 2024-03-25T18:39:25.100 INFO:teuthology.orchestra.run.smithi001.stderr: 86.9%/var/log/ceph/ceph-client.admin.39949.log: -- replaced with /var/log/ceph/ceph-client.admin.39466.log.gz 2024-03-25T18:39:25.100 INFO:teuthology.orchestra.run.smithi001.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40612.log 2024-03-25T18:39:25.100 INFO:teuthology.orchestra.run.smithi001.stderr: 86.3% -- replaced with /var/log/ceph/ceph-client.admin.39949.log.gz/var/log/ceph/ceph-client.admin.41705.log: 2024-03-25T18:39:25.100 INFO:teuthology.orchestra.run.smithi001.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41237.log 2024-03-25T18:39:25.101 INFO:teuthology.orchestra.run.smithi001.stderr: 85.6%/var/log/ceph/ceph-client.admin.40612.log: -- replaced with /var/log/ceph/ceph-client.admin.41705.log.gz 2024-03-25T18:39:25.101 INFO:teuthology.orchestra.run.smithi001.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39871.log 2024-03-25T18:39:25.101 INFO:teuthology.orchestra.run.smithi001.stderr: 86.4%gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41041.log 2024-03-25T18:39:25.101 INFO:teuthology.orchestra.run.smithi001.stderr: -- replaced with /var/log/ceph/ceph-client.admin.40612.log.gz 2024-03-25T18:39:25.101 INFO:teuthology.orchestra.run.smithi001.stderr:/var/log/ceph/ceph-client.admin.41237.log: /var/log/ceph/ceph-client.admin.39871.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40690.log 2024-03-25T18:39:25.101 INFO:teuthology.orchestra.run.smithi001.stderr:/var/log/ceph/ceph-client.admin.41041.log: 86.4% -- replaced with /var/log/ceph/ceph-client.admin.41237.log.gz 2024-03-25T18:39:25.101 INFO:teuthology.orchestra.run.smithi001.stderr: 86.3% -- replaced with /var/log/ceph/ceph-client.admin.39871.log.gz 2024-03-25T18:39:25.101 INFO:teuthology.orchestra.run.smithi001.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40456.log 2024-03-25T18:39:25.101 INFO:teuthology.orchestra.run.smithi001.stderr: 85.8% -- replaced with /var/log/ceph/ceph-client.admin.41041.log.gz 2024-03-25T18:39:25.101 INFO:teuthology.orchestra.run.smithi001.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39075.log 2024-03-25T18:39:25.101 INFO:teuthology.orchestra.run.smithi001.stderr:/var/log/ceph/ceph-client.admin.40690.log: /var/log/ceph/ceph-client.admin.40456.log: gzip -5 --verbose -- /var/log/ceph/ceph-mon.c.log 2024-03-25T18:39:25.101 INFO:teuthology.orchestra.run.smithi001.stderr:/var/log/ceph/ceph-client.admin.39075.log: 86.5% -- replaced with /var/log/ceph/ceph-client.admin.40690.log.gz 2024-03-25T18:39:25.101 INFO:teuthology.orchestra.run.smithi001.stderr: 86.3% -- replaced with /var/log/ceph/ceph-client.admin.40456.log.gz 2024-03-25T18:39:25.101 INFO:teuthology.orchestra.run.smithi001.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.37146.log 2024-03-25T18:39:25.101 INFO:teuthology.orchestra.run.smithi001.stderr: 86.3% -- replaced with /var/log/ceph/ceph-client.admin.39075.log.gz 2024-03-25T18:39:25.101 INFO:teuthology.orchestra.run.smithi001.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40885.log 2024-03-25T18:39:25.101 INFO:teuthology.orchestra.run.smithi001.stderr:/var/log/ceph/ceph-mon.c.log: /var/log/ceph/ceph-client.admin.37146.log: /var/log/ceph/ceph-client.admin.40885.log: 87.4% -- replaced with /var/log/ceph/ceph-client.admin.37146.log.gz 2024-03-25T18:39:25.102 INFO:teuthology.orchestra.run.smithi001.stderr: 86.3% -- replaced with /var/log/ceph/ceph-client.admin.40885.log.gz 2024-03-25T18:39:25.142 INFO:teuthology.orchestra.run.smithi115.stderr:/var/log/ceph/ceph-mon.b.log: 92.6% -- replaced with /var/log/ceph/ceph-mgr.x.log.gz 2024-03-25T18:39:25.904 INFO:teuthology.orchestra.run.smithi001.stderr: 92.9% -- replaced with /var/log/ceph/ceph-osd.3.log.gz 2024-03-25T18:39:25.913 INFO:teuthology.orchestra.run.smithi115.stderr: 93.0% -- replaced with /var/log/ceph/ceph-osd.4.log.gz 2024-03-25T18:39:25.960 INFO:teuthology.orchestra.run.smithi115.stderr: 93.0% -- replaced with /var/log/ceph/ceph-osd.5.log.gz 2024-03-25T18:39:26.198 INFO:teuthology.orchestra.run.smithi001.stderr: 92.9% -- replaced with /var/log/ceph/ceph-osd.2.log.gz 2024-03-25T18:39:26.260 INFO:teuthology.orchestra.run.smithi115.stderr: 93.0% -- replaced with /var/log/ceph/ceph-osd.6.log.gz 2024-03-25T18:39:26.288 INFO:teuthology.orchestra.run.smithi001.stderr: 92.9% -- replaced with /var/log/ceph/ceph-osd.0.log.gz 2024-03-25T18:39:26.290 INFO:teuthology.orchestra.run.smithi115.stderr: 93.0% -- replaced with /var/log/ceph/ceph-osd.7.log.gz 2024-03-25T18:39:26.294 INFO:teuthology.orchestra.run.smithi001.stderr: 93.0% -- replaced with /var/log/ceph/ceph-osd.1.log.gz 2024-03-25T18:39:30.000 INFO:teuthology.orchestra.run.smithi115.stderr: 90.6% -- replaced with /var/log/ceph/ceph-mon.b.log.gz 2024-03-25T18:39:30.002 INFO:teuthology.orchestra.run.smithi115.stderr: 2024-03-25T18:39:30.002 INFO:teuthology.orchestra.run.smithi115.stderr:real 0m4.955s 2024-03-25T18:39:30.002 INFO:teuthology.orchestra.run.smithi115.stderr:user 0m8.901s 2024-03-25T18:39:30.002 INFO:teuthology.orchestra.run.smithi115.stderr:sys 0m0.319s 2024-03-25T18:39:30.115 INFO:teuthology.orchestra.run.smithi001.stderr: 90.7% -- replaced with /var/log/ceph/ceph-mon.c.log.gz 2024-03-25T18:39:30.115 INFO:teuthology.orchestra.run.smithi001.stderr: 90.5% -- replaced with /var/log/ceph/ceph-mon.a.log.gz 2024-03-25T18:39:30.117 INFO:teuthology.orchestra.run.smithi001.stderr: 2024-03-25T18:39:30.117 INFO:teuthology.orchestra.run.smithi001.stderr:real 0m5.098s 2024-03-25T18:39:30.117 INFO:teuthology.orchestra.run.smithi001.stderr:user 0m14.205s 2024-03-25T18:39:30.117 INFO:teuthology.orchestra.run.smithi001.stderr:sys 0m0.522s 2024-03-25T18:39:30.117 INFO:tasks.ceph:Archiving logs... 2024-03-25T18:39:30.117 DEBUG:teuthology.misc:Transferring archived files from smithi001:/var/log/ceph to /home/teuthworker/archive/amathuri-2024-03-25_13:28:33-rados:verify-main-distro-default-smithi/7621279/remote/smithi001/log 2024-03-25T18:39:30.118 DEBUG:teuthology.orchestra.run.smithi001:> sudo tar c -f - -C /var/log/ceph -- . 2024-03-25T18:39:32.584 DEBUG:teuthology.misc:Transferring archived files from smithi115:/var/log/ceph to /home/teuthworker/archive/amathuri-2024-03-25_13:28:33-rados:verify-main-distro-default-smithi/7621279/remote/smithi115/log 2024-03-25T18:39:32.585 DEBUG:teuthology.orchestra.run.smithi115:> sudo tar c -f - -C /var/log/ceph -- . 2024-03-25T18:39:34.050 ERROR:teuthology.run_tasks:Saw exception from tasks. Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_ceph-c_1fb5d10608dccee73edd6ef8a2c7b804a097c324/qa/tasks/ceph.py", line 329, in valgrind_post yield File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/contextutil.py", line 30, in nested vars.append(enter()) File "/usr/lib/python3.8/contextlib.py", line 113, in __enter__ return next(self.gen) File "/home/teuthworker/src/git.ceph.com_ceph-c_1fb5d10608dccee73edd6ef8a2c7b804a097c324/qa/tasks/ceph.py", line 401, in create_rbd_pool teuthology.wait_until_osds_up( File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/misc.py", line 861, in wait_until_osds_up while proceed(): File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/contextutil.py", line 134, in __call__ raise MaxWhileTries(error_msg) teuthology.exceptions.MaxWhileTries: reached maximum tries (91) after waiting for 540 seconds During handling of the above exception, another exception occurred: Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/run_tasks.py", line 109, in run_tasks manager.__enter__() File "/usr/lib/python3.8/contextlib.py", line 113, in __enter__ return next(self.gen) File "/home/teuthworker/src/git.ceph.com_ceph-c_1fb5d10608dccee73edd6ef8a2c7b804a097c324/qa/tasks/ceph.py", line 1913, in task with contextutil.nested(*subtasks): File "/usr/lib/python3.8/contextlib.py", line 113, in __enter__ return next(self.gen) File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/contextutil.py", line 54, in nested raise exc[1] File "/usr/lib/python3.8/contextlib.py", line 131, in __exit__ self.gen.throw(type, value, traceback) File "/home/teuthworker/src/git.ceph.com_ceph-c_1fb5d10608dccee73edd6ef8a2c7b804a097c324/qa/tasks/ceph.py", line 252, in ceph_log yield File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/contextutil.py", line 46, in nested if exit(*exc): File "/usr/lib/python3.8/contextlib.py", line 131, in __exit__ self.gen.throw(type, value, traceback) File "/home/teuthworker/src/git.ceph.com_ceph-c_1fb5d10608dccee73edd6ef8a2c7b804a097c324/qa/tasks/ceph.py", line 346, in valgrind_post raise valgrind_exception Exception: valgrind error: Leak_StillReachable operator new[](unsigned long) UnknownInlinedFun UnknownInlinedFun 2024-03-25T18:39:34.259 ERROR:teuthology.util.sentry: Sentry event: https://sentry.ceph.com/organizations/ceph/?query=d2944dc623e541a0b710f1474f198101 Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_ceph-c_1fb5d10608dccee73edd6ef8a2c7b804a097c324/qa/tasks/ceph.py", line 329, in valgrind_post yield File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/contextutil.py", line 30, in nested vars.append(enter()) File "/usr/lib/python3.8/contextlib.py", line 113, in __enter__ return next(self.gen) File "/home/teuthworker/src/git.ceph.com_ceph-c_1fb5d10608dccee73edd6ef8a2c7b804a097c324/qa/tasks/ceph.py", line 401, in create_rbd_pool teuthology.wait_until_osds_up( File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/misc.py", line 861, in wait_until_osds_up while proceed(): File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/contextutil.py", line 134, in __call__ raise MaxWhileTries(error_msg) teuthology.exceptions.MaxWhileTries: reached maximum tries (91) after waiting for 540 seconds During handling of the above exception, another exception occurred: Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/run_tasks.py", line 109, in run_tasks manager.__enter__() File "/usr/lib/python3.8/contextlib.py", line 113, in __enter__ return next(self.gen) File "/home/teuthworker/src/git.ceph.com_ceph-c_1fb5d10608dccee73edd6ef8a2c7b804a097c324/qa/tasks/ceph.py", line 1913, in task with contextutil.nested(*subtasks): File "/usr/lib/python3.8/contextlib.py", line 113, in __enter__ return next(self.gen) File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/contextutil.py", line 54, in nested raise exc[1] File "/usr/lib/python3.8/contextlib.py", line 131, in __exit__ self.gen.throw(type, value, traceback) File "/home/teuthworker/src/git.ceph.com_ceph-c_1fb5d10608dccee73edd6ef8a2c7b804a097c324/qa/tasks/ceph.py", line 252, in ceph_log yield File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/contextutil.py", line 46, in nested if exit(*exc): File "/usr/lib/python3.8/contextlib.py", line 131, in __exit__ self.gen.throw(type, value, traceback) File "/home/teuthworker/src/git.ceph.com_ceph-c_1fb5d10608dccee73edd6ef8a2c7b804a097c324/qa/tasks/ceph.py", line 346, in valgrind_post raise valgrind_exception Exception: valgrind error: Leak_StillReachable operator new[](unsigned long) UnknownInlinedFun UnknownInlinedFun 2024-03-25T18:39:34.264 DEBUG:teuthology.run_tasks:Unwinding manager ceph 2024-03-25T18:39:34.273 DEBUG:teuthology.run_tasks:Unwinding manager install 2024-03-25T18:39:34.289 INFO:teuthology.task.install.util:Removing shipped files: /home/ubuntu/cephtest/valgrind.supp /usr/bin/daemon-helper /usr/bin/adjust-ulimits /usr/bin/stdin-killer... 2024-03-25T18:39:34.290 DEBUG:teuthology.orchestra.run.smithi001:> sudo rm -f -- /home/ubuntu/cephtest/valgrind.supp /usr/bin/daemon-helper /usr/bin/adjust-ulimits /usr/bin/stdin-killer 2024-03-25T18:39:34.305 DEBUG:teuthology.orchestra.run.smithi115:> sudo rm -f -- /home/ubuntu/cephtest/valgrind.supp /usr/bin/daemon-helper /usr/bin/adjust-ulimits /usr/bin/stdin-killer 2024-03-25T18:39:34.382 DEBUG:teuthology.run_tasks:Unwinding manager clock 2024-03-25T18:39:34.391 INFO:teuthology.task.clock:Checking final clock skew... 2024-03-25T18:39:34.391 DEBUG:teuthology.orchestra.run.smithi001:> PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2024-03-25T18:39:34.394 DEBUG:teuthology.orchestra.run.smithi115:> PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2024-03-25T18:39:34.407 INFO:teuthology.orchestra.run.smithi001.stderr:bash: line 1: ntpq: command not found 2024-03-25T18:39:34.410 INFO:teuthology.orchestra.run.smithi115.stderr:bash: line 1: ntpq: command not found 2024-03-25T18:39:34.412 INFO:teuthology.orchestra.run.smithi001.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2024-03-25T18:39:34.412 INFO:teuthology.orchestra.run.smithi001.stdout:=============================================================================== 2024-03-25T18:39:34.413 INFO:teuthology.orchestra.run.smithi001.stdout:^+ hv01.front.sepia.ceph.com 2 6 377 2 +499us[ +499us] +/- 77ms 2024-03-25T18:39:34.413 INFO:teuthology.orchestra.run.smithi001.stdout:^* hv02.front.sepia.ceph.com 3 6 377 20 -1052us[-1052us] +/- 62ms 2024-03-25T18:39:34.413 INFO:teuthology.orchestra.run.smithi001.stdout:^+ hv03.front.sepia.ceph.com 3 6 377 1 +790us[ +790us] +/- 67ms 2024-03-25T18:39:34.413 INFO:teuthology.orchestra.run.smithi001.stdout:^? hv04.front.sepia.ceph.com 0 10 0 - +0ns[ +0ns] +/- 0ns 2024-03-25T18:39:34.415 INFO:teuthology.orchestra.run.smithi115.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2024-03-25T18:39:34.415 INFO:teuthology.orchestra.run.smithi115.stdout:=============================================================================== 2024-03-25T18:39:34.415 INFO:teuthology.orchestra.run.smithi115.stdout:^+ hv01.front.sepia.ceph.com 2 6 377 1 +473us[ +473us] +/- 77ms 2024-03-25T18:39:34.415 INFO:teuthology.orchestra.run.smithi115.stdout:^* hv02.front.sepia.ceph.com 3 6 377 3 -1075us[-1076us] +/- 63ms 2024-03-25T18:39:34.415 INFO:teuthology.orchestra.run.smithi115.stdout:^+ hv03.front.sepia.ceph.com 3 6 377 8 +756us[ +754us] +/- 67ms 2024-03-25T18:39:34.415 INFO:teuthology.orchestra.run.smithi115.stdout:^? hv04.front.sepia.ceph.com 0 10 0 - +0ns[ +0ns] +/- 0ns 2024-03-25T18:39:34.416 DEBUG:teuthology.run_tasks:Unwinding manager ansible.cephlab 2024-03-25T18:39:34.452 INFO:teuthology.task.ansible:Skipping ansible cleanup... 2024-03-25T18:39:34.452 DEBUG:teuthology.run_tasks:Unwinding manager selinux 2024-03-25T18:39:34.460 DEBUG:teuthology.orchestra.run.smithi001:> mkdir /home/ubuntu/cephtest/archive/audit && sudo cp /var/log/audit/audit.log /home/ubuntu/cephtest/archive/audit && sudo chown $USER /home/ubuntu/cephtest/archive/audit/audit.log && gzip /home/ubuntu/cephtest/archive/audit/audit.log 2024-03-25T18:39:34.533 DEBUG:teuthology.orchestra.run.smithi115:> mkdir /home/ubuntu/cephtest/archive/audit && sudo cp /var/log/audit/audit.log /home/ubuntu/cephtest/archive/audit && sudo chown $USER /home/ubuntu/cephtest/archive/audit/audit.log && gzip /home/ubuntu/cephtest/archive/audit/audit.log 2024-03-25T18:39:34.605 DEBUG:teuthology.orchestra.run.smithi001:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 2024-03-25T18:39:34.637 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-25T18:39:34.637 DEBUG:teuthology.orchestra.run.smithi115:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 2024-03-25T18:39:34.668 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-25T18:39:34.668 DEBUG:teuthology.run_tasks:Unwinding manager pcp 2024-03-25T18:39:34.680 DEBUG:teuthology.run_tasks:Unwinding manager internal.timer 2024-03-25T18:39:34.689 INFO:teuthology.task.internal:Duration was 7898.645892 seconds 2024-03-25T18:39:34.690 DEBUG:teuthology.run_tasks:Unwinding manager internal.syslog 2024-03-25T18:39:34.699 INFO:teuthology.task.internal.syslog:Shutting down syslog monitoring... 2024-03-25T18:39:34.699 DEBUG:teuthology.orchestra.run.smithi001:> sudo rm -f -- /etc/rsyslog.d/80-cephtest.conf && sudo service rsyslog restart 2024-03-25T18:39:34.701 DEBUG:teuthology.orchestra.run.smithi115:> sudo rm -f -- /etc/rsyslog.d/80-cephtest.conf && sudo service rsyslog restart 2024-03-25T18:39:34.745 INFO:teuthology.orchestra.run.smithi001.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2024-03-25T18:39:34.753 INFO:teuthology.orchestra.run.smithi115.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2024-03-25T18:39:35.037 INFO:teuthology.task.internal.syslog:Checking logs for errors... 2024-03-25T18:39:35.037 DEBUG:teuthology.task.internal.syslog:Checking ubuntu@smithi001.front.sepia.ceph.com 2024-03-25T18:39:35.038 DEBUG:teuthology.orchestra.run.smithi001:> egrep --binary-files=text '\bBUG\b|\bINFO\b|\bDEADLOCK\b' /home/ubuntu/cephtest/archive/syslog/kern.log | grep -v 'task .* blocked for more than .* seconds' | grep -v 'lockdep is turned off' | grep -v 'trying to register non-static key' | grep -v 'DEBUG: fsize' | grep -v CRON | grep -v 'BUG: bad unlock balance detected' | grep -v 'inconsistent lock state' | grep -v '*** DEADLOCK ***' | grep -v 'INFO: possible irq lock inversion dependency detected' | grep -v 'INFO: NMI handler (perf_event_nmi_handler) took too long to run' | grep -v 'INFO: recovery required on readonly' | grep -v 'ceph-create-keys: INFO' | grep -v INFO:ceph-create-keys | grep -v 'Loaded datasource DataSourceOpenStack' | grep -v 'container-storage-setup: INFO: Volume group backing root filesystem could not be determined' | egrep -v '\bsalt-master\b|\bsalt-minion\b|\bsalt-api\b' | grep -v ceph-crash | egrep -v '\btcmu-runner\b.*\bINFO\b' | head -n 1 2024-03-25T18:39:35.058 DEBUG:teuthology.task.internal.syslog:Checking ubuntu@smithi115.front.sepia.ceph.com 2024-03-25T18:39:35.058 DEBUG:teuthology.orchestra.run.smithi115:> egrep --binary-files=text '\bBUG\b|\bINFO\b|\bDEADLOCK\b' /home/ubuntu/cephtest/archive/syslog/kern.log | grep -v 'task .* blocked for more than .* seconds' | grep -v 'lockdep is turned off' | grep -v 'trying to register non-static key' | grep -v 'DEBUG: fsize' | grep -v CRON | grep -v 'BUG: bad unlock balance detected' | grep -v 'inconsistent lock state' | grep -v '*** DEADLOCK ***' | grep -v 'INFO: possible irq lock inversion dependency detected' | grep -v 'INFO: NMI handler (perf_event_nmi_handler) took too long to run' | grep -v 'INFO: recovery required on readonly' | grep -v 'ceph-create-keys: INFO' | grep -v INFO:ceph-create-keys | grep -v 'Loaded datasource DataSourceOpenStack' | grep -v 'container-storage-setup: INFO: Volume group backing root filesystem could not be determined' | egrep -v '\bsalt-master\b|\bsalt-minion\b|\bsalt-api\b' | grep -v ceph-crash | egrep -v '\btcmu-runner\b.*\bINFO\b' | head -n 1 2024-03-25T18:39:35.085 INFO:teuthology.task.internal.syslog:Compressing syslogs... 2024-03-25T18:39:35.086 DEBUG:teuthology.orchestra.run.smithi001:> find /home/ubuntu/cephtest/archive/syslog -name '*.log' -print0 | sudo xargs -0 --no-run-if-empty -- gzip -- 2024-03-25T18:39:35.101 DEBUG:teuthology.orchestra.run.smithi115:> find /home/ubuntu/cephtest/archive/syslog -name '*.log' -print0 | sudo xargs -0 --no-run-if-empty -- gzip -- 2024-03-25T18:39:35.154 INFO:teuthology.task.internal.syslog:Gathering journactl -b0... 2024-03-25T18:39:35.155 DEBUG:teuthology.orchestra.run.smithi001:> sudo journalctl -b0 | gzip -9 > /home/ubuntu/cephtest/archive/syslog/journalctl-b0.gz 2024-03-25T18:39:35.172 DEBUG:teuthology.orchestra.run.smithi115:> sudo journalctl -b0 | gzip -9 > /home/ubuntu/cephtest/archive/syslog/journalctl-b0.gz 2024-03-25T18:39:35.298 DEBUG:teuthology.run_tasks:Unwinding manager internal.sudo 2024-03-25T18:39:35.314 INFO:teuthology.task.internal:Restoring /etc/sudoers... 2024-03-25T18:39:35.315 DEBUG:teuthology.orchestra.run.smithi001:> sudo mv -f /etc/sudoers.orig.teuthology /etc/sudoers 2024-03-25T18:39:35.342 DEBUG:teuthology.orchestra.run.smithi115:> sudo mv -f /etc/sudoers.orig.teuthology /etc/sudoers 2024-03-25T18:39:35.369 DEBUG:teuthology.run_tasks:Unwinding manager internal.coredump 2024-03-25T18:39:35.378 DEBUG:teuthology.orchestra.run.smithi001:> sudo sysctl -w kernel.core_pattern=core && sudo bash -c 'for f in `find /home/ubuntu/cephtest/archive/coredump -type f`; do file $f | grep -q systemd-sysusers && rm $f || true ; done' && rmdir --ignore-fail-on-non-empty -- /home/ubuntu/cephtest/archive/coredump 2024-03-25T18:39:35.384 DEBUG:teuthology.orchestra.run.smithi115:> sudo sysctl -w kernel.core_pattern=core && sudo bash -c 'for f in `find /home/ubuntu/cephtest/archive/coredump -type f`; do file $f | grep -q systemd-sysusers && rm $f || true ; done' && rmdir --ignore-fail-on-non-empty -- /home/ubuntu/cephtest/archive/coredump 2024-03-25T18:39:35.407 INFO:teuthology.orchestra.run.smithi001.stdout:kernel.core_pattern = core 2024-03-25T18:39:35.435 INFO:teuthology.orchestra.run.smithi115.stdout:kernel.core_pattern = core 2024-03-25T18:39:35.463 DEBUG:teuthology.orchestra.run.smithi001:> test -e /home/ubuntu/cephtest/archive/coredump 2024-03-25T18:39:35.487 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-25T18:39:35.487 DEBUG:teuthology.orchestra.run.smithi115:> test -e /home/ubuntu/cephtest/archive/coredump 2024-03-25T18:39:35.517 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-25T18:39:35.518 DEBUG:teuthology.run_tasks:Unwinding manager internal.archive 2024-03-25T18:39:35.528 INFO:teuthology.task.internal:Transferring archived files... 2024-03-25T18:39:35.528 DEBUG:teuthology.misc:Transferring archived files from smithi001:/home/ubuntu/cephtest/archive to /home/teuthworker/archive/amathuri-2024-03-25_13:28:33-rados:verify-main-distro-default-smithi/7621279/remote/smithi001 2024-03-25T18:39:35.529 DEBUG:teuthology.orchestra.run.smithi001:> sudo tar c -f - -C /home/ubuntu/cephtest/archive -- . 2024-03-25T18:39:35.564 DEBUG:teuthology.misc:Transferring archived files from smithi115:/home/ubuntu/cephtest/archive to /home/teuthworker/archive/amathuri-2024-03-25_13:28:33-rados:verify-main-distro-default-smithi/7621279/remote/smithi115 2024-03-25T18:39:35.565 DEBUG:teuthology.orchestra.run.smithi115:> sudo tar c -f - -C /home/ubuntu/cephtest/archive -- . 2024-03-25T18:39:35.600 INFO:teuthology.task.internal:Removing archive directory... 2024-03-25T18:39:35.600 DEBUG:teuthology.orchestra.run.smithi001:> rm -rf -- /home/ubuntu/cephtest/archive 2024-03-25T18:39:35.602 DEBUG:teuthology.orchestra.run.smithi115:> rm -rf -- /home/ubuntu/cephtest/archive 2024-03-25T18:39:35.649 DEBUG:teuthology.run_tasks:Unwinding manager internal.archive_upload 2024-03-25T18:39:35.659 INFO:teuthology.task.internal:Not uploading archives. 2024-03-25T18:39:35.660 DEBUG:teuthology.run_tasks:Unwinding manager internal.base 2024-03-25T18:39:35.670 INFO:teuthology.task.internal:Tidying up after the test... 2024-03-25T18:39:35.670 DEBUG:teuthology.orchestra.run.smithi001:> find /home/ubuntu/cephtest -ls ; rmdir -- /home/ubuntu/cephtest 2024-03-25T18:39:35.672 DEBUG:teuthology.orchestra.run.smithi115:> find /home/ubuntu/cephtest -ls ; rmdir -- /home/ubuntu/cephtest 2024-03-25T18:39:35.684 INFO:teuthology.orchestra.run.smithi001.stdout: 262166 4 drwxr-xr-x 2 ubuntu ubuntu 4096 Mar 25 18:39 /home/ubuntu/cephtest 2024-03-25T18:39:35.704 INFO:teuthology.orchestra.run.smithi115.stdout: 262166 4 drwxr-xr-x 2 ubuntu ubuntu 4096 Mar 25 18:39 /home/ubuntu/cephtest 2024-03-25T18:39:35.705 DEBUG:teuthology.run_tasks:Unwinding manager kernel 2024-03-25T18:39:35.718 DEBUG:teuthology.run_tasks:Unwinding manager console_log 2024-03-25T18:39:35.740 INFO:teuthology.nuke:Checking targets against current locks 2024-03-25T18:39:35.765 DEBUG:teuthology.nuke:shortname: smithi001 2024-03-25T18:39:35.765 INFO:teuthology.task.internal.check_lock:Checking locks... 2024-03-25T18:39:35.788 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi001.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/amathuri-2024-03-25_13:28:33-rados:verify-main-distro-default-smithi/7621279', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'centos', 'os_version': '9', 'arch': 'x86_64', 'locked': True, 'locked_since': '2024-03-25 16:18:57.127681', 'locked_by': 'scheduled_amathuri@teuthology', 'mac_address': None, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBJLX2RU8lPyaGpFOX5F2XcKa3PqVDhgC0eHcD7GRZ/nAnWQ6WgV2gQWu4nRS1XboDX2zvaMZYEkHmNn70BXWi7o='} 2024-03-25T18:39:35.789 DEBUG:teuthology.nuke:shortname: smithi115 2024-03-25T18:39:35.789 INFO:teuthology.task.internal.check_lock:Checking locks... 2024-03-25T18:39:35.813 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi115.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/amathuri-2024-03-25_13:28:33-rados:verify-main-distro-default-smithi/7621279', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'centos', 'os_version': '9', 'arch': 'x86_64', 'locked': True, 'locked_since': '2024-03-25 16:18:57.128889', 'locked_by': 'scheduled_amathuri@teuthology', 'mac_address': None, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBNV2pGoe90+assupoD4niScAzAzu4+FXUpDaFYAxOMgNegi3l1KLiZsa3tGXUVJrb6Id8T/FVoGK7g9F8xwHv9c='} 2024-03-25T18:39:35.840 INFO:teuthology.orchestra.console.smithi001:Power off 2024-03-25T18:39:35.840 DEBUG:teuthology.orchestra.console.smithi001:pexpect command: ipmitool -H smithi001.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power off 2024-03-25T18:39:35.872 INFO:teuthology.orchestra.console.smithi115:Power off 2024-03-25T18:39:35.873 DEBUG:teuthology.orchestra.console.smithi115:pexpect command: ipmitool -H smithi115.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power off 2024-03-25T18:39:35.882 DEBUG:teuthology.orchestra.console.smithi001:power off output: Chassis Power Control: Down/Off 2024-03-25T18:39:35.882 DEBUG:teuthology.orchestra.console.smithi001:pexpect command: ipmitool -H smithi001.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-03-25T18:39:35.895 DEBUG:teuthology.orchestra.console.smithi115:power off output: Chassis Power Control: Down/Off 2024-03-25T18:39:35.896 DEBUG:teuthology.orchestra.console.smithi115:pexpect command: ipmitool -H smithi115.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-03-25T18:39:35.905 DEBUG:teuthology.orchestra.console.smithi001:check power output: Chassis Power is on 2024-03-25T18:39:35.919 DEBUG:teuthology.orchestra.console.smithi115:check power output: Chassis Power is on 2024-03-25T18:39:39.906 DEBUG:teuthology.orchestra.console.smithi001:pexpect command: ipmitool -H smithi001.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-03-25T18:39:39.920 DEBUG:teuthology.orchestra.console.smithi115:pexpect command: ipmitool -H smithi115.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-03-25T18:39:40.018 DEBUG:teuthology.orchestra.console.smithi001:check power output: Chassis Power is on 2024-03-25T18:39:40.029 DEBUG:teuthology.orchestra.console.smithi115:check power output: Chassis Power is on 2024-03-25T18:39:44.022 DEBUG:teuthology.orchestra.console.smithi001:pexpect command: ipmitool -H smithi001.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-03-25T18:39:44.030 DEBUG:teuthology.orchestra.console.smithi115:pexpect command: ipmitool -H smithi115.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-03-25T18:39:44.133 DEBUG:teuthology.orchestra.console.smithi001:check power output: Chassis Power is off 2024-03-25T18:39:44.139 DEBUG:teuthology.orchestra.console.smithi115:check power output: Chassis Power is off 2024-03-25T18:39:44.234 INFO:teuthology.orchestra.console.smithi001:Power off completed 2024-03-25T18:39:44.240 INFO:teuthology.orchestra.console.smithi115:Power off completed 2024-03-25T18:39:44.344 INFO:teuthology.run:Summary data: description: rados:verify/{centos_latest ceph clusters/{fixed-2 openstack} d-thrash/default/{default thrashosds-health} mon_election/connectivity msgr-failures/few msgr/async-v1only objectstore/bluestore-stupid rados tasks/rados_api_tests validater/valgrind} duration: 7898.645892381668 failure_reason: 'valgrind error: Leak_StillReachable operator new[](unsigned long) UnknownInlinedFun UnknownInlinedFun' flavor: default owner: scheduled_amathuri@teuthology sentry_event: https://sentry.ceph.com/organizations/ceph/?query=d2944dc623e541a0b710f1474f198101 status: fail success: false 2024-03-25T18:39:44.345 DEBUG:teuthology.report:Pushing job info to https://paddles.front.sepia.ceph.com/ 2024-03-25T18:39:44.426 INFO:teuthology.run:FAIL